[RELEASED] Realistic Eye Movements

Asset Store link

Realistic Eye Movements makes your character’s eyes and head look around, at the player, or at objects in a lifelike fashion. Just drop two scripts on your character and set a few parameters. If your character’s eyelids can be moved via bones or blendshapes, this asset can take advantage of that to produce an even more convincing effect. It has additional support for usage in Oculus VR projects and is the code behind the VR demo Coffee without Words.

More information and webplayer demo

Full documentation, describing how to apply the component to your character: Google Doc

Quickstart video:

If you have any questions about this asset, need help setting things up, or requests for features you need, please let me know by emailing me at tamulur@yahoo.com or posting in this thread.

Good luck with your projects!
Tore Knabe

 

Any video on how to use this asset???

 

My characters the eyes are separate objects parented to the head (not boned, not blendshapes). Can I still use your system?

 

That's a good idea, I will work on one.

 

Yes, you just select "Eye gameobjects" from the Eye Control combobox and drag your eye gameobject into the two slots for left and right eye.

 

Until I have made a video, here is the full documentation with detailed instructions: pdf

 

The demo is eerie. I love it.

 

I have added a quickstart video.

 

Really nice pack. There are some features that would be really usefull, and I think that they are really easy to implement:

-value for limiting horizontal eye movement. (sometimes eye drifts away from a natural position)

-value for minimum and maximum head movement (Vertical and horizontal), think of a person that is scared or sorprised.

-possibility to make faster head movements. (head weight). Sometimes the head isn't moving fast enough.

-when using blendshapes, a possibility to add a movement to blendshape. Usualy, in real humans, when eye looks down, upper eyelid closes more, and lower eyelid moves down, and when looking up uper eyelid opens, and lower closes. Thos behaviour would help to make avatars more realistic (and it will make them look more intelligent).

- A good feature would be a possibility to influence a ROLL head axis. Some people on some ocasions, when looking to left or right roll their head over the Z axis.

-Adrenaline VS. Sleepy mode: we humans, are reacting to different stimuls in different manner. When we are tired, we are slow and our eyes move slower. When we are scared, our eyes move faster.

-Anti-cross eye function: eyes focus on a target cam, and that's OK. But sometimes, when camera comes close, eyes get crossed. Humands tend not to cross their eyes when looking at someone. They tend to cross them when reading or looking at small details, but not on faces. It could be very handy to have a "small focus distance" value, where avatar won't try to focus, avoiding a silly crossed eye look.

I'm making only a suggestions, this isn't meant to be a critic on a plugin that is gorgeous.
And finaly, here is a test video that I made today:

 

@olix4242:
Thanks for that very informative and constructive feedback! These features are indeed easy to implement, they will be in the next update.

About the eyelids, you are right, it adds a lot to realism if the eyelids move up and down with the eye. The package already does that for eyelids that are controlled via bones, the webplayer demo is an example of that. I can implement that for blendshape-controlled eyelids as well. I haven't done it because I don't know whether people actually have a setup with separate blendshapes for the upper and lower eyelid. When the eye moves down, the upper eyelid has to close, the lower has to open, so if you have a single blendshape that controls how much both open or close at the same time, this won't work. I see in your video that you use a character from Mixamo Fuse, they come with a single "eye close" blendshape I think, did you separate the blendshape into upper and lower eyelid manually? I actually wrote a post on Mixamo's forum some time ago requesting they provide separate controls for the eyelids.

Speaking of the video, she looks gorgeous! Will you release your Rift demo some day? I cannot wait to meet her in VR!

 

Hi, thanks for your fast reply.
In Mixamo there are Squint.left and Squint.right blendshapes that moves together both eyelids, while EyeClose works only for upper lid. So, by using them together you can move eyelids independently by a reasonable amonunt. So I think it should work also with scripting.
Another idea: you have a character distance activation part, and this is great. one small adittion could make experience better: distance treshold value for Personal space. If we come too close, into a face of someone that doesn't knows us, people tend to look away.
Now I'm impatient to see a new update
If you need some help, count on me.
For a Rift demo, I think I will be releasing it really soon.

 

I was just playing around with the blendshapes in my Mixamo character. All eye blendshapes move both eyelids either towards or away from each other, so I don't think I can get the effect of the upper eyelid moving down and the lower eyelid moving down when the eye looks down. But maybe it is possible to approximate this with the current blendshapes. Oli, could you set the eye bones' x axes to 20 degrees (eyes look down) and set the eyelids blendshapes to values you feel right, then do the same for -20 and tell me which blendshapes you set to which values? That would be helpful for me to find out how to best approach this kind of setup.

The personal space thing I can do, would be interesting to see an NPC getting shy

 

"Oli, could you set the eye bones' x axes to 20 degrees (eyes look down) and set the eyelids blendshapes to values you feel right, then do the same for -20 and tell me which blendshapes you set to which values?"

For 20 degrees looking down:
Facial_Blends.Blink = 72
Facial_Blends.EyesWide = 100
Faccial_Blends.Squint = 0

For -8 degrees looking up (can't go further with my character)
Facial_Blends.Blink = 0
Facial_Blends.EyesWide = 50
Facial_Blends.Squint = 100

Normal face looking forward in my setup is:
Facial_Blends.Blink = 10
Facial_Blends.EyesWide = 0
Faccial_Blends.Squint = 0

 

Thanks Oliver! I'll look into that.

 

Thanks for this!

 

@wetciruit: That looks great, well done!

 

The latest updates implemented most of the features olix mentioned (except for roll of the head, which I'll look into in a future update). For example, you can now make the eyelids follow the yes vertically also for blendshape-controlled eyelids. I still hope Mixamo will eventually provide more control for the eyelids in their Fuse program to make that look even better.

 

Hi Tactical,
Thanks for the screenshot. Could you check the calibration of Looking Straight/Looking Up etc outside of Play mode, by pressing the respective Load button and see whether the eyes orient correctly? If that looks alright, could you set Look At Player Ratio to 0 and drag an object into the Points of Interest list, so the character keeps looking at the object only, then uncheck micro- and macrosaccades and move the object up and down so the character's eyes follow it and you get a better picture of under what circumstances exactly this happens? If you want, you can also send me a test scene and I'll find out what's wrong with the scene setup.

 

Hi Tore, just bought realistic eye movement but am having a problem. I am using the DK2 and am using the eyes on another character. However, your script seems to be overriding the occulus viewports with some truely nausiating results
Initially set up using the mechanim eye gameobjects, then set them directly to the character's eyes (makehuman import and am targeting the python eye.L eye.R (transform). Both seem to cause the same issue, its is as if they are overiding the OVR left right anchors. Have checked to make sure the scripts are not attached to the OVR rig in any way. Not sure if I have missed something or if this is a known issue. Am using Unity 5, any help would be greatly appreciated, your software looks great and very much want to make it work on my end. Adam

 

Update. I shut the project down loaded a saved version and reattached the scripts it seems to work fine as long as I use mechanim not the python transforms. Possibly this is to do with scripts on my end but problem seems sorted.

 

Hi Adam,

I cannot reproduce your issue with the info you gave me (Unity 5, Makehuman character, targetting eye.L and eye.R). My scripts don't modify the OVR anchors, so the problem must be somewhere else in the scene. If your problem persists, can you send me a test scene so I can help you find it?

 

Hi Tore @Faikus , I am using the latest version of Realistic Eye Movements in Unity 5. I've brought a Mixamo Fuse character in with facial blendshapes and am having challenges getting the asset to work. I've configured as best I can based on your video, disabled eyelid control since I can't find those particular gameobjects... but the character's eyes stay totally fixed upon play. Help! 1) getting eyes to move at all, 2) getting eyelids to work with a Fuse character.

 

Here are my settings with a Fuse figure... Hope it helps.

 

Hi Greg,

There is an error message in your console output. The eyes don't move because of that error. It seems your scene doesn't have a camera that is tagged as MainCamera. The script needs that to find out which way to look at the player. Just tag the main camera as MainCamera.

As wetcircuit showed, Mixamo Fuse characters' eyelids are controlled via blendshapes. Just set the combox for eyelid control to Blendshapes and follow the steps in the pdf for Blendshapes.

Let me know if you still have any problems.

 

@wetcircuit ,
I replicated your settings. My question is, how do you get the eyes to close, and thus save the "Eyes Closed, Looking Straight" position, when there are no available controls for that in the Heirarchy?

 

Haha, I had to read that 4 times. TAGGED was the keyword. I haven't used tags before.
YES!!! It worked after applying that fix.

Now I will need to read your PDF to understand how to tune the eyelid Blendshapes.

THANK YOU!! EXCITED!

PS - here is what we're using it for, inspired by Ready Player One:
http://dsky.co/apps/sceneplay/

 

@Faikus , @wetcircuit
I read the PDF, and am in the "Skinned Mesh Renderer" component of Mixamo Avatar's:
a) Body, and
b) Eyelashes

I've found the component "Facial_Blends.Blink_Left"... which gets me halfway there with the upper lid : see screenshot; I set this at 100 for both the body and eyelash gameobjects. Additionally I've tried "Facial_Blends.Squint_Left", which gets me another 10%... can somebody tell me where the lower lid Blendshape control is located?!

 

It actually looks completely awesome even with the only 50% blink... Tore, you are a GENIUS!

 

@Faikus
In the spirit of @olix4242 here are some things that would be Nice to Haves, perhaps a "R.E.M. pro" version?

- have the head follow the eyes. As a result of implementing your code, I've started studying how humans actually respond to 'Points of Interest'. Generally, they move their eyes first, and if it merits actual attention, within 1 second their heads follow, allowing the eyes to come back to center simultaneous with the head rotation.

- have some form of dynamic assignment of the active point of attention. We're building some machinima type cutscenes, and it would be nice to be able to send a message to LookTargetController saying, essentially, "direct your gaze at this transform now"

Other than that, it is superb. Along with Salsa LipSync, I feel our characters have suddenly 'come alive'! Thank you, Tore!

 

Hi Greg,

The Blink_Left, Blink_Right sliders should close the eyes completely, are you sure all other slider values are 0? Are you using a current version of Fuse?

Cool looking demo, can't wait to try it with Vive!

About dynamic assignment of the active point of attention: LookTargetController already supports that, for example you can call lookTargetController.LookAtPoiDirectly( targetTransform); Check the documentation for the LookTargetController functions at the end of the pdf.

About the head following the eyes: I do something like that in my demo Coffee without Words: when she notices you looking at her, she looks at you without moving her head, and when you keep looking at her she turns her head as well. It is not in the asset though. The asset does have a parameter for head latency, which does something similar, but currently it is set internally in a function. Maybe I will add it as public function parameter to a future update. For now, if you want you can hack the scripts yourself, just find the places where headLatency is set inside the LookAt... functions in EyeAndHeadAnimator script and set it how you want.

Let me know if there are any more problems.

Good luck with your project!

 

I have made a Fuse figure where the eyes couldn't blink all the way... it was because of my eye shape settings in Fuse.

 

Interesting, good to know, thanks!

 

I'm having an issue with Mixamo Fuse character in Unity 5.1 - the eye movements works perfectly, however the characters head never moves. I have a controller on the character playing an idle animation and the only head movements the character performs are from that animation.

Is there anything obvious I need to do to make the head move around through your scripts?

 

Hi, the "obvious":



Open the Animator Controller's IK settings, and check on IK PASS

 

That was exactly it! Thanks for the help

 

I am working with DAZ models. Can I send you a model so that you try if your asset would work on it? Or perhaps you could check if the free Morph 3D (which actually is DAZ) works with your asset?
https://www.assetstore.unity3d.com/en/#!/content/38086

Thank you for the help.

 

Thanks wetcircuit, that's looking great!

hermie, the Morph 3D character works without problems. Just set the combobox selections to Mecanim bones for eye control, blendshapes for eyelid control and save the required looking up/down limits, and your character is ready.

 

@Faikus
I have purchased your asset, but I don't understand the blendshapes yet.
I am using RealEyes, and it does not come with eye lids. Also, my model does not have eye lids.
I thought that "Blendshapes" would perhaps fake them, but I guess that was just a dream, right?

Edit: I have purchased "Advanced Eye" now, and I like this asset a lot more than "RealEyes", however, I am still having the same problem.

 

Hi Hermie,

Blendshapes are morphing information for the mesh that is saved in the model file (the .fbx or .max etc file). If you are using a model like the one in Morph 3D, you have eyelid blendshapes: on your model, find the child object that has the skinned mesh renderer for the face (something like Genesis2Female.Shape_LOD1) and in this skinned mesh renderer's blendshape section, move any of the sliders that have "EyesClosed" in their name between 0 and 100. You will see the respective eye lid moving.

Real Eyes and Eye Advances are just models and shaders for eyeballs, they do not contain eyelids.

 

Faikus, Great asset! Is it possible to have the character always look at you when you're within range (and stay looking at you), and then look away when you're out of range? I've been playing around with the settings but I can seem to get it to work that way.

 

@Faikus Thank you for your reply. Do you think you might be able to auto-select it? Anything that takes a burden from me is much appreciated, and I think it would result in less support requests for you.

 

@adamz There is something similar, but it does a bit of a different thing: Notice Distance. When set to a value greater than 0, it will let the character look at the player briefly if the player approaches from outside of this distance to closer to the character. If you just want to always look at the character when inside a certain distance and don't look at him when outside then you can control that by adding a script component to your character that checks the distance to the player every frame and sets the LookTargetController's lookAtPlayerRatio variable to 1 when the distance is below your threshold and to 0 otherwise. Let me know if you need more info for what you want to do.

 

@hermie Auto-selecting is unfortunately not possible because different models have different blendshapes, their names and effects are different so I cannot deduce from the script which blendshapes have to be set to which values. The model might have several sliders for closing eyelids, so the user has to try them out and find those that look best.

 

@Faikus, another question... my eyelids are bone driven, and I rotate them on the x axis and position them accordingly. After saving all the eye presets, when the scene plays the eyelids rotate side-to-side, on the z axis.

Also the "Eye Widen Or Squint" doesnt do anything either.

Whats the best way to correct that? I'm using Max if there's a solution within there too. Thanks.

 

@adamz Could you test whether the lid positions have been correctly saved by clicking on the Load button next to each pose ("Eyes open, looking straight" etc) and see for each pose whether the eyes and eylids are ok? The axis for the rotation of the eyelids is determined by the difference of the rotation between the default pose and the look-down pose, so maybe something is wrong with one of these saved poses. Once all poses are saved correctly, the eyelids should work, including the widen/squint slider.

 

Each of the load positions are correct. I even re-assigned each of the two scripts and I'm still having the problem, along with not being able to move the slider for widen or squint.

Update... The slider does work but only when Playing the scene. When I do move the slider however it rotates along the wrong axis, like outlined above.

 

@adamz Thanks for checking. When you compare the straight-ahead pose and the looking-down pose for the eyelids, are the eyelids actually rotated down a bit in the second compared to the first? That difference is what the rotation axis is determined from. If you want, you can also send me a test scene with just the model (or even just the head) and I'll have a look.

 

It works now. Before I only rotated the eye bones for look up/down. Once I also rotated the eyelids up/down slightly for look up/down, that corrected the problem. Thanks for your help.

 

Can this work with Final IK (or even better, could it be ported for final IK)?