[RELEASED] Realistic Eye Movements

I second this suggestion, .. at least ;-)

 

Hi cly3d, thanks for the picture, that looks great!

I can see how such an action system would be useful. It encompasses a lot of different non-eye movement related things however, and is not a trivial addition. I think a system like that would be better placed in a separate asset which has more to do with general AI behaviour, where people expect it more than the current asset which is called Realistic Eye Movements.

Thanks for the suggestion though, I'll keep it in mind!

 

Hi Faikus,

Really awesome asset - it adds a LOT of life to some of my characters. I have it working for the most part, but am experiencing an issue where the eye direction is somehow offset. This doesn't really hurt anything other than if the Look At Player Ratio is set to 1, the character *never* looks at the camera - but off to the side...



Example of movement + Dynamics



Another issue is that this doesn't seem to work in a web-build. I realize the web player is being discontinued, but I'm confused why this works in a stand-alone build, but not the web-player?

I'm also curious as to why I can't just pick my head bone and let the system animate that - i.e. why it has to be a mecanim rig or Fuse rig to work?

In light of these, a few suggestions:

Add the ability to define Rotational Offsets to help refine the "look at" target.
Add the ability to pick a Custom Head Bone for the system to apply animation to
Add a "Speed Factor" that allows the animations to be sped up (hyper, scared, alert) or slowed down (sleepy, drunk, sexy) - and POSSIBLY the ability to animate this speed factor.

Thanks again for a great asset. It doesn't seem like these additional features would be that difficult to implement, but then I'm not a coder

 

Hi jwilliamsen,

About the eye angle: your model seems similar to one a previous poster had. His model had the center of the eye bones far back in the head, not in the center of the eyeballs. Is it possbile your model is like that as well? REM only works correctly if the center of the eye bones (or the center of the eye objects, if you set eye control to eye objects instead of Mecanim bones) are in the center of the eyeballs. If they aren't, can you reparent your eyeball meshes to fix that?

About the head bone movement, I'll try to add support to head movement without using Mecanim or FinalIK in a future version.

Do the Head Speed Modifier and Nervousness sliders already do what you had in mind with the speed of the animation?

I just tested WebPlayer, you're right, it seems Unity doesn't have some defines set in WebPlayer. The fix will be in my next update, until then you can do it yourself by changing line 165 in LookTargetController.cs from this:

Code (csharp):

1. #if UNITY_5_1 || UNITY_5_2 || UNITY_5_3 || UNITY_5_4 || UNITY_5_5 || UNITY_5_6 || UNITY_5_7 || UNITY_5_8 || UNITY_5_9

to this:

Code (csharp):

1. #if !UNITY_WEBPLAYER && (UNITY_5_1 || UNITY_5_2 || UNITY_5_3 || UNITY_5_4 || UNITY_5_5 || UNITY_5_6 || UNITY_5_7 || UNITY_5_8 || UNITY_5_9)

 

Hi Faikus.
So, I'm kind of set with the wanting of "random animations to playback" which I asked for a couple of posts earlier in this thread.
I'm slowly learning Playmaker, and to get things going, i've ditched Mecanim - it's a PITA for non experienced Unity users such as myself.
Legacy animation is very straighforward for what I want to do - an interactive movie in Unity.
Anyway.. the reason for this post is:

- Previously via mecanim, it was really smooth to get the character to "turn her head" and look at my FPS player object even while an animation was playing on her.

Now not so much. In legacy rig mode, if there is an idle animation playing and even if I've set REM to always look at player etc, I find only the eyes seem to be tracking my Fps controller the head seems to not want to move even with all constraints off.

Is there some fix that you know about that I'm missing?
Everything else works (using eye objects instead of mecanim bones, and blendshapes for eyelids)

Regards.

 

Hi cly3d,

I'm planning to integrate head-bone-only control as an option (so without Mecanim or FinalIK), hopefully it will be in the next update.

 

I just noticed an issue when using the Oculus Rift positional tracker.
Here's my setup:
- No final IK used
- Mecanim rig, IK pass through enabled on animator
- Eyes driven by bones, eyelids by blendshapes. (it's a Daz character that's properly mapped for mecanim)
- Head Weight= 1 and limit head angle =0.098
- Even gave it a 'dummy object' for Player eye center, and set dummy object to be about waist height of FP player.

The issue:
If I don't use the positional tracker for Oculus, all works fine by setting the OVRplayer controller prefab (which contains an oculus camera) to desired height

If I DO use positional tracker, the character (the NPC) with REM seems to look a couple of inches 'over' your head and stares into space. The character does turn toward you as you move around, but this not-looking-at your eyes (not looking into the player's eyes) is very apparent when you get close to the NPC.

This is evident if you're using positional tracking and sitting while playing the game. If you stand, the NPC will lift his/her head to look at you.

It seems the NPC does not lower their head if you're sitting while they are standing and thus "above" you in height.
Hope I'm explaining this right.

Anyone else notice this with Positional tracker and OVR Player prefab?

***edit**
here's the workaround I had to do:
- Drop look at player to almost zero
- Drop stare back to almost zero
- Added a "dummy" game object as child of the Oculus Camera rig and positioned it about -0.22 below "zero" on the Y axis.

Now she does look directly at you when you're upclose as well.

 

Hi cly3d,

Positional tracking should work, whether using OVR Prefab or not; it does for me. Can you please try to load the demo scene that came with REM, check "Virtual Reality supported" in the player settings, replace the main camera with an OVR prefab, put a plane for you to walk on, and see whether you get the problem there? In the scene with the character where you get the problem, if you check "Draw sightlines in editor" in the REM component, are they going above the target the character is looking at? Outside of play mode, are the eyes looking straight ahead or slightly upwards?

 

Hi Faikus,
Definately going above. Even when I set to look only at player and stareback at full, with the other POIs deleted.

I'll now try out what you mentioned in a new project.
and also will try with default "main camera" and not the oculus prefab and see if it makes a difference.

Regards

 

hmm.. It is completely correct in the demo scene with the Oculus Prefab.
I'll have to investigate why it's not the case with my scene. I should have thought of trying the demo first. Thanks, Faikus.



*edit*

So could it be because of the idle animation playing on my character?
Even though I've set REM to take control of head movement?

 

The animation shouldn't affect it, because REM runs in LateUpdate, with the animation already applied. Is there some other component on your character or outside that might be changing the characters position/pose? What happens if you put EyeAndHeadAnimator.cs at the end of the script execution order?

 

There's Lipsync Pro, that's it. Disabled it, but still no luck.
I moved EyeAndHeadAnimator.cs to the end, yet nothing. Will have to investigate more.

 

Could you try out these:
- replace your camera rig with a simple Camera tagged as MainCamera in front of the character
- let her look at a fixed object in the scene and see whether the eye sight lines look at it directly
- have a new empty scene with just the character, just the Main Camera, the original eyes, no eyelid control, no animator controller and see whether she looks at the camera directly

 

Thanks for this great asset.

I've found a bug when trying to export a custom preset using mecanim eye bones + eyelid bones:

NullReferenceException: Object reference not set to an instance of an object
RealisticEyeMovements.ControlData.GetExport (UnityEngine.Transform startXform) (at Assets/RealisticEyeMovements/Scripts/ControlData.cs:906)
RealisticEyeMovements.EyeAndHeadAnimator.ExportToFile (System.String filename) (at Assets/RealisticEyeMovements/Scripts/EyeAndHeadAnimator.cs:497)
RealisticEyeMovements.EyeAndHeadAnimatorEditor.OnInspectorGUI () (at Assets/RealisticEyeMovements/Editor/EyeAndHeadAnimatorEditor.cs:43)
UnityEditor.InspectorWindow.DrawEditor (UnityEditor.Editor editor, Int32 editorIndex, Boolean rebuildOptimizedGUIBlock, System.Boolean& showImportedObjectBarNext, UnityEngine.Rect& importedObjectBarRect) (at /Users/builduser/buildslave/unity/build/Editor/Mono/Inspector/InspectorWindow.cs:1231)
UnityEditor.DockArea:OnGUI()

I'm using unity 5.3.

p.s. I'm using IK to let me character look at the direction the camera is set (it's a third person cam), but it seems the eyes are always looking forward, is this intended? Btw. I've set the headWeight to 0 to let IK do it's job.

 

Hi silentneedle,

You're right, the fix will be in the next update.

 

I forgot to update this thread.
I found out that if I tick the "reset tracker on load" switch in OVRManager (the script that is part of the OVRCharacterController prefab) then the players eyes are tracked properly.

Player's eyes tracking does go slightly 'off' if I disable "use profile data" in OVRplayerController script.

So for now I have both, reset tracker on load, and use profile data enabled and eye tracking works fine for me.

 

I'm glad you got it working! It should work without having to check those settings, however (it does so in my tests). Maybe there is something different with your OVR setup in the scene. Do you have several OVRCameraRig game objects in your scene?

 

I have only one, but had to coax Oculus via their forums for a hot fix to their "positional tracking" toggle that is badly implemented (imo). The hotfix they sent, bypasses left/right camera check... but at any one point, I do have only one OVRCamera rig and only center camera is marked "main camera"

The first video clip in this article shows my progress so far. (it does look like the debug eyelines are not looking straight at the camera, but with the headset on, she does look directly at you)

http://realvision.ae/blog/2016/06/the-dilemma-of-positional-tracking-in-cinematic-vr-films/

 

Hi cly3d,

Realistic Eye Movements uses the position of left and right camera as well so it can look the user in the eyes. If you did modify the code for setting left and right cameras, that might be responsible for the problem. Can you send me a scene that has your changes and displays the problem?

 

Hi @Faikus,

Just bought your asset and really liking it, easy to setup POI etc, but I've instantly hit an issue with the eyelids.
Basically my character has 3 bones per eyelid, they all need to move different amounts to open the eye. From what I can see the system can only handle 1 bone per eyelid?
Its more a problem with my characters setup than with your system but it would be really helpful to me if you could give me some pointers to get around it, changing the rigging/weighting or the bone hierarchy are not really options.

 

Hi PopMark,

The only option right now is to change the bone hierarchy a bit in Unity's hierarchy panel to choose one bone of the eyelid as parent and make the others its children. This does not change how each bone influences the skin, so you don't have to redo the skinning/weighting. This does not let REM move them different amounts each though I'm afraid. Maybe I'll support more than one bone per eyelid in the future.

 

Hey Faikus,

when you'll release the next update?

 

Hi silentneedle,

Sorry for the late reply, it seems I was still checking the old forum. I hope to bring out an update within two weeks or so.

 

A new version is out, changes:

- Added preset for MakeHuman 1.1 default rig
- Fixed exporting preset when no blendshapes are used
- Fixed issue when using both FinalIK's LookAt and FullBodyBiped components
- Added OnStartLookingAtPlayer and OnStopLookingAtPlayer events

 

Hi Faikus

Sorry about the super-late reply - for some reason I was never notified of your initial reply....

About the eye angle: your model seems similar to one a previous poster had. His model had the center of the eye bones far back in the head, not in the center of the eyeballs. Is it possbile your model is like that as well? REM only works correctly if the center of the eye bones (or the center of the eye objects, if you set eye control to eye objects instead of Mecanim bones) are in the center of the eyeballs. If they aren't, can you reparent your eyeball meshes to fix that?

The bones in the eyes are dead-center of the eyeball - so that isn't the issue. What *might* be the issue is that the head is posed on import - i.e. it's starting out NOT at it's neutral rotation (but the eyes are centered) - could this be the problem?

About the head bone movement, I'll try to add support to head movement without using Mecanim or FinalIK in a future version.

I think that would be a great feature to have.

Do the Head Speed Modifier and Nervousness sliders already do what you had in mind with the speed of the animation?

Not really. There only seems to be a longer delay between actions - the actions themselves are not sped up or slowed down - which is what I would expect (as an animator, this is how I would change the "nervousness" of a character - faster movement)

I just tested WebPlayer, you're right, it seems Unity doesn't have some defines set in WebPlayer. The fix will be in my next update, until then you can do it yourself by changing line 165 in LookTargetController.cs from this:

Code (csharp):

1. #if UNITY_5_1 || UNITY_5_2 || UNITY_5_3 || UNITY_5_4 || UNITY_5_5 || UNITY_5_6 || UNITY_5_7 || UNITY_5_8 || UNITY_5_9

to this:

Code (csharp):

1. #if !UNITY_WEBPLAYER && (UNITY_5_1 || UNITY_5_2 || UNITY_5_3 || UNITY_5_4 || UNITY_5_5 || UNITY_5_6 || UNITY_5_7 || UNITY_5_8 || UNITY_5_9)

[/QUOTE]

Thanks for those fixes - I'll have to go back and see if I can add those to my project (I'll have to downgrade a machine to get back into the project since 5.4 dumped the web player (boo Unity... sigh).

 

Hi jwilliamsen,

The head being rotated at start is actually a problem. REM's definition of looking straight is in relation to the transform the REM is a component of at startup. Is it possible for you to turn the head so it's looking straight at startup?

The head bone feature has been added in a recent update. When REM is added to a character that doesn't have a Mecanim head bone, there is a slot for setting a custom transform as head bone.

The speed of eye movement setting makes sense, I will add that.

 

Hi @Faikus

I'm having a problem now where the character's eyes aren't lining up with the object they're supposed to be looking at, when their target point of interest is at eye level 1 meter in front they always look cross eyed looking down , any ideas what could be causing that?

I'm using the eye game objects control type, I also get null references on eyeXForms sometimes, when I change the points of interest transform array of the lookatcontroller.

 

Hi @popMark,

Can you check "Draw sightlines in editor" and see whether the lines are correctly looking at the target? Does the eye looking work when no pois are assigned and the character is just looking around idly in front of him?

When you check the point-of-interest array, it might be that the old poi transforms are destroyed while the eye animator component is still looking at one. When you change the array, you might call a function like LookAroundIdly after changing it, to make REM choose a poi from the new array.

 

With "Draw Sightlines" on the sightlines point to where the character is looking, basically down his nose, not at the POI 1m in front.

Calling LookAroundIdly seems to have sorted the NRE

 

So what happens if there are no pois assigned, does the idle looking look ok?


In EyeAndHeadAnimator.cs, could you insert this line at the start of the function public void LookAtAreaAround( Transform poi ) (so after line 998):

Code (csharp):

1. MiscUtils.PlaceDummyObject("poi", poi.position);

This should place a pink box at the poi that was chosen as the next one to look at. Is the box drawn at a valid poi from the list?

Also, in the same file, could you insert this line at the end of the function UpdateEyeMovements (after line 1400):

Code (csharp):

1. MiscUtils.PlaceDummyObject("eye target", eyeTargetGlobal);

This should put another box, named "eye target" at the position of the actual target the character eyes are looking at. Is this box near the box called "poi"?

Btw, are there any other scripts active on the character that could influence the eye and head animation?

 

With no POIs assigned he still looks down his nose

Adding the dummy objects showed that poi.position was in the correct place, but eyeTargetGlobal was normally appearing around the the world origin, offset at random values. This is probably what hes looking at since he is standing on the world origin.

as far as I know no other scripts are affecting the eyes, when I turn off all scripts affecting the character except the EyeAndHeadAnimator (including the LookAtController) it shows the same behaviour

 

Can you send me a screenshot of the two REM components (LookTargetController and EyeAndHeadAnimator) in the inspector panel so I can see the setup?

 

Sure thing, http://pasteboard.co/1neYTLMYU.png

 

Thanks. The settings look alright, the problem must be somewhere else. Is it possible to send me a test scene that shows the problem? Then I could definitely fix it.

 

Definately possible, I've sent you a test scene in a PM

 

Hi, I;m wondering How I can close the eyes when the ai dies? Thanx

 

If your character's eyelids are controlled by eyebones, you can set the EyeAndHeadAnimator's parameter EyeWidenOrSquint to 0 to keep the eyes closed, and the head weight to 0 to not move the head anymore.

 

Hello! I need to character look up most of the time and sometime look on camera. So I add "UpperLight" (It is definitely at the top) object to "Points of interest". Сharacter watch on camera nice, but when it stopped he just starting looking on side, or look forward, but not on the top. What am I doing wrong? Here is my setup.

 

@id0 Did you move the eyes up enough when saving the position "Looking up"? The Looking Up and the Looking Down positions save the maximum and minimum vertical angles that the eyes can move within. When outside of play mode you press the Load button next to "Looking Up", are the eyes set to look up as much as possible for the character?

 

It's all right with that, when I moving the camera around he is looking at camera well, head and eyes are moving, he is just not want to look up for some reason, on the points of interest... And what is that mean "player eye center"? It's not doing anything for me.

 

I see, thanks for the info.

PlayerEyeCenter is for third person games, where "looking at the player" means something other than looking into the game camera.

Could you try the following?
- Check "Draw Sightlines in Editor"
- Set the slider Look at player ratio to 0, so that the character should look at the poi all the time. Then move the poi around and look at the sightlines. Does the character look at the poi at all / only when it's not up?
- In EyeAndHeadAnimator.cs, at the start of the function LookAtAreaAround, place this line:
MiscUtils.PlaceDummyObject("lookTarget", poi.position);
This should place a pink cube where the character is looking at. Is the cube at the position of your poi?

 

I suppose top object was too far away. I move it closer and it's ok now. Thanks for the help

 

It seems the preset export still doesn't work 100%, I'm receiving the following exception:

 

I just bought REM and am excited to get it working with my little learning project. My characters are cartoons and do not have eyes that are spheres that rotate within a socket. eg. to make my character look up and down I translate the pupil up or down relative to the white of the eye, which is a separate object. Am I correct in assuming that this style of eye is not supported? Should I replace the eyes with spheres and do the regular sphere rotation?

 

Hi silentneedle,
Is this a character that had the scripts applied before the REM update? Could you try deleting the REM components, reapplying them and then export and tell me whether that makes the exception go away?

 

Hi c0nno0r,

You're right, the asset needs the eyes to be spherical. With spherical eyes the asset works well on cartoony cartoony characters, as you can see in the WebGL demo when you switch the character to abstract humanoid by pressing C a few times.

 

Yes, I'll try that.

 

I've run into some errors that prevent this from working when targeting the HoloLens. I get the following errors when Build Settings are set to the following target platform (Switching to PC/Standalone removes the errors, but I can't compile for HoloLens that way)

Assets/RealisticEyeMovements/Editor/EyeAndHeadAnimatorEditor.cs(43,25): error CS1061: Type `RealisticEyeMovements.EyeAndHeadAnimator' does not contain a definition for `ExportToFile' and no extension method `ExportToFile' of type `RealisticEyeMovements.EyeAndHeadAnimator' could be found. Are you missing an assembly reference?

Assets/RealisticEyeMovements/Editor/EyeAndHeadAnimatorEditor.cs(50,29): error CS1061: Type `RealisticEyeMovements.EyeAndHeadAnimator' does not contain a definition for `CanImportFromFile' and no extension...

Assets/RealisticEyeMovements/Editor/EyeAndHeadAnimatorEditor.cs(52,26): error CS1061: Type `RealisticEyeMovements.EyeAndHeadAnimator' does not contain a definition for `ImportFromFile' and no extension...​

The Build Platform Settings that cause this issue:


Thanks!

 

Ah, I see. I'll add the fix to the next update. Until then, you can add it yourself, just put these lines at the top of EyeAndHeadAnimatorEditor.cs:

Code (csharp):

1.  
2. #if UNITY_4_6
3.    #if !UNITY_WP8 && !UNITY_WP_8_1 && !UNITY_METRO
4.      #define SUPPORTS_SERIALIZATION
5.    #endif
6. #else
7.    #if !UNITY_WP8 && !UNITY_WP_8_1 && !UNITY_WSA
8.      #define SUPPORTS_SERIALIZATION
9.    #endif
10. #endif
11.  

And in the function OnInspectorGUI(), above the line GUILayout.BeginHoriztonal(), add this line:

and below the line GUILayoutEndHorizontal(), add this line:

The form of serialization that is used is not supported on this platform, so to import presets you would have to switch the platform in Build Settings, import the preset, then switch back. I know it's not the most convenient way, maybe I can come up with a better method in the future. Let me know if you still have problems.

 

So far everything works like a charm. Very easy setup and the eye/head movements work like intended.
However, I sometimes get a bug in play mode, where the head suddenly "snaps" back to the original position and from then on won't turn again. This appears to happen randomly so I have no idea where it comes from. Sometimes it happens a few seconds after entering play mode, sometimes after a few minutes and sometimes it doesn't happen at all.
The characters I'm using are made with Mixamo/Fuse.

Do you have any idea / is this an already know bug?