[RELEASED] Realistic Eye Movements

I haven't heard about that before. I use Mixamo characters a lot as well and don't see that problem. Could you check the console, is there an error message?

 

I can't remember an error message in the console. I'll try to reproduce this bug this evening, but as said before: it appears to happen randomly.

 

So I managed to capture this on a video. On 0:09 you can see the character looking at the player and right after that the head snaps back to the default position and won't move (the eye movements still work though). No errors in the console.

 

Thanks for the video. Are there any other scripts active on the character that interact with REM? If you could PM me a test scene that shows that behaviour, I'll find the problem.

 

eye blinking is wrong, Human eye blinking is not a liner interpolation between A-B possition. The eye closes faster than openes, The more nervous person is the quicker eye is opening, the more nostalgic person is the longer it takes for the eye to open itself up

 

Hi pan-master, you're right, the blinking can be better. I'll look up some more concrete references and implement their findings in a future update.

 

Hi, great asset. We had it working perfectly on IOS, but I'm just starting to try an Android build and am getting an error trying to load the REMsettings.dat file from the streamingassets folder, using the exact same code that was working perfectly on IOS. I'm targetting Android 5.0 and building using unity 5.5.2f1

eyeAndHeadAnimator.ImportFromFile(Application.streamingAssetsPath + "/REMsettings.dat");

This works fine in the editor but not on a connected Android tablet. Any ideas?

 

Hey Faikus, Great work on this. I have been using this asset for a while now but I ran across a weird issue when using it on a character that is walking forwards. It tracks the player object fine for a while but as it moves towards the object the head spins around and looks at the floor behind the character.

Check out the video to see what I mean.

I am using Unity 5.5 and there are no other scripts on that object other than REM. This issue does not appear when the animation contains no forward movement.

 

Thanks for the video. I'm working on an update that has (among other things) better support for when the character moves. Until then, you can add these lines at the end of the function Initialize of EyeAndHeadAnimator.cs:

Code (csharp):

1.  
2.             foreach (Transform t in createdTargetXforms)
3.                 t.parent = transform;
4.  

or you can disable eye animation when starting to move with lookTargetController.ClearLookTarget() and later start it again with lookTargetController.LookAroundIdly().

 

Hi, on Android the contents of StreamingAssets is in the packaged file, so you have to access the folder differently:

path = "jar:file://" + Application.dataPath + "!/assets/";

(from https://docs.unity3d.com/Manual/StreamingAssets.html)

 

Thanks Faikus,

I added the following:

if (Application.platform == RuntimePlatform.Android)
{
eyeAndHeadAnimator.ImportFromFile("jar:file://" + Application.dataPath + "!/assets/REMsettings.dat");
}
else
{
eyeAndHeadAnimator.ImportFromFile(Application.streamingAssetsPath + "/REMsettings.dat");
}

But it's still failing at ImportFromFile when running on an Android device.

The page you linked to says that on Android you need to use the www class to load from the streaming assets folder, so is it possible you could provide an example of how to modify ImportFromFile (and/or CanImportFromFile) to achieve this?

 

Sure, try this:

Code (csharp):

1.  
2. using System.Collections;
3. using System.IO;
4.  
5. IEnumerator LoadPreset(string presetFilename)
6. {
7.    string path = "jar:file://" + Application.dataPath + "!/assets/" + presetFilename;
8.    WWW loadPreset = new WWW(path);
9.  
10.    yield return loadPreset;
11.  
12.    string newPath = Application.persistentDataPath + "/" + presetFilename;
13.    File.WriteAllBytes(newPath, loadPreset.bytes);
14.    eyeAndHeadAnimator.ImportFromFile(newPath);
15. }
16.  
17.  

Just call this with StartCoroutine(LoadPreset("MyPreset.dat"));

 

Thanks, that gets me further, but it's still not perfect.

On the Android device I'm getting a "NullReferenceException: Object reference not set to an instance of an object at RealisticEyeMovements.EyeAndHeadAnimator.SetMicroSaccadeTarget (vector 3 target global)" immediately after the ImportFromFile

In the Editor, I get a 'SerializationException: serializationStream supports seeking but its length is 0' at the same point.

Does this give you enough to figure out what's going on?

 

BTW I can see the dat file being created in the files folder on the device, and it looks the right size (7.52KB) so that bit appears to be working.

 

In the editor you would still have to load it the old way, as the Streaming Assets are not in a jar file. So you could do something like this:

Code (csharp):

1.  
2. #if UNITY_EDITOR || UNITY_STANDALONE
3.    EyeAndHeadAnimator eyeAndHeadAnimator = newCharacterGameObject.AddComponent<EyeAndHeadAnimator>();
4.    eyeAndHeadAnimator.ImportFromFile(Application.streamingAssetsPath +
5.    "/mypreset.dat");
6.    LookTargetController lookTargetController = newCharacterGameObject.AddComponent<LookTargetController>();
7.    lookTargetController.Initialize();
8. #else
9.    StartCoroutine(LoadPreset("mypreset.dat"));
10. #endif
11.  

As to the error on the Android device, maybe your code creates the LookTarget component before the EyeAndHeadAnimator is initalized? Can you try replacing, in the LoadPreset coroutine, the line

Code (csharp):

1.  
2. eyeAndHeadAnimator.ImportFromFile(newPath);
3.  

with

Code (csharp):

1.  
2. EyeAndHeadAnimator eyeAndHeadAnimator = gameObject.AddComponent<EyeAndHeadAnimator>();
3. eyeAndHeadAnimator.ImportFromFile(newPath);
4. LookTargetController lookTargetController = gameObject.AddComponent<LookTargetController>();
5. lookTargetController.Initialize();
6.  

and delete the code you have in some other place that creates the two components?

 

Thank you so much. Works perfectly in both the editor and the android build now.

 

Hi, I just purchased REM and tried to implement in my Mixamo/Fuse model, but unfortunately it's been a lost cause. The problem is my Mixamo model's eyes are labeled "default" (which I changed to Eyes) but even worse, the eyes are not separate objects and REM seems to be looking for these as separate objects. When trying to use the Mixamo preset, here's the error I get:

X_Nurse2 (1): Cannot find path mixamorig:Hips/mixamorig:Spine/mixamorig:Spine1/mixamorig:Spine2/mixamorig:Neck/mixamorig:Head/mixamorig:LeftEye
UnityEngine.Debug:LogWarning(Object, Object)
RealisticEyeMovements.Utils:CanGetTransformFromPath(Transform, String) (at Assets/RealisticEyeMovements/Scripts/Utils.cs:25)
RealisticEyeMovements.EyeRotationLimiter:CanImport(EyeRotationLimiterForExport, Transform) (at Assets/RealisticEyeMovements/Scripts/ControlData.cs:48)
RealisticEyeMovements.ControlData:CanImport(ControlDataForExport, Transform) (at Assets/RealisticEyeMovements/Scripts/ControlData.cs:747)
RealisticEyeMovements.EyeAndHeadAnimator:CanImportFromFile(String) (at Assets/RealisticEyeMovements/Scripts/EyeAndHeadAnimator.cs:319)
RealisticEyeMovements.EyeAndHeadAnimatorEditor:OnInspectorGUI() (at Assets/RealisticEyeMovements/Editor/EyeAndHeadAnimatorEditor.cs:50)
UnityEditor.DockArea:OnGUI()

Here's a screenshot of my model's tree:

I'm not sure if there was some setting I should've checked or missed in Fuse or not, but I could use some help here. I also made sure IK was uncheked, but that doesn't seem to have any effect.

 

R.E.M. works with Fuse figures.

Make sure your figure's rig is set to Humanoid (import settings), and the Mecanim Animator Controller is set to "IK Pass" (see blue checkmark in this pic)



Here's my screen cap from very early in this forum (page 1 or 2) for my Fuse figures. The REM options may have changed but the basics are still there… Most important is to set:
Eye Control: MECANIM EYE BONES
Eyelid Control: BLENDSHAPES



Good luck!

 

Hi,

REM doesn't look for separate eye meshes, but for eye bones. In addition to what wetcircuit said about the Humanoid rig, in that same import tab (where you choose between Humanoid and Generic rig), make sure that either "Optimize Game Objects" is unchecked (otherwise it removes bones like eye bones from the game object) or if you want to keep it checked, expose the eye bones.

If the bones still don't appear when you uncheck Optimize Game Objects, then this Mixamo figure wasn't exported with eye bones. I think there is a checkbox in Mixamo Fuse where you can set the number of bones to export, maybe look for that. Otherwise, what you could do is to separate the eye meshes in a 3D program and set REM's eye control to eye game objects instead of Mecanim eye bones. Or you just disable the eye mesh and add eyes from a Unity asset like Eye Advanced.

 

Hi,

Great asset. Had it working well in another project then updated from asset store last night and imported into project but cant see the Head Transform control in either script so nowhere to reference head and as a result, not getting any head movement...how do I fix this?

 

The head transform field appears if your character doesn't have an animator or if the rig doesn't have a head bone. Did you add an animator to your character?

 

I have an issue which I think is able to be solved some way. It's just haven't managed to figure it out.

I'm using a LookAtIk component from FINAL IK and it is also moves the body to follow the player. Only then it causes the npc to shake like crazy. When I set the slider of body weight to zero it works normally again. Why is that? I have another scripts attached that does this in the update:

eyeAndHeadAnimator.Update1();
lookAtIK.solver.Update();
eyeAndHeadAnimator.Update2();

 

If you just use LookAtIK and don't need complicated updates between different FinalIK components, you don't need the Update1/Update2 thing (it should work anyway of course, maybe there is something else in your setup that causes the problem), just add a LookAtIK component, set the desired spine objects and set the head/body sliders, and you don't need to call any update functions. Is there a reason you manually call the update functions?

 

Greetings. Does anyone know which variable of the REM look target controller can be used to change the LookAt position dynamically? I've tried "targetPoi" but that does nothing.

 

If you want to switch the target from one object or position to another via code you can call a method like LookAtPOIDirectly (see the API). If you want to move the target around (without switching between targets), just assign an object (can be an empty GameObject) to the POI list and move that. Let me know if you need any more info.

 

Thanks for the hint, though if I use 'LookAtPOIDirectly(dynamicTargetTransform)' nothing happens. I guess this is because I need to add the dynamicTargetTransform to the POI list, right? It would be nice if you could add a method to make the look target controller work on any Transform/Vector3 without the need to add it to a list first.

 

I made it like that:

Code (CSharp):

1. using System.Collections;
2. using System.Collections.Generic;
3. using UnityEngine;
4. using RealisticEyeMovements;
5.  
6. public class InteractivePoints:MonoBehaviour{
7.  
8.     public float gizmoOffset = 2;
9.     public float pointGizmoOffset = 0;
10.  
11.     public Transform[] points;
12.     public LookTargetController playerLook;
13.  
14.     void OnTriggerEnter(Collider other){
15.  
16.         if(other.CompareTag("Player")){
17.  
18.             other.GetComponent<LookTargetController>().pointsOfInterest = points;
19.         }
20.     }
21.  
22.     //___________Trigger Icon____________
23.  
24.     void OnDrawGizmos(){
25.  
26.         Gizmos.DrawIcon(new Vector3(transform.position.x, transform.position.y + gizmoOffset, transform.position.z), "ITrigger.png");
27.  
28.         foreach(Transform point in points){
29.            
30.             Gizmos.DrawIcon(new Vector3(point.transform.position.x, point.transform.position.y + pointGizmoOffset, point.transform.position.z), "LookPoint.png");
31.         }
32.     }
33.  
34.  
35.     void OnTriggerExit(Collider other){
36.  
37.         if(other.CompareTag("Player")){
38.  
39.             other.BroadcastMessage("ChangePoints", SendMessageOptions.DontRequireReceiver);
40.         }
41.     }
42.  
43.     public void ChangePoints(){
44.        
45.         playerLook.pointsOfInterest = points;
46.     }
47. }
48.  

 

@id0 Thanks but that's still overkill for my use case. For now I'll just stick with Unity's "Head Look Controller" as it's a single script with a simple 'target' variable. Still, REM will always be one of my favourite assets when it comes to human behaviour simulation.

 

You don't need to add it to the list, the method should make the character look directly at the transform or position specified. In the demo scene, that method is what makes the character look at an object when you press the corresponding button. If you send me a screenshot of your setup (so I can see the two components LookTargetController and EyeAndHeadAnimator in the inspector and the values they have), I hopefully will be able to tell you what's going on. Also, do you get an error message in the console?

 

@Faikus Sorry, the error was on my side. :-/ It took some time until I realised I had another IK script overriding the head movement. Nevertheless, thanks for the support!

 

Any updates, progression, direction with the asset?

 

Yes, there will be new features in the next update like more control over the eyelids. I don't have a concrete timeline yet, but as soon as my client projects will leave me time, I'll finalize it. Stay tuned!

 

Hi,

I'm having issues. I'm trying to set up REM with FinalIK look at but my character looks at the floor:

 

Did you uncomment the line "// define USE_FINAL_IK" at the top of EyeAndHeadAnimator.cs?

What Final-IK components do you have on your character?

 

Hi, yes I did uncomment that line.

I just use look at ik from final ik, however I have salsa on there too and my character is an UMA so I followed your instructions in the manual for adding the components at runtime.

 

I've just set up UMA with FinalIK's LookAtIK, this code works fine if you call StartCoroutine(AddREM(newGO)) with the generated character as newGO:

Code (csharp):

1.  
2.        IEnumerator AddREM(GameObject newGO)
3.        {
4.            yield return null;
5.            yield return null;
6.  
7.            Transform[] spines = {RealisticEyeMovements.Utils.FindChildInHierarchy(newGO, "Spine").transform,
8.                                            RealisticEyeMovements.Utils.FindChildInHierarchy(newGO, "Spine1").transform };
9.  
10.            RootMotion.FinalIK.LookAtIK lookIK = newGO.AddComponent<RootMotion.FinalIK.LookAtIK>();
11.            lookIK.solver.SetChain(   spines,
12.                                                RealisticEyeMovements.Utils.FindChildInHierarchy(newGO, "Head").transform,
13.                                                null,
14.                                                newGO.transform);
15.  
16.            RealisticEyeMovements.EyeAndHeadAnimator eyeAndHeadAnimator = newGO.AddComponent<RealisticEyeMovements.EyeAndHeadAnimator>();
17.            eyeAndHeadAnimator.ImportFromFile(Application.streamingAssetsPath + "/UMA.dat");
18.  
19.            RealisticEyeMovements.LookTargetController lookTargetController = newGO.AddComponent<RealisticEyeMovements.LookTargetController>();
20.            lookTargetController.Initialize();
21.        }
22.  
23.  

Can you try that and compare with your code and tell me if there still is a problem?

 

Hi, that works perfectly now. I was using the UMA create bones feature and adding final IK to the prefab, so perhaps this is why.

Thanks for your help.

 

Whoops, I spoke too soon. Once in my main scene and spawned in I get this error on each character:

 

It seems animator.GetBoneTransform(HumanBodyBones.LeftEye) returns null, so either there are not eye bones or the animator doesn't have eye bones assigned to the avatar. Is there anything about your setup that could cause this?

 

I removed all the bones created for the UMA and then added a 2 second delay to adding the REM scripts and it works without the errors but now the characters are not looking directly at me, they're looking to the side of me. Strange.

 

Does the head movement work correctly if you just use FinalIK without REM? Also, can you send me a test project that shows the problem?

 

It seems my male UMA characters have the issues, the females are fine.

Here's the male with issues:

Here's the female which is correct (ignore the male voice, not recorded female voices yet)

 

Thanks for the videos. That is weird. Maybe something about the UMA setup for male and female is different. It would be helpful if you could send me a small test project that has the problem.

 

Thanks, I'll see what I can do.

 

Hi!
It seems that the REM script completely overrides the eyes animations.
For example when I import an animation where the character has the eyes a little closed (like when you look at the sun), the eyelids animations are completely ignored.
Is there a way to blend together imported eyelids animations and REM?
Thx

 

Not right now, but what you could do is to animate the property EyeWidenOrSquint (if your eyelids are controlled by bones) in such situations.

 

Hi, loving REM; works great. Have a feature request. Would it be (or is it) possible to trigger user-configured blendshape or animation changes in response to the player entering the REM character's personal space?

Right now the character can look away if you get too close, but I would like to perform other responses, such as sticking out a tongue, closing eyes, puckering lips for a kiss, or even generating a sound event ("Hey!"). Would that be possible?

 

Hi Eldurin,

actions that go beyond where the character is looking are outside of the intended scope of the package; but what I can do is to add events that you can subscribe to that are triggered in those situations. In your event handlers you can then activate any blendshapes or make the character say what you want. I will add these events to the next update.

 

Sounds great; thank you!

 

I just bought this last night, and the asset store page says it's no longer available. Is there some info about this?