New FaceFX Integration With Unity

Okay. Your software is incredible; I'd buy it but I unfortunately don't have the money. But I'll be watching, silently, from a distance.

 

Is there an ETA for the new release? I'm very much looking forward to getting started.

 

Sorry Phil. The release slipped a bit and now one of our team members is on vacation. September 18th at the latest.

 

Thanks Doug.

 

So the morph still doesn't work in unity integration. I'm using megafilers, how would I integrate those to get morph based animation?

 

It would take some custom scripting.

Probably the easiest way would be to create a bone pose with 0 bones for each morph target that you want to control. You would name the bone pose such that you could find the correct morph target (i.e. "Megafilers-open"). Then in the update call when we get the value of all the bone poses, you would set the value of the morph target using whatever interface megafilers uses.

 

I see that the 2013 download is now available, but the shop just has the Studio Pro license available for $2000, which I think is the old price. Do I need to email to pay for an upgrade or what?

 

OK, we are live! For upgrades, start up FaceFX 2012 and click on "More Information" when it tells you that there is a new version. For new purchases, just check out the website!

 

FaceFX Studio 2013 looks like a really nice upgrade from the 2012 version. I especially like the fact that I, as a programmer, no longer need to go into Maya to set up the export to FaceFX. One question I have, though, is about the way FaceFX Studio 2013 imports body animations from the .fbx to play alongside the facial animations. Is the only way to use this feature to have separate CharacterName@AnimationName.fbx files, or is there some way to put them all in the same .fbx? Assuming that's the only way, would it be possible in a future update to define a series of frames in our batch export file for animations in a single .fbx the way we do for face poses?

 

Currently CharacterName@AnimationName.fbx is the only way, but there are 2 ways this could work:

1) Export partial FBX files from Studio. Just set the start and end-frame in FaceFX Studio to define what you want in the exported animation. (could be automated with other scripts inside of studio. Not an ideal solution but worth mentioning)

2) Modify the FBXImporter python plugin. Extracting the animations from FBX happens from the ExecFxOgreFBX command-line executable, and that supports extracting animations using a start and end frame. It's just that by default (line 77 of Scripts\FBXImporter\ogrefbxproxy.py), we always pass in very large negative and positive numbers for start and end time so that the animation gets truncated to the FBX file's start end time.

3) Add support for FBX "takes" in the ExecFxOgreFBX utility

#3 is on my todo list for the next update, because the takes have names, and it would "just work". But then you couldn't have all of the animations in one timeline like you are asking for...is that important?

 

Currently, that's the convention for our company. It also seems to be a supported convention by Mecanim with the way the new animation previewer lets you carve up animations from a single .fbx into multiple animation clips. Still, I suppose it wouldn't be a huge paradigm shift for us to use CharacterName@AnimationName .fbx files if that's the way you continue to support. We've started using the Unity 4 pre-order beta, though, and won't be able to use FaceFX again until your script for converting FaceFX events into Mecanim clips is ready.

 

Apologies if this has come up before, but I can't find it. I'm finding that the animation FaceFX generates is not as long (in seconds) as the audio clip it was given. The audio and animation sync fine in FaceFX, but I'm not sure which of the possibilities I'm supposed to use to ensure that this happens in Unity.

So should I:

1) Synchronize the start points of the audio and animation and ignore the fact that the end points don't match.
2) Synchronize the end points of the audio and animation and ignore the fact that the start points don't match.
3) Synchronize the start and end points of the audio and animation and ignore the fact that I've had to rescale the animation to match the audio.
4) Something I haven't thought of.


At the moment, I'm doing 3, but I'm not sure it's spot on. It might just be the streaming audio which is causing the lack of perfect synchronization though, so I thought I'd make sure I'm doing things the right way before I try to fix that.

 

In general, FaceFX will generate animations that are slightly longer than the audio file, not the other way around. If there isn't much silence at the beginning of the audio file, negative time keys are used to get the mouth into the correct position before the character starts speaking. Dealing with negative time keys is a pain, but for the facial animation to be correct when insufficient lead-in silence is provided, negative-time keys are the only correct way to handle the situation.

Unity doesn't support negative time keys, so you have to shift the animation forward, then trigger the audio at the correct point in the animation. The XML integration does this for you. If you are using FBX files, you will have to trigger the audio at the correct point on your own.

Are you using XML or FBX? Since all of this is taken care of for you with XML and the animation is force-synced to the audio, I'm guessing you are using FBX. In that case, the most likely explanation for the FaceFX animation being too short is that your selection range in FaceFX was off when you exported the FBX file. In FaceFX 2013, whatever is selected in the time range is what is exported to FBX.

So the correct answer is 4) because it depends if there are negative time keys.

As far as tips for dealing with negative time keys with FBX animations, here are your options:

• Ignore them and export the FBX animation starting at time 0. When you play the animation from unity, simply provide a little blend-in time. The beginning of the animation will be truncated but it is an easy solution that might yield acceptable results for your application.
• Insert more silence at the beginning of the audio so that negative time keys aren't a problem
• Record the (negative) start time of the animation, multiply it by negative one, and trigger audio at that time in the animation.

For completeness, I will mention that when analyzing audio in FaceFX you can select an analysis actor that will not generate negative time keys for the gesture curves (head rotations, blinks, eyebrow raises, etc.) It doesn't fix the problem with the lip-synch targets, but the gesture targets can be responsible for the most-negative keys (it frequently looks good to start speech with a head nod or other gesture)

If you have correctly exported the FBX file, but still think the animation is too short, contact us directly at support with files to reproduce the problem and I will look into it.

 

Thanks for the explanation - it makes a lot of sense. None of my three options had given me particularly good results, so I was hoping there was a fourth I hadn't considered. I am using the FBX pipeline, since I need to add animations at runtime. I didn't check to see what time region was selected when I exported - I just exported immediately after generating the curves. (They're only test animations for now.) So it sounds like there probably were negative time keys that I did not spot. It sounds as though the best solution will be to add a setting to manually enter the (negative) start time for each animation. That should give the best results, and I see no downside to this.

Thanks!

EDIT: And I'm pleased to report that doing so has resulted in the sync between animation and voice now being perfect. Thanks again for the fast help.

 

Great, glad that helped and thanks for bringing it up. Negative time keys and audio sync in general are important topics.

It's also worthwhile to note that the "play pray" method of audio synch (starting the animation and audio together and hoping they stay in synch) may not be reliable for longer files or in all situations. If you run into that problem, you will want to manually set the animation time based upon the audio time the way we do in the XML integration. Manually setting the animation time won't trigger events, but you can keep a separate animation for events.

 

Thanks for the suggestion. I'm already doing that. I'm streaming the audio from disk, so I figured it wouldn't be safe to rely on automatic synchronization just happening. I'm not using animation events, so it's no loss, and I like the peace of mind of knowing I've locked the audio and animation together.

 

I'm willing to get the FaceFx professional studio license. But before that, i got a couple of questions it would like to ask just to be sure it fits our Unity animation workflow. The pipeline we are using is purely FBX, no 3ds max/maya plugin, no XML data, we only works with FBX animations. So here is:
- When our character facial poses (phonemes/emotions) are done in our 3Ds max for example, we only need to define in a text file their keys (and that's what FaceFX studio needs)?
- Once the FBX animation files are imported in Unity, we can control and manipulate those animations as standard FBX animations clips i guess?
Tanks in advance,

 

You need to animate the poses in your Max timeline. At minimum, you should create the first 6 lip-synch targets. You might as well use the same frame numbers that we do, so that you can just use our default text file that specifies the pose names and frame numbers. Then FaceFX will use an FBX file exported from Max and the batch export text file.

Yes. If you are going into Unity via FBX, they are standard FBX animation clips.

 

That's exactly what the 2013 update was all about. The plugins are now not required at all if you're using Studio Pro, and there is no need to use XML as everything is baked into FBX animations if you prefer. You can still use XML, as far as I know - the menu options are certainly still there. But FBX in and FBX out seems to be the recommended path now, and it does indeed work just that way.

 

I've tried the No-save version and i should say, it really works! After finished with the facial poses in Max, it only took two clicks in FaceFX to make my character talking. I was impressed cause, the default lipsync generated by FaceFX requires almost no tweak at all. Definitely, I'm getting a license right away! Too bad, would have been nice to have FaceFX fully integrated into Unity.

 

Doug,

Any news on Unity 4 Mecanim integration?

 

Yes, I've made a proof of concept demo, but I would still encourage everyone to use the FBX pipeline. See below
demo
project
video

 

I'm glad to see a workflow on using FaceFX with Mecanim, but it's too bad that it's becoming more complicated to incorporate FaceFX, as the simplicity of adding the facial animations was something I liked before. Also, this workaround is fine for now, but I can see a future where Unity drops the legacy animation system and uses Mecanim exclusively. Are you guys working on a way to use FaceFX directly with Mecanim and not using the old animation system? Maybe something using Mecanim's Animation Curves (pro only) to drive the body animations?

 

For a Mecanim-only solution, just export FBX files from FaceFX Studio and bring them into Unity that way. Because face bones aren't included in the Mecanim control rig, you will want to "keep additional bones" (the facial animations won't transfer from one character to another because of this as well)

If you have a large game with lots of facial animations, then you may want to use the XML pipeline. The animation data is smaller, and the same audio-based animations can be used to drive different characters provided each character is set up with its own bone poses. I worked pretty hard to simplify this pipeline (there is only a single button now), but I'll admit that this pipeline is not for everyone. It is more difficult to use than the FBX pipeline, so only use it if you need to.

There are lots of improvements planned for Mecanim, and when more features are exposed we'll be sure to keep the FaceFX integration up to date. For now, you can't drive Mecanim with FaceFX data except via baked-out FBX files that are specific to that character.

 

Hi how can i put the phonemes , a oh , i , e ect i dont see thta in the batch file text . there is a list here

http://www.facefx.com/documentation/2012/W74
And i dont see those in the batch file , and dont see how i put those in the batch or the right procedure.

This are the speech targets
http://www.facefx.com/documentation/2012/W76

But what about the a, oh , e , i t, ect?
please help.

Thanks

Smiljan

 

Just create the speech targets for your character, you don't need to create targets for each phoneme.

There are over 60 phonemes facefx recognizes, but there is no need to create that many targets because many phonemes share the same target (P, M, B for example). The mapping between phonemes and speech targets is defined in the mapping tab. The default mapping will work for most people however, so there is no need to create a custom mapping.

 

Thanks so much for the reply !

 

One last thing if i make the morph targets inside max , for instance and then put inside face fx and prepare the file for to be used in unity , the morphs will work? or i will need to use bones? becuase will be amazing if i ca use the morph targets.

 

No, that won't work. This pipeline works with bones only. For morphs you would need to do some scripting and use some morphnig package from the Unity Asset Store.

 

Just a quick note to say that we have posted a thread about our new Text-To-Animation Demo here.

 

How exactly would eyes work in this case? Eyes are by default hooked up to Mecanim. Unhooking them would give FaceFX full access to them but then you can't do lookats etc through Mecanim.

 

Mecanim support?

 

You would have to decide if you want the eyes controlled by Mecanim or FaceFX. Mecanim doesn't have enough hooks yet to let FaceFX drive bones directly through the Mecanim interface. I'd probably go with Mecanim. FaceFX mainly just counter-rotates the eyes based on head movements, so if Mecanim controls the head, it should control the eyes too.

 

If all of our characters have the same rig, and all the characters have their own bone poses, then do we need to bring all those extra characters into FaceFX or are you saying we would just need to bring one character in for the initial audio and voice sync work and then when we move into the xml pipeline in Unity, we should be able to just have another character with the same bone poses and facial rig play the animation that gets exported from FaceFX is that correct?

 

That's exactly what I do, and it works great.

 

Thanks for jumping in Tony. That is correct, altgames, the animations will be interchangeable between your characters.

If you have different bone poses for each character, you will need to load each character up in FaceFX once and export a collapsed XML for for each character so that Unity can drive the bones poses with additive animations.

This next part doesn't apply to you because you have already created your poses, but for those who are trying to do that, keep in mind that you can save some work and use the bonepose transfer command in FaceFX Studio to transfer bone poses between characters with the same rig but slightly different bone positions. Alternatively, you can set the UseReferencePoseFromFBX boolean in the FaceFXControllerScriptGUI.cs file if you only want to use one XML file to define bone poses for all of your characters (the transfer is done inside of Unity in this case)

 

Hi,

sorry for the disturbing but i have some problemes for make the integration work I follow the mechanim video(http://www.youtube.com/watch?v=x4J36lnsI24&feature=youtu.be). My probleme is below:
*In FaceFx Studio Professional:
1- I load the exo.fbx and batch-export-exo files from C:\Program Files (x86)\FaceFX\FaceFX Studio Professional 2013\Samples\Src and the mechanim_facefx audio file from my FaceFx_Mechanim folder ( E:\Unity-FaceFX\FaceFX_Mecanim_Demo\Assets\FaceFX\Example Files) but when i play the animation, only the facial animation is executated while the rest of the body remains inactive.

2- I tried otherwise by taking the mechanic_facefx and exo-animations.fbx files from E:\Unity-FaceFX\FaceFX_Mecanim_Demo\Assets\FaceFX\Example Files with batch-export-exo from (C:\Program Files (x86)\FaceFX\FaceFX Studio Professional 2013\Samples\Src) but the character is load with different appearance and when i play the animation, only the audio and the gesture body is executed; the head stay inactive.

* I use Unity4 version 4.1 and i don't see the asset->import FaceFx XML and others buttons like that, is it normal or must i do some seettings?

Can someone can give me somes solutions?

 

1. Find the script FaceFXControllerScript and add it to your character. This will give you the "Import XML" button in the Inspector. Make sure to set the Scale Factor to the same value that you used to import your model.

2. The FaceFX integration currently uses the legacy animation system. The curves in the XML file reference specific bone names. Your model's bones must match these names exactly. Also, make sure Mecanim is not configured to use any bones that FaceFX will control with the legacy animation system.

3. Finally, check FaceFXControllerScript.cs and make sure that sections #1, #2, and #3 match your model. For example, below is a modified version that I use for characters that use the Darwin Default skeleton. The bones in Darwin Default are different from the bones in Exo, so I had to modify the file.

Hope it helps!

Code (csharp):

1. // EDITED VERSION (Tony Li)
2. // - Added Darwin bone targets and commented out Exo bone targets.
3. // - Uncommented head bone targets because we're using legacy animation instead of Mecanim.
4. //
5. //----------------------------------------------------------------------------------------------------------------------------
6. // A FaceFX Controller script to drive bones-based animations created with the FFaceFXControllerScriptGUI.cs file
7. //
8. // Owner: Doug Perkowski
9. //----------------------------------------------------------------------------------------------------------------------------
10. //  License Information
11. //
12. // Copyright (c) 2002-2012 OC3 Entertainment, Inc.
13. //
14. // Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated
15. // documentation files (the "Software"), to deal in the Software without restriction, including without limitation
16. // the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and
17. // to permit persons to whom the Software is furnished to do so, subject to the following conditions:
18. //
19. // The person or entity using the Software must first obtain a commercial license to FaceFX Studio Professional,
20. // FaceFX Unlimted, or a FaceFX Plugin from OC3 Entertainment, Inc. or an authroized reseller.
21. //
22. // The above copyright notice and this permission notice shall be included in all copies or substantial portions of
23. // the Software.
24. //
25. // THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,
26. // INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A
27. // PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
28. // HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF
29. // CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
30. // SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
31. //
32. //----------------------------------------------------------------------------------------------------------------------------
33. using UnityEngine;
34. using System.Collections;
35.  
36. [RequireComponent(typeof(Animation))]
37. [RequireComponent(typeof(AudioSource))]
38.  
39. public class FaceFXControllerScript : MonoBehaviour {
40.  
41. // If you need to add a new target, you must add it in 3 places:
42.  
43. // If you need to add a new target, you must add it in 3 places:
44.  
45. // 1.
46. public float open = 0;
47. public float W = 0;
48. public float ShCh = 0;
49. public float FV = 0;
50. public float PBM = 0;
51. public float wide = 0;
52.  
53. public float tBack = 0;
54. public float tTeeth = 0;
55. public float tRoof = 0;
56.    
57. public float Eyebrow_Raise = 0; // Convert spaces to underscores.
58. public float Blink = 0;
59. public float Squint = 0;
60.    
61. //--- Begin Darwin targets... [TL]
62. public float BrowAngry = 0;
63. //public float Head_Pitch_Pos = 0;
64. //public float Head_Yaw_Pos = 0;
65. //public float Head_Roll_Pos = 0;
66. public float LeftEye_Pitch_Pos = 0;
67. public float LeftEye_Yaw_Pos = 0;
68. public float RightEye_Pitch_Pos = 0;
69. public float RightEye_Yaw_Pos = 0;
70. //public float Head_Pitch_Neg = 0;
71. //public float Head_Yaw_Neg = 0;
72. //public float Head_Roll_Neg = 0;
73. public float LeftEye_Pitch_Neg = 0;
74. public float LeftEye_Yaw_Neg = 0;
75. public float RightEye_Pitch_Neg = 0;
76. public float RightEye_Yaw_Neg = 0;
77. public float Look_Up_Lids = 0;
78. public float Look_Down_Lids = 0;
79. //--- ...end Darwin targets.
80.        
81. // The head needs to be controlled by mecanim for humanoid characters.
82. //public float Head_Pitch_Pos = 0;
83. //public float Head_Pitch_Neg = 0;
84. //public float Head_Yaw_Pos = 0;
85. //public float Head_Yaw_Neg = 0;
86. //public float Head_Roll_Pos = 0;
87. //public float Head_Roll_Neg = 0;
88.    
89. // Exo targets, don't apply to Darwin:
90. //public float Eyes_Yaw_Neg = 0;
91. //public float Eyes_Yaw_Pos = 0;
92. //public float Eyes_Pitch_Pos = 0;
93. //public float Eyes_Pitch_Neg = 0;
94. //public float Mouth_Sad = 0;
95. //public float Mouth_Anger = 0;
96. //public float Mouth_Happy = 0;
97. //public float Mouth_Snarl = 0;
98. //public float Mouth_Pain = 0;
99. //public float Mouth_Pout = 0;
100. //public float Brows_Sad = 0;
101. //public float Brows_Anger = 0;
102. //public float Brows_Happy = 0;
103. //public float Brows_Pain = 0;
104.  
105.    
106. // 2..
107. private void playPoseAnims(){
108.     playPoseAnim("facefx open", open);
109.     playPoseAnim("facefx W", W);
110.     playPoseAnim("facefx PBM", PBM);
111.     playPoseAnim("facefx FV", FV);
112.     playPoseAnim("facefx ShCh", ShCh);
113.     playPoseAnim("facefx wide", wide);
114.     playPoseAnim("facefx tBack", tBack);
115.     playPoseAnim("facefx tRoof", tRoof);   
116.     playPoseAnim("facefx tTeeth", tTeeth);
117.        
118.     playPoseAnim("facefx Eyebrow Raise", Eyebrow_Raise);
119.     playPoseAnim("facefx Blink", Blink);
120.     playPoseAnim("facefx Squint", Squint);
121.    
122.     //--- Begin Darwin targets...
123.     playPoseAnim("facefx BrowAngry", BrowAngry);
124.     //playPoseAnim("facefx Head_Pitch_Pos", Head_Pitch_Pos);
125.     //playPoseAnim("facefx Head_Yaw_Pos", Head_Yaw_Pos);
126.     //playPoseAnim("facefx Head_Roll_Pos", Head_Roll_Pos);
127.     playPoseAnim("facefx LeftEye_Pitch_Pos", LeftEye_Pitch_Pos);
128.     playPoseAnim("facefx LeftEye_Yaw_Pos", LeftEye_Yaw_Pos);
129.     playPoseAnim("facefx RightEye_Pitch_Pos", RightEye_Pitch_Pos);
130.     playPoseAnim("facefx RightEye_Yaw_Pos", RightEye_Yaw_Pos);
131.     //playPoseAnim("facefx Head_Pitch_Neg", Head_Pitch_Neg);
132.     //playPoseAnim("facefx Head_Yaw_Neg", Head_Yaw_Neg);
133.     //playPoseAnim("facefx Head_Roll_Neg", Head_Roll_Neg);
134.     playPoseAnim("facefx LeftEye_Pitch_Neg", LeftEye_Pitch_Neg);
135.     playPoseAnim("facefx LeftEye_Yaw_Neg", LeftEye_Yaw_Neg);
136.     playPoseAnim("facefx RightEye_Pitch_Neg", RightEye_Pitch_Neg);
137.     playPoseAnim("facefx RightEye_Yaw_Neg", RightEye_Yaw_Neg);
138.     playPoseAnim("facefx Look_Up_Lids", Look_Up_Lids);
139.     playPoseAnim("facefx Look_Down_Lids", Look_Down_Lids);
140.     //--- ...end Darwin targets.
141.        
142.     // Let Mecanim control these:
143.     //playPoseAnim("facefx Head_Pitch_Pos", Head_Pitch_Pos);
144.     //playPoseAnim("facefx Head_Pitch_Neg", Head_Pitch_Neg);
145.     //playPoseAnim("facefx Head_Yaw_Pos", Head_Yaw_Pos);
146.     //playPoseAnim("facefx Head_Yaw_Neg", Head_Yaw_Neg);
147.     //playPoseAnim("facefx Head_Roll_Pos", Head_Roll_Pos);
148.     //playPoseAnim("facefx Head_Roll_Neg", Head_Roll_Neg);
149.        
150.     // Exo targets, don't apply to Darwin:
151.     //playPoseAnim("facefx Eyes_Yaw_Neg", Eyes_Yaw_Neg);
152.     //playPoseAnim("facefx Eyes_Yaw_Pos", Eyes_Yaw_Pos);
153.     //playPoseAnim("facefx Eyes_Pitch_Pos", Eyes_Pitch_Pos);
154.     //playPoseAnim("facefx Eyes_Pitch_Neg", Eyes_Pitch_Neg);
155.     //playPoseAnim("facefx Mouth_Sad", Mouth_Sad);
156.     //playPoseAnim("facefx Mouth_Anger", Mouth_Anger);
157.     //playPoseAnim("facefx Mouth_Happy", Mouth_Happy);
158.     //playPoseAnim("facefx Mouth_Snarl", Mouth_Snarl);
159.     //playPoseAnim("facefx Mouth_Pain", Mouth_Pain);
160.     //playPoseAnim("facefx Mouth_Pout", Mouth_Pout);
161.     //playPoseAnim("facefx Brows_Sad", Brows_Sad);
162.     //playPoseAnim("facefx Brows_Anger", Brows_Anger);
163.     //playPoseAnim("facefx Brows_Happy", Brows_Happy);
164.     //playPoseAnim("facefx Brows_Pain", Brows_Pain);
165. }
166.    
167.    
168. // 3.
169. private void initializeAnims(){
170.     ArrayList bonePoseAnims = new ArrayList();
171.     bonePoseAnims.Add("facefx open");
172.     bonePoseAnims.Add("facefx W");
173.     bonePoseAnims.Add("facefx PBM");
174.     bonePoseAnims.Add("facefx FV");
175.     bonePoseAnims.Add("facefx ShCh");
176.     bonePoseAnims.Add("facefx wide");
177.     bonePoseAnims.Add("facefx tBack");
178.     bonePoseAnims.Add("facefx tRoof"); 
179.     bonePoseAnims.Add("facefx tTeeth");
180.     bonePoseAnims.Add("facefx Eyebrow Raise");
181.     bonePoseAnims.Add("facefx Blink");
182.     bonePoseAnims.Add("facefx Squint");
183.        
184.     //--- Begin Darwin targets...
185.     bonePoseAnims.Add("facefx BrowAngry");
186.     //bonePoseAnims.Add("facefx Head_Pitch_Pos");
187.     //bonePoseAnims.Add("facefx Head_Yaw_Pos");
188.     //bonePoseAnims.Add("facefx Head_Roll_Pos");
189.     bonePoseAnims.Add("facefx LeftEye_Pitch_Pos");
190.     bonePoseAnims.Add("facefx LeftEye_Yaw_Pos");
191.     bonePoseAnims.Add("facefx RightEye_Pitch_Pos");
192.     bonePoseAnims.Add("facefx RightEye_Yaw_Pos");
193.     //bonePoseAnims.Add("facefx Head_Pitch_Neg");
194.     //bonePoseAnims.Add("facefx Head_Yaw_Neg");
195.     //bonePoseAnims.Add("facefx Head_Roll_Neg");
196.     bonePoseAnims.Add("facefx LeftEye_Pitch_Neg");
197.     bonePoseAnims.Add("facefx LeftEye_Yaw_Neg");
198.     bonePoseAnims.Add("facefx RightEye_Pitch_Neg");
199.     bonePoseAnims.Add("facefx RightEye_Yaw_Neg");
200.     bonePoseAnims.Add("facefx Look_Up_Lids");
201.     bonePoseAnims.Add("facefx Look_Down_Lids");
202.     //--- ...end Darwin targets.
203.        
204.     // Let Mecanim control these:
205.     //bonePoseAnims.Add("facefx Head_Pitch_Pos");
206.     //bonePoseAnims.Add("facefx Head_Pitch_Neg");
207.     //bonePoseAnims.Add("facefx Head_Yaw_Pos");
208.     //bonePoseAnims.Add("facefx Head_Yaw_Neg");
209.     //bonePoseAnims.Add("facefx Head_Roll_Pos");
210.     //bonePoseAnims.Add("facefx Head_Roll_Neg");
211.        
212.     // Exo targets, don't apply to Darwin:
213.     //bonePoseAnims.Add("facefx Eyes_Yaw_Neg");
214.     //bonePoseAnims.Add("facefx Eyes_Yaw_Pos");
215.     //bonePoseAnims.Add("facefx Eyes_Pitch_Pos");
216.     //bonePoseAnims.Add("facefx Eyes_Pitch_Neg");
217.     //bonePoseAnims.Add("facefx Mouth_Sad");
218.     //bonePoseAnims.Add("facefx Mouth_Anger");
219.     //bonePoseAnims.Add("facefx Mouth_Happy");
220.     //bonePoseAnims.Add("facefx Mouth_Snarl");
221.     //bonePoseAnims.Add("facefx Mouth_Pain");
222.     //bonePoseAnims.Add("facefx Mouth_Pout");  
223.     //bonePoseAnims.Add("facefx Brows_Sad");
224.     //bonePoseAnims.Add("facefx Brows_Anger");
225.     //bonePoseAnims.Add("facefx Brows_Happy");
226.     //bonePoseAnims.Add("facefx Brows_Pain");
227.        
228.     ArrayList succesfulInits = new ArrayList();
229.     ArrayList unsuccesfulInits = new ArrayList();
230.     for( int i = 0; i < bonePoseAnims.Count; ++i )
231.     {
232.         if(initializeAnim(bonePoseAnims[i] as string))
233.         {
234.             succesfulInits.Add( bonePoseAnims[i] as string);
235.         }
236.         else
237.         {
238.             unsuccesfulInits.Add( bonePoseAnims[i] as string);
239.         }
240.     }
241.     if( succesfulInits.Count == 0 )
242.     {
243.         Debug.LogWarning("FaceFX: No bone pose animations were found on the character. Did you import an XML?  Have the animatiuons been deleted reverting to the prefab");
244.     }
245.     if( unsuccesfulInits.Count > 0 )
246.     {
247.         string warning = "FaceFX: The following " + unsuccesfulInits.Count + " pose animations were not present in the Animation component:";
248.         for( int i = 0; i < unsuccesfulInits.Count; ++i )
249.         {
250.             warning = warning + " \n" + unsuccesfulInits[i];
251.         }
252.         warning = warning + "\n Animation data on these channels will have no effect.";
253.         Debug.Log(warning);
254.     }
255. }
256.    
257. protected Animation animationComponent;
258. protected AudioSource audioComponent;
259.  
260. //The rest is the same from this point down.

 

There is some additional setup required to get body animations working. After generating the mechanim_facefx animation, notice in the events tab that there are some stub animations defined that were created from the text file. But they don't do anything in your actor because you haven't set them up. Load up the FaceFX Studio Professional/Samples/Src/Exo.facefx to see what they should do.

I think Tony answered your question about the Import XML button (Thanks Tony!). To elaborate:

This part is not completely accurate. The curves reference "bone poses", not bone names. A bone pose can contain multiple bones and it specifies how the transforms are different from the rest pose. You have to make sure that the bone pose names defined in your script match the curves that are in your animations and the bone poses in your FaceFX actor.

You never really have to worry about bone names. Assuming that you are creating the XML file from FaceFX using your character, it will have the correct bone names and the bone poses will create the additive animations for Unity's legacy animation system.

 

Hi,

Thank TonyLi and Doug, i follow your precious advices and it look much better than before.

let me explain my new difficulties:
Now i have my Exo-BodyAnims.fbx character in Unity and when i execute the application, only the iddle animation play;
So, in the inspector, i tried to attach on him(Exo-BodyAnim) the mecanic_facefx audio file And i ear the song play but he always do the iddle animation without facial animation.

Bellow i expose all the configuration i did:
*In FaceFx Pro 2013, i open the Exo.facefx and my facial and body animations have been imported correctly but i don't ear any song i just see the character moving and talking.
Then i load him and export to Collapsed Xml ACtor and go to Unity.
*In Unity i open the mecanic-facefx project, i put the Exo-BodyAnims on the scenne, i select him and in the inspector section:
i firstly attach, in the Animator section, the "ExoController" file on the variable "Controller"
( i press play and i can see the character in idlle animation whitout song and facial animation)
-then i attach the mecanic script to the character:
I attach the "mecanic_facefx" audio file on his "Audio Clip" varriable
I attach too, the "Exo-BodyAnims" on his "Player Object" variable
On his "Animation Clip Name" variable i do nothing cause i don't know what i need to do

-Then i attach the FaceFxControlleurScript as you told me,
the "Audio Source" and "Animation Section" appear
i import the FaceFx xml that i load before with FcaFX Pro(i press play and nothing new happen)
In the "Audio Source ", i attach the "mecanic_facefx" audio file on the "Audio Clip" varriable and when i press play in Unity,
the the song play and the character is always in iddle animation whitout facial and gestures animations

*In Unity Console i have these Logs:
1. Initializing FaceFX Controller.
UnityEngine.Debug:Log(Object)
FaceFXControllerScript:initalizeController() (at Assets/FaceFX/StandardAssets/FaceFXControllerScript.cs:252)
<PlayAnimCoroutine>c__Iterator0:MoveNext() (at Assets/FaceFX/StandardAssets/FaceFXControllerScript.cs:333)
UnityEngine.MonoBehaviour:StartCoroutine(IEnumerator)
FaceFXControllerScriptlayAnim(String, AudioClip) (at Assets/FaceFX/StandardAssets/FaceFXControllerScript.cs:382)
mechanim:Start() (at Assets/FaceFX/Example Files/mechanim.cs:43)

2. The following 3 pose animations were not present in the Animation component:
facefx Mouth_Happy
facefx Mouth_Snarl
facefx Brows_Happy
Animation data on these channels will have no effect.
UnityEngine.Debug:Log(Object)
FaceFXControllerScript:initializeAnims() (at Assets/FaceFX/StandardAssets/FaceFXControllerScript.cs:183)
FaceFXControllerScript:initalizeController() (at Assets/FaceFX/StandardAssets/FaceFXControllerScript.cs:268)
<PlayAnimCoroutine>c__Iterator0:MoveNext() (at Assets/FaceFX/StandardAssets/FaceFXControllerScript.cs:333)
UnityEngine.MonoBehaviour:StartCoroutine(IEnumerator)
FaceFXControllerScriptlayAnim(String, AudioClip) (at Assets/FaceFX/StandardAssets/FaceFXControllerScript.cs:382)
mechanim:Start() (at Assets/FaceFX/Example Files/mechanim.cs:43)

3. No AnimationState for animation: on player Exo-Bodyanims
UnityEngine.Debug:Log(Object)
<PlayAnimCoroutine>c__Iterator0:MoveNext() (at Assets/FaceFX/StandardAssets/FaceFXControllerScript.cs:369)
UnityEngine.MonoBehaviour:StartCoroutine(IEnumerator)
FaceFXControllerScriptlayAnim(String, AudioClip) (at Assets/FaceFX/StandardAssets/FaceFXControllerScript.cs:382)
mechanim:Start() (at Assets/FaceFX/Example Files/mechanim.cs:43)

I left Animation Type to "Humanoid", so i removed the comments bars reserved to the Humanoid section into the FaceFXControllerScript.

That's my configurations, so tell me what wrong, what configurations, even the most little, i need to do; either in the scripts or in the inspector or somewhere else.

Thank you for your reply

 

I think this is the problem. In the original scene, this is set to "Default_mecanim_facefx", because that's the name of the animation to play (the "mecanim_facefx" animation is in the "Default" group). Change this to your animation name and it might work (The format is YourAnimationGroupName_YourAnimationName).

You can check the name of your animation if you go in Unity to Window->Animation. You will see all of the "bone pose" animations that start with "facefx" (select these and hit play to make sure they do what they should). You will also see the names of your animation (and possibly also the name of your animation followed by _eventtrack which contain the body animation tracks). If you hit play on these from the Animation Window, you won't see anything play because the FaceFxControllerScript needs to be running to evaluate the bone pose animation tracks. It should work for you when you hit play in your game.

 

Greaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaattttttttttttttttttttttttt it work! you right, i just put Default_mecanim_facefx and it work!
Thank you so much! it's maybe a little thing, but it's a great step for my motivation

So if i understand rightly, i can make some others animations(walk, run, ...) to my own fbx character,
then import him into FaceFX and load audio file animation and facial animation like Exo
And play all these differentes animations(facial, bodygesture and my own animations(walk, run, ...)) into Unity?
And for the audio file, what i need to do if i want my own audio animation based on my own text animation file(in french for example) ?

Maybe my questions have already been answered! i ask this cause for me the next step is to use my own fbx character with some animations(exple: run, hit, walk, ...) and associate to him the facial and body-gestures animations with my own audio and text animation

Thank you again, and i'm waiting your reply

 

When I loaded this project in Unity 4.2, the mecanim hand gestures were not firing. In the mecanim controller, in "Gesture Layer 1" and "Gesture Layer 2" I had to update the "none" state to have speed=1 to get things working. I've attached links to the updated project:

demo
project
video

Abel, let me know if you still have any questions. You need to buy FaceFX to create new audio-based animations and save them into XML files for your character (or create FBX files from FaceFX to use directly in Unity without the FaceFXControllerScript).

 

Hi,

I own FaceFX Pro and Unity Pro and have been working for a while to master all that is possible by combining FaceFX and Unity animation through scripting. I am trying for a simple goal of manually controlling the eyes of a model either through the FaceFX Unity Animation component or through the Mecanim component or totally separately. I am using the Exo sample from above and have made a lot of headway in many regards, but just cannot seem to control the eyes of Exo. I am currently playing around with rotating the eye bones directly via script and that works, but only if the Animation and Animator components are not enabled. If either of these components are enabled, they seize control of the eyes and my script can no longer control them. Do you have any advice as to what I could do to disconnect the Exo eyes from the FaceFX Animation and Animator components without backing up to character rigging step prior to FaceFX export? If I could at least break the Animation control, I could add to the Animator component to control the eyes, I think. I have tried several things at the Animation and Animator level to break the eyes connection, but thus far nothing has totally succeeded.

Thanks

 

What about removing the eye bone poses from the FaceFX XML before importing it? (I suppose you could do it earlier in the pipeline; this is just what I did with my experiment.) This way, FaceFX/legacy Animation won't touch the eye bones, and you can let Mecanim control them.

Similarly, you could unassign them from your Mecanim avatar so Mecanim won't touch them, and then manually set the animation time of the facefx eye pitch/yaw animations to look in the correct direction.

Or remove eye control from both Mecanim (avatar) and FaceFX (XML bone poses) and control them manually by script.

 

I took your advice and edited the XML file and removed the eye poses and eye bone references and then reimported the XML. Now my scripted eye bone rotations work with both (or either) the FaceFX legacy Animation and Mecanim Animator components enabled. I'll see if I can move my scripted animations into the Mecanim Animator now.

Thanks for your help.

 

Happy to help another FaceFX user! Doug at FaceFX might have a more official solution, but this works for me, too.

 

That works, thanks Tony. If you don't want FaceFX to control a bone, it's best to keep it out of the XML file so FaceFX doesn't know anything about it. Hand-editing the XML works, or if you want to do it from inside of FaceFX, there is a python tool (Plugins->Remove Bones) that you can use to remove the bone from the "Rest Pose". Once a bone is removed from the Rest Pose, it will automatically remove it from any bone poses too. Then you need to save re-export your XML.

If for some reason you had to have the FaceFX script running AND you wanted to control things via script, you would have to solve the problem with the Animation component's "layers", but it would greatly complicate things because your script would have to create AnimationState objects and add them to the Animation Component. The FaceFX controller script (and accompanying editor script) could be a useful reference because that's exactly what they do.

 

Thanks for the information. I should mention that to get this working I also removed the eye bones transforms via the "configure avatar". After doing these two things, I was able to write a script that allows Exo's eyes to track the mouse cursor while playing both the FaceFX Animation and Animator components. And I can cause the Default_mecanim_facefx animation to start over or restart by right clicking on the avatar and calling PlayAnim(). I also exposed the FaceFX controller playPoseAnim() function as public so I can call it from my script. This type of functionality is enough to let me go further down this path.

 

Is it possible to use morph targets (blend shapes) with Face FX, or does it have to be bone driven? If so, is there a performance hit between the two? Thanks!