New FaceFX Integration With Unity

Morph targets work in FaceFX Studio. And FBX files exported from FaceFX Studio with morph target animations will work when you import them onto your character in Max, Maya, and Unity. (Note, these FBX files do not contain the original mesh data, but they have everything they need to drive the original character's morph targets and bones).

As far as the FaceFX-XML pipeline, one of its main jobs is getting curves from XML into Unity. Once you have the curves, driving morph targets is actually easier than driving bones...you just shove the values into the morphs but with bones you need to drive additive bone animations and set those up as well. The only issue is that, for a morph character, you need to know the name of the mesh that contains the morph (and the blend shape modifier in Maya), and that info isn't in the FaceFX XML file.

For example, a Project Pinocchio morph character has a morph target called MouthOpen_m.

To drive that morph in unity, you need to create the following curve:
H_DDS_MidRes : Skinned Mesh Renderer.Blend Shape.m_expressions.MouthOpen_m

Generating that curve requires knowing the mesh name (H_DDS_MidRes) and the Blend Shape Modifier name (m_expression), both of which you can't get from the FaceFX XML file. So the hard part will be figuring out the easiest way possible to let you specify those. It might require a little hand tweaking of the script file, which isn't very user friendly.


I plan on updating the example XML integration to support morphs soon, and I will post it here. Until then you can try the FBX pipeline. The only benefit to using the XML pipeline with morph targets is that it takes care of starting the audio at the appropriate time in the case of animations that start prior to the audio start time. If your audio has some padding in the beginning, then you can avoid negative time keys and just start the audio when you play the FBX file.

As far as performance goes, the general rule is that bones are faster because they are hardware accelerated rather than computed on the CPU, but it would really depend on the character and how many bones it has, how many vertices it has, etc.

 

I have been using the techniques described above to add in procedural interactive animation that works with the facefx and mecanim animations. That is working well for eye saccades, blinks, pupil dilation, breathing, head moves etc. This is currently on the Slade facefx demo character. I'm currently doing this by calling ffxController.playPoseAnim() function from within my character C# script for most movements, but I am actually controlling the eyeball bones transforms directly for the eye saccades after having removed those bones from the facefx XML file and from the Mecanim. I would like to better integrate this into the facefx pipeline and am considering adding a new animation layer as Doug suggested and my procedural animations would take place on that layer. I'm thinking I should place this at layer "8", higher than the facefxcontrollerscript FACEFX_REFERENCEPOSE_LAYER at "7" and lower than the FACEFX_ANIMATION_LAYER at "9". Does this sound reasonable?

I'm probably going to create the new AnimationState's via script, using existing bones.

Can anyone think of any caveats with doing this?

 

Hi,
I wander is there any way to use fxa files in Unity? Yes, creating facial animations in some third party editor and then importing them in Unity doesn't seem to be end of the world, but still it is tedious and very long process (especially if your game uses many cut-scenes). So I want to know, is there any way to use fxa files and generate animations at runtime? Will FaceFX studio license allow this, or one still have to create animations in studio, import them in Unity, apply to models manually etc. etc.? And if this is the only possible pipeline for Unity, what is the advantage of using FaceFX? For example iClone has the same capabilities, even more - not only you can make some facial animations, but animate whole body and export it as fbx and use STRAIGHT in Unity, without any additional plug-in sh**. And it is cheaper than FaceFX studio. Yes, making just ONE animation in FaceFX studio and importing it unity may be funny... but when you have to create hundreds of animations, or may be thousands, it is not funny at all.

 

No there is not. FaceFX doesn't even use FXA files anymore, it is an old and unsupported format. To go into Unity, you can use XML or FBX.

You need to generate in FaceFX Studio, then bring the animations into Unity.

In FaceFX, you can export as FBX. Just go to File->Export. You can use FaceFX to trigger body gesture animation clips, though FaceFX is definitely a tool designed primarily for the face.

 

[/URL]http://

Can ANYONE please shed some light on what could be happening here in UNITY with a Face FX XML?
When i do the same animation but driven with an FBX it works as intended..whenever i use an XML i get...The predator face. predator face is BAD. ...sigh
This happened before but went away when we placed an animation controller on the model..This time? ...No luck, animation controllers have NO effect.
Has this occurred to anyone else? when i look at the file in 3d Max and face FX the vertices assignments move as they intend and do not stretch the model out. i have done numerous reexports of all parts included in both traditional XML and collapsed XML and i get the same result.

 

Hi Blacklion72%, in your FaceFXControllerScript, does Scale Factor match the scale factor of the FBX model? For example, I import my models at a Scale Factor of 0.01, so I have to set FaceFXControllerScript's Scale Factor to 0.01 also. If you import the XML at, say, 1.0, then all of the XML curves will move the bones 100x too far, causing your "predator face."

 

Hey everyone, we've released FaceFX 2015, and I've posted a new tutorial showing how to use it with Unity. I've also modified the controller script to work with morph targets, but you will have to edit the scripts to specify the morph target names in your content.

Project Files including new Controller Script with Morph Support (Note: 5/16/2018 - see below for a more recent download that is still compatible with recent versions of Unity. This version will not work)

 

We're upgrading a project to Unity 5, and the NPC dialogs/animations are not working at all. No errors or warnings in the console, though. Has anyone tried FaceFX in Unity 5?

 

Works just fine for me in 5.0.0f2. Are you using the FaceFX integration package? Did the API updater have a chance to run and update the scripts? Any issues with your models?

 

Glad to hear that, and thanks much for the quick test. We did have some import errors on the character models, and I thought I fixed them by switching the rig to "Store in Root (New)" but possibly I need to change that on all the character models and animations. I'll do that, and hopefully that'll solve the problem. I'll post back either way.

 

After importing the morph/bones package linked above, I upgraded it to Unity 5. You do get an error that obsolete APIs are used, but you can allow the process that automatically fixes them and everything works for me.

The automatic process changes Editor/FaceFXControllerScriptGUI.cs so that all instances of:
_target.animation
are changed to
_target.GetComponent<Animation>()

 

Yeah, that appears to have upgraded properly. So we can't find the problem. We used FaceFX 2012 for this project -- any reason that might be too old to upgrade nicely?

 

FaceFX 2012 should work just fine. The parts of the XML file format that the Unity integration uses have not changed since the format was created initially.

Someone else is having the same problem as you however. They emailed our support just now. So it is not specific to your code. Something is going on, I just haven't reproduced it yet.

The first thing I do when troubleshooting the system is to make sure that your bone pose animations are still working. The animations with names like "facefx open". Those are "standard" one-second animations that should play back from the Unity editor if you go to Window->Animation and hit play. If those aren't working, then that is definitely the problem. Are those working for you?

Have you tried to do a fresh import of your FaceFX XML file? That would be a simple-enough fix.

 

Well, the problem turned out to be something else entirely. FaceFX works great in Unity 5. Thanks!

 

After importing the package, scripts were updated automatically.
After trying to use the package in Unity 5.0.0p1 I'm getting these errors (both for bones and morph actually).
I'm just placing Morph/Bones object in my scene, attaching morph/bones script and trying to import XML as in the video.
Was something changed in the latest Unity patches or maybe it's something on my end?

"The AnimationClip '' used by the Animation component 'Morph' must be marked as Legacy."

Imported the following 10 additive bone pose animations:

Failed to import the following 12 additive bone pose animations because they contained no bones:

 

I've posted a new package that has the Unity 5 auto-updates, and it also fixes the issue sqallpl brought up by marking clips as legacy (clip.legacy = true; added at the top of FaceFXControllerScriptGUI.cs::AddOrReplaceClip ) Thanks sqallpl, let me know if you run into any other issues.

Unity 5 integration (Note: 5/16/2018 - this is the correct download compatible with recent versions of unity)

 

What is the prime difference between FaceFX Professional and Unlimited? Been reading old threads and seems how things are done has changed. On your website it says Professional doesn't include batching - so you can't batch export a bunch of audio files/lip sync?

 

Batch processing audio is the main difference other than price. FaceFX Studio Professional is fully scriptable, but with Unlimited you also have access to the analyze command. Unlimited is sold on a per-title basis with a contract and has unlimited seats and includes source code.

 

Thanks - so if I have several hours of voice acting, with the Professional version I would need to manually process every single lip sync?

 

That is correct. But you can drag audio files onto the application relatively quickly with Professional. And obviously you will be doing many operations "manually" for each file (naming the files, creating a parallel text file and naming it, playing the animation at the appropriate point in your game, etc.).

That said, the batch processing comes in very handy with large-scale productions. You can re-analyze all audio to try out new analysis settings, generate animations from source assets as part of a nightly build process, or automatically detect updated audio files and reanalyze them when you get a new batch of files from your voice actors.

 

Thanks for the information. Might need to be in touch at some point to discuss Unlimited.

 

Is there a difference between buying FaceFx from Asset Store or from your official website?

 

No, it is the same product.

 

Alright. Thanks for answering. Will purchase it in the near future

One more question. Does FaceFx allow capturing facial expressions through audio or do I need to manually animate them?

 

Hey @facefx ,

Are persistent events included in the Unity integration for FaceFX? Is it possible tog get smooth blends between the animations with non-zero blend times in Unity and FaceFX? Or maybe do we need to start and finish all animations with the neutral pose? How is it handled in Unity integration?

http://facefx.com/documentation/2015/W23

Another question. Would it be possible to achieve emotes that don't use any audio (idles, reactions etc. for characters that are just listening and not speaking at the time) with FaceFX for Unity and smoothly blend between them and speaking animations? What if I want my character to stay in pose from the latest frame of an animation? Let's say that I have short 'emote_sad' animation from FaceFX and I want to play it and then left the character in the last frame from this pose so he can wait for the next animation (another silent emote or voiced dialogue line).

How is the switching between separate FaceFX animations handled? Can you just play an FaceFX animation from the animation component and then play another one? Would it be smoothly blended or maybe it will change the animation values in one frame?

 

Sorry squallpl, I missed the notification that this thread was active.

No, there is no support for persistent events in the Unity demo, and I don't think they would mesh well with Unity's legacy animation system, so I wouldn't recommend trying to do anything with them. If you are an advanced FaceFx user you can use persistent events in an analysis actor to create gesture curves, but by the time they get exported to XML to go into Unity, the persistent events will have been baked out to curves.

Animations generated with default settings from audio in FaceFX Studio will start and end at the reference pose. So the easiest thing is to try to enforce that. If you modify animations by hand or change default settings, you can wind up with animations that do not start and end at the reference pose however. If you need to support animations that start or stop outside of the reference pose, then you have to look into blending. The blending functions that are available to the Animation component (like using Blend instead of Play) don't work because the FaceFX Controller force-ticks the animation to specific times in order to stay in sync with the audio.

So any blending system has to be written without help from Unity's internal functions. A very simple system would involve a "blendWeight" variable in FaceFXControllerScript_Base. In Update(), you would set the blendWeight variable to how you want it (depending on the tick time of the animation), and in playPoseAnim you would multiply blendWeight by "val" before setting the normalizedTime. As a simple test, I set the blendWeight to anim_eval_time / animState.length in Update, and I watched the test animation barely move the mouth at the beginning of the animation to fully moving it towards the end.

Sure, you can play animations without any audio using the FaceFx Controller Script. So there shouldn't be any problem playing the emote_sad animation without audio. And if that animation leaves the character in the sad pose, the character will stay that way until you play another FaceFX animation. And then it will snap to the reference pose or whatever the first frame of the new animation is set to in one frame, which you will want to fix.

This is where you will need to implement a blending system a little more complicated than the one mentioned above. Each bone pose (or morph) will need to cache its value when you start a new animation. During the blend-in time, the value that is set would need to be blended with the cached value accordingly.

I'll think about adding something like this in the next version.

 

Hi @facefx, I would like to do on-the-fly lip sync with a talking character. Is it possible to do this with the current release of FaceFX for Unity? In other words I would like to use a client based speech engine along with a simple Eliza chatbot script to answer questions that I ask.

 

(This is just my input as a customer; I don't represent any of the products mentioned in this post.) The FaceFX for Unity integration plays animations that have been generated at design time based on prerecorded audio. It doesn't do on-the-fly lip sync. You might consider SALSA with RandomEyes (runtime lipsync approximation) and RT-Voice (text-to-speech), both of which are available on the Asset Store. They'll do exactly what you're asking. However, SALSA does lipsync approximation, not phoneme matching. The results are good, but they don't match FaceFX's accuracy. FaceFX is the AAA industry standard, though, and produces top of the line animation.

 

Hey folks! I've been using the straight FaceFX to Unity system for a while and it's been working nicely, but now I'd like to play around with doing FaceFX > Maya > Unity. The long and short end of it is that I want to do some clean up and tinkering in Maya and then bring it into Unity. First step is easy, of course, I just import the FBX animation into a copy of my character Maya file and do the necessary clean-up.

The second stage where I import into Unity and get in functioning in there is where I run into some serious problems. In Unity, the exported animation (chloe@talk.fbx) runs fine on the model it's imported with in legacy mode. But, I cannot get it to work with the version of the character (chloe.fbx) on the same scene that is a humanoid and does not have the animation included with it in the FBX. If I switch the chloe@talk.fbx rig to another mode like Humanoid, not only does the model in chloe@talk.fbx disappear, but the animation still won't play on chloe.fbx. I'm not sure if I messed up the export from Maya or if I'm not configuring chloe@talk.fbx properly.

 

I'm not sure I know exactly what is going on, but remember that you can't play facial animations with the Animator component (other than jaw flapping) because the Humanoid skeleton doesn't define any facial bones. Make sure you are using the legacy pipeline and Animation component for all facial animations.

Are you sure this forum post is the right place to ask the question? Once the animation is imported into Maya, FaceFX's role is done, so the problem is not related to FaceFX if I understand this correctly.

 

i have facefx professional stdio , i import the xml file that i export from the facefx into unity3D ，when i play the anim with facefx xml ,my model is warpped. i do not why. The animation play normally in facefx ,but it cannot work in unity. anyone else has the case like this?

my progress: export fbx flile ->save the actor file ->export-> export collapsed xml actor
export the unity
the fbx is based on bone-pose animation

 

This is normally the result of the ScaleFactor not matching between your FBX import settings, and the one in the FaceFX Controller Script GUI. (See BlackLion72's message above). Make sure the scale factor is set correctly before you import the XML

 

i have set my scalefactor matching my fbx import setting before i import the xml,but i do not work at all.

 

i find a intersting thing that the model is wrapped as long as the facefxcontroller is initalizated. although i do not play animation with facefxcontroller.PlayAnim.

 

can you send me your .fbx file to support (at) oc3ent.com?

 

I am trying to get a character working with facial blend shapes, and I am confused on the part in the video where I am renaming variables in the script. Where can I find the names that I am supposed to rename variables to?

 

Yeah, renaming the morph parameters is confusing...but hopefully only something you have to do once at character setup.

In the FaceFX/StandardAssets/Morph_FaceFXControllerScript_Setup.cs file, there are variables defined at the top of the file. These variable names are generated from FaceFX morph target node names, but periods and spaces are converted to underscores (variable names can't have these characters). So drag your FBX file onto FaceFX Studio, and from the Face Graph Select widget (bottom left) select FxMorphTargetNodes to see all of the morph nodes. For each one of those, create a variable at the top of the script.

Then inside of the playMorphAnims function a bit lower in the script file, we map these variables to the morphs they drive in Unity. The best way to see the morph names in Unity is to select the mesh file, then expand the Skinned Mesh Renderer/BlendShapes property to see all of the morph names. The unity morph names should be similar enough to the variable names that you can match them up easily at that point.

 

@facefx Failed to import the following 12 additive bone pose animations because they contained no bones:
facefx tRoof
facefx tTeeth
facefx Blink
facefx tBack
facefx W
facefx FV
facefx wide
facefx ShCh
facefx PBM
facefx open
facefx Squint
facefx Eyebrow Raise

I'm facing this error when import xml for morph, pls help

 

Sorry for the delayed response. If your content is morph-only then it would be expected that that any bone poses you have would not have any bones in them. So you should be able to ignore this warning. FaceFX will always create a bone pose (even if there are no bones in it) for each entry in your batch export text file.

 

@facefx Unity mecanim do not support the legacy animation exported from FaceFX. Do I need more step to convert ?

 

@CuongO Unity mecanim does not support facial bones other than the jaw. When using FaceFX with mecanim, turn off the jaw bone in mecanim so that FaceFX can control it.

You can't "convert" FaceFX animations to mecanim. You have to play FaceFX animations on top of mecanim animations and control the bones that mecanim doesn't support.

 

@facefx Thank, I tried to export .fbx animation instead of collapsed xml & use the animation in .fbx exported file in Mecanim, it worked successfully. But need your confirm is I take the right way ?

 

Using the FaceFX FBX pipeline to go into unity is a great option, and it is the easiest too. The collapsed XML solution should really only be used if you have a very large pipeline and you are comfortable integrating the scripts into your solution. It is useful if you want to keep the animation data really small, or if you need to dynamically create animations. For most people, the FBX pipeline will be the way to go.

 

Was this discontinued?

 

No, still here making characters talk. Unity no longer sells external programs like ours on their store however. You have to buy from FaceFX.com

 

How much does FaceFX Unity runtime cost?

 

The unity integration posted in this thread is free to use. You can also use FBX files exported from FaceFX. Either way, you only need to buy the $899 FaceFX Studio professional license.

 

And, It's up to date with latest version of Unity?
Also, can it do real-time lipsync? I see on your website that * FaceFX Runtime licenses are sold separately and frankly what we need is run-time lipsyncs. Or animations needs to be baked somewhere and imported into Unity? Care to explain the workflow?
Sorry to ask but the thread started back to 2011 and I'm not sure if workflow remain the same.
Thanks,

 

The unity integration posted above in this thread does not use the FaceFX Runtime. It is free. And you can also use the FBX pipeline, also free. All you need is FaceFX Professional.

But FaceFX doesn’t work with live audio, only pre-recorded audio. Drag your FBX character into FaceFX, drag a WAV file onto the application, export an FBX file, bring it into unity.

 

Right so what's the FaceFX runtime for? If not to do real-time lipsync with audio files?