New FaceFX Integration With Unity

The Runtime is like a highly efficient version of the scripts posted in this thread, along with an implementation of unity’s component animation system.

We haven’t found a need for real-time audio analysis. If you are interested in audio files, it’s better to pre analyze and store efficient animation data. If you are interested in microphone (live) audio, you are probably better off using video as an input like the new iPhone or performance capture solutions.

 

Here's our current issue:
We have gazillions of real-time game dialogues, so importing and managing lot of FBX animation files is not a good solution. That's why we where looking into something that will animate our avatars and keep animation data really low (real-time audio analysis or the like).
Another solution that can work for us is something with text scripts to drive characters face phonemes and emotions. We have a similar system but heck is clunky and very limited.

 

Unless you can use text to speech for audio, then you are stuck with dealing with heavy audio data. If you are using FBX animation data, depending on your character setup, animations can take up more space than compressed audio! This is the scenario where our Unity scripts can come in handy. Animation data size will go down to perhaps 10% of the audio data. Could you get it down further doing realtime analysis? Sure...but at a heavy cost of CPU, latency, code complexity, and development time (we don't offer a solution to analyze in game). You will also take a hit on quality at the very least because you won't have the option of tweaking animations on important audio files.

So if you are storing audio data, my recommendation is to store the animation data too. If you have a lot of audio data, store the animation data efficiently (use a system like our Unity scripts), and spend your development time coupling the audio & animation data, streaming it, etc.

If you are using TTS or microphone audio, then it can make sense to analyze on the fly, but expect quality to go down a bit.

 

Forgot to ask something. How big the data generated by FaceFX studio is compared to let's say regular animation data exported in FBX?

 

Both depend on your setup. Bones or morph, and how many of each, and how rigorously the fbx animation data is compressed. For our Slade sample character, our FaceFX Runtime animation data runs about 850 bytes per second. FBX animation data stored per bone per frame can be 10X that. You would have to test your character in unity to get exact figures, but you can expect to see major savings over FBX.

 

Oh so FaceFX do support both, morphing and bones facial animation? That's quite interesting. :O
Lastly, the 899$ license includes the Unity C# components to play the facial animations and the FaceFX studio to generate the animation data using voice/text? Is that right?

 

FaceFX Professional is $899, it supports morphs and bones, and the C# components are free. Before you buy, try the free no-save evaluation on our website, and download the most recent C# scripts from this thread. You can also try the C# import process using the sample XML files in the no-save installation (you can't create your own XML files due to the no-save restriction)

 

Are you guys planning support for image-based facial phose-animation anytime in the future? Pretty much like Faceshift and Faceware? This will make FaceFX extremely competitive against those guys.

 

The new iPhone and all of the facial capture data it can provide with its 3D camera system may prove to be a tempting target for a FaceFx Studio integration.

 

Indeed and both of mentioned software works in a similar way as FaceFX (with facial poses). So it's a matter of getting the data input and converting it into FaceFX facial poses.

 

Hello, I tried the evaluation version extensively and really like it, but in the end I could not settle for it.
These are some shortcomings for me :
1. I would love to be able to export phoneme data and their timings (same for matched text timings), also id love to export amplitude data. In general a lot more export options would be great.
2. Text matching is fantastic, I would love to be able to mark audio chunks dynamically in the facefx software and assign the matching text chunk, rather than the clunky other way around. This would be super useful for generating subtitle timings right in facefx.

I might ask for features that facefx has not been designed to deliver, I wish it would though : )

 

Thanks for the feedback.

1) Phoneme times can be exported via the getPhonemeList python function or by exporting an xml file. These options are not available in the “no-save” evaluation however.

2) You can select a range of phonemes from FaceFX and select “reanalyze range” to provide text that matches that section. Is this not the feature you are asking for?

 

Hello

Thank for the quick reply!

I totally missed reanalyse range, it works great!

Would you be so kind to upload an example xml with phoneme and matched text timings + the processed audio file?
My use case is, that I want to be able to swap mouth sprites based on the current phoneme.
Is it possible to export the reanalysed text with timings as well? Would be a fantastic starting point for creating subtitle segments.

Thanks a lot !

 

there is a sample (Slade.XML) file in the Samples/Src directory of the FaceFX installation folder. It has the animations from the Slade example character.

 

And just for fun, if you select the “cartoon coarticulation” analysis actor from the analyze wizard, you can see an animation on a 3D character without any transitions between the phonemes....so it will look like a 2D sprite animation where the sprites are screenshots of the character pronuncing the phonemes.

 

How can we combine the body movement actor with the cartoon coarticulation, so that we don't need to use them one after another and still get the head and eye movement which is not part of the cartoon coarticulation? Ideally by having the eyelid movement also "cartoonlike" optionally, while keeping head and eye movement fluid.

Also we want to use mecanim for all the other animations and combine it with faceFX. While this works out of the box, mecanim overwrites the head movement of FaceFX. Ideally it would be additive, so we can have the headmovement in sync with the speech while having control over the direction the character looks for example as part of our motion captured animations.

And one last question: We are going to have over 4k individual speech files, is there a way to have at least the first automatic analysis in some kind of one-click batch process? This would save us a lot of time

Thanks for this amazing product by the way!

 

I'm not 100% sure I understand this, but I think what you want to do is generate a custom analysis actor using the python scripts in the Scripts directory. There is GestureLib.py for a "normal" analysis actor, BodyGestures.py for one that has additional curves for driving body gestures, and CartoonCoarticulation.py for the cartoon coarticulation. So if you wanted to create a combination of the Cartoon Coarticulation and the normal head rotations & blinks, you could open the cartoon coarticulation actor and run the GestureLib.py file...or open a blank actor and run CartoonCoarticulation.py followed by GestureLib.py. You would need to tweak the scripts to make the blinks cartoony...line 218 of GestureLib defines the Blink curve so you could try tweaking that.

You would need to set up your rig with an extra head bone in the hierarchy. I can't stop mecanim from driving what it thinks is the head bone, but if I'm driving a child of the mecanim head bone there are no conflicts.

Yes, but it's much more expensive. "Batch Processing" is how we differentiate the market between triple-A developers and the folks that really need a discount to make their projects work.
https://documentation.facefx.com/doc/batch-processing

You bet!