Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

New FaceFX Integration With Unity

Discussion in 'Assets and Asset Store' started by facefx, Feb 15, 2011.

  1. facefx

    facefx

    Joined:
    Jan 18, 2010
    Posts:
    116
    There is an updated FaceFX Integration with Unity. Some of the information contained in this post is no longer relevant, so please post here with new comments.

    In general, the integration is easier to use and understand. You can still:

    • Rapidly generate bones-based facial animations for your Unity characters using FaceFX
    • Save about 90% on animation filesize compared to default FBX export

    The new integration has the following changes:
    • Bone poses can be transferred from Unity, removing the need for a FaceFX Studio Professional license to perform this operation.
    • Scaling meshes in Unity is now supported
    • XML control files are imported from the Assets menu, or from the control file for procedural updating of the FaceFx controller.
    • The actor name variable in the XML file is no longer used to match XML files to the correct Game Object.
    • A new more generic control mechanism is used to control the conversation flow. The same unity executable can be used to create vastly different conversations, with different characters by updating the XML control file that is loaded (by updating it in the executable, or changing it on the web).

    There is a new video tutorial explaining the pipeline with the FaceFX evaluation plugins:


    As always, feedback is welcome. I have Unity Professional on my machine, so please let em know if there are any problems running through the tutorial with Unity Free.
     
  2. pkid

    pkid

    Joined:
    Jul 10, 2009
    Posts:
    201
    I know you have mentioned before that the animations from facefx are pretty compact, I'm just curious to know if you have an estimate of how many megabytes per minute of audio the animations from facefx take.
     
  3. facefx

    facefx

    Joined:
    Jan 18, 2010
    Posts:
    116
    Using the Facefx demo as an example, my calculations are showing about 54KB (.054 MB) per minute of animation added to the streamed web executable. Plus another 10KB of "fixed" cost on a per-character basis. So the animation data size is negligable compared to the audio.

    If you stream in the audio, you can also stream in the animation data in XML form. In XML form, the animations are about 250KB/minute, but that's still negligable compared to the audio. The integration supports streaming animation data in and loading it at runtime, but the demo does not show it off. On the todo list is to remove the reliance on System.XML to further reduce the web player size.
     
  4. GWPGearWorx

    GWPGearWorx

    Joined:
    Apr 29, 2007
    Posts:
    336
    I am a proud license owner for Face FX Studio Professional, and I would honestly say I am blown away by ease of use and how powerful this tool is. Now with the all new unity integration, which makes it even easier to integrate. It makes it a steal of a deal to anyone that has lots of dialog in their games as it saves LOADS of time..

    I am a character animator and I key all of my animations including lip sync and facial expressions, however for the current project there is a lot of dialog and it would take me forever to key frame all of these animations, so in short this saves my life!

    I setup a sample shot for one of my clients for the project I am working on, and the feedback I received was "Why did Oblivion not use this tool, then the face animation would not of sucked so bad" and "Wow!, the end result was perfect and life like. Thanks!"

    The node based facial setup system allows you to blend in facial poses to really bring the characters to life. With very little setup in regards to the face poses in your 3D app of choice (see requirements), you can amp up the performance with ease with the use of this system.

    To clear up one thing that people might be confused about is when you buy Face FX Studio Pro you get the full app and ALL of the evaluation plug-ins (the ones that cost $199) Then when your working in your 3d app of choice you export your actor using the evaluation plug-in, you do not use the evaluation plug-ins to analyze the dialog because they only support english, and since it is evaluation plug it is phrase locked, and then import it into Face FX Studio Pro, where you would analyze the audio with the much more powerful tool set, which also support other languages. Then you would collapse the XML data in Studio and then import it into Unity..... So simple it hurts.

    So in short you use the evaluation plugs to setup your characters facial poses in your 3d app of choice, export the actor to Face FX Studio Pro, analyze the audio in studio, export the collapsed xml data, import into unity... DONE.

    I could go on and on about this tool but I will stop there, because I need to get the rest of these faces rigged, for our project.

    Just my two cents about this product from a very happy Face FX customer.

    Regards;

    Steve
     
  5. ddesmond

    ddesmond

    Joined:
    Feb 9, 2010
    Posts:
    163
    How would this integrate with Blender or Messiah? I use Blender and just purchased Messiah and was wondering if this would be an option.
     
  6. pkid

    pkid

    Joined:
    Jul 10, 2009
    Posts:
    201
    Are there any changes to Unity-FaceFx pipeline for FaceFx studio pro or just this change for the free version of facefx plus the plugins?
     
  7. facefx

    facefx

    Joined:
    Jan 18, 2010
    Posts:
    116
    We don't currently have any support for Blender or Messiah, but Blender has been on my wish list for a while. It's possible that sometime in the future we will have basic bone pose import / export for Blender but I can't promise a date.

    With FaceFX 2012 (we are skipping 2011 to synch up with supported Autodesk versions) due this summer we're going to have a feature to import bone poses directly inside of FaceFX Studio from an Ogre animation, and Ogre does have working exporters for Blender so there should be a basic pipeline for Blender later in the year; I just can't promise when or if there will be native Blender support.

    1. The way XML files are linked up to GameObjects has changed. You used to drop them in the Assets folder, and the actor name variable would be used to find the appropriate GameObject. Now you select the Game Object and select the appropriate option from the Asset menu.
    2. Professional customers would have the option of transferring bone poses inside of FaceFX Studio, or using the bone pose transfer in the Unity integration now.
    3. Professional customers can also scale the GameObjects now by updating the scale parameter in the FaceFX Controller.

    It's on my list to do a new video tutorial for Professional, even though the plugin tutorial covers everything but collapsing the face graph. I won't cover the benefits of working with the Face Graph and using the power of FaceFX Studio Professional (which are extensive), but at least the Unity-specific workflow will be covered.

    Thanks for the endorsement GWPGearWorx. We are happy to get any feedback you have when you have cranked through everything for your project.
     
  8. yogibear

    yogibear

    Joined:
    Jan 10, 2009
    Posts:
    26
    Found that the conversation moved here. Posting the question again: If I buy the FaceFX Plugin for Maya (say), then can I make the facial animation of the character follow a user recorded audio clip? I tried to substitute the first speech of Jake in the Demo with my own audio clip but the animation is still the old animation. What do I do in order to make the facial animation of Jake sync with a new audio clip? Thanks in advance.
     
  9. facefx

    facefx

    Joined:
    Jan 18, 2010
    Posts:
    116
    You need to buy a license to use your own audio. The only exception is the phrase-locked file that the evaluation plugins can analyze. Follow the video tutorial above for instructions on that.
     
  10. facefx

    facefx

    Joined:
    Jan 18, 2010
    Posts:
    116
    FYI, new integration files have been posted. A C# port has been included, and a bug with the ScaleFactor feature was fixed. Visit the demo page for more information.
     
  11. yogibear

    yogibear

    Joined:
    Jan 10, 2009
    Posts:
    26
    I am still not clear. This is what I want to do-
    - I have written a Unity Windows Standalone Application whereby a User selects a male or female character and records his/her voice into a .wav file
    - When the user hits a Play Button after recording his voice for say 10 secs, the recorded audio has to play with the male/female character lip syncing to the audio
    - Using the $199 FaceFX plugin for Maya and FaceFX Studio Free can I achieve the above?
     
  12. yogibear

    yogibear

    Joined:
    Jan 10, 2009
    Posts:
    26
    To make my question clearer, If I buy the Plugin License, can I rig the character so that the analysis of audio is automatic and not manual? The character lip syncs to any audio that I throw at it automatically. Exactly like the Talking Tom App.
     
  13. vreference

    vreference

    Joined:
    Mar 22, 2011
    Posts:
    154
    This is kind of funny - and I realize I'm just one internet moron - but I was actually about to leave a glowing heartfelt response that would indicate how much I appreciated - I literally nearly shed a tear of joy - a company whose business model was so friendly to indy developers after the first bit of the video that seemed to indicate you could use the evaluation version for free. I wrongly assumed that meant you would then purchase a license to create a for-profit release. I guess that's why I felt this perceived betrayal so keenly when I realized it was indeed an extremely limited evaluation version.

    For all I know this is the most fully featured amazing tool available and there is no need to cater to the indy crowd - not to mention the difficulty in policing licensing for releases that originate from countries with minimal or nonexistent copyright law... Still, I would like to register my protest and suggest that there's a reason there is likely no FaceFX support for Unigine: No one uses Unigine because their business model is antiquated and stupid but strangely familiar.
     
    Last edited: May 2, 2011
  14. facefx

    facefx

    Joined:
    Jan 18, 2010
    Posts:
    116
    The $199 plugin can't do what you want, unfortunately. The plugin only works inside of Max/Maya/XSI/MotionBuilder, and your users don't have those animation packages.

    We do let you share a single license with multiple computers, however. So you can share a plugin license with other members of your team. Combined with the fact that a single purchase supports up to 5 different versions of a particular animation package, we hope that we are providing a solid value for the indy developer.
     
  15. yogibear

    yogibear

    Joined:
    Jan 10, 2009
    Posts:
    26
    Actually this is one of the educational games that I am working on for a client. I was under the impression that with the FaceFx plugin, Maya and Unity Pro, I can create a game/application whereby end-users (of the game) can speak into a mic and the 3D character can talk back with lip sync using the recorded voice. Basically a simpler version of Talking Tomcat.

    Apparently this is not possible with the FaceFX plugin as Doug clarified in a separate email.

    Thanks anyway.
     
  16. KITT

    KITT

    Joined:
    Jul 17, 2009
    Posts:
    221
    Hi Doug,

    I am using Facefx plugin for Max and am processing a bunch of audio files - they are in a thick australian accent and I am finding the curves generated are sometimes lagging or off by a bit.

    Would I be best suited to a) break up the paragraphs into smaller chunks and cue several animations rather than one long one b) use phonetic spelling in my txt files or c) does Facefx Studio free allow me to address the slight differences in articulation between regular accents and slight variants i.e a thick australian one?

    Other than that, facefx is greatly speeding up the process of lip sync and facial animation.

    cheers.
     
  17. facefx

    facefx

    Joined:
    Jan 18, 2010
    Posts:
    116
    This can help, but instead of doing that, use audio chunking text tags to analyze in smaller pieces but without having to deal with breaking the file up manually.

    This is probably your best bet. Make sure your text matches what is actually said, not the "proper English" version of what was said.

    There isn't much you can do in Free for this problem. Even with FaceFX Studio Professional, your best bet would be to try the above two options first. There is a "UK English" analysis option in FaceFX Studio Professional that could possibly do a better job, but text is the more important variable.

    If things are still off, you can send the audio and text to support (at) oc3ent.com and we'll take a look to see if anything else is going on.
     
  18. KITT

    KITT

    Joined:
    Jul 17, 2009
    Posts:
    221
    Thanks - I followed the online documentation and the results are much better! cheers!
     
  19. mehware

    mehware

    Joined:
    Nov 19, 2007
    Posts:
    739
    I just bought the plugin for Maya and have been following the tutorial videos.

    I created a character (of myself hehe) using Evolver. I paid the $69 to be able to download it in the maya format.. I thought the fbx would have also come with it.

    I think I set up everything right with my evolver character using the plugin, I was able to play a custom audio file in FaceFX No save to view the animation then i exported it to xml (the facefx generated file from the plugin) in the Free version. The problem is the actual model that needs to be imported in Unity.

    How do I properly export the Evolver character in Maya to fbx? When I tried the polys in unity were all squished and highly messed up.

    EDIT: I was able to export the fbx properly to work in Unity but it exports as a mesh renderer not a skinned mesh renderer (which is unity's animation mesh). Any ideas?
     
    Last edited: May 9, 2011
  20. facefx

    facefx

    Joined:
    Jan 18, 2010
    Posts:
    116
    I've never had any trouble getting the evolver files into Unity. I use the latest FBX plugins to export the FBX files, and I use evolver files with the "facial bone rig" rather than morph targets.
     
  21. mehware

    mehware

    Joined:
    Nov 19, 2007
    Posts:
    739
    I forgot that Unity accepts the .mb file and converts it to a fbx. So I was able to drop the .mb in Unity and use that as the model.

    I have the FaceFX Maya Plugin and I am a bit unclear on going from the Evolver Maya file to a correct FaceFX .facefx format.

    I can run the plugin, add the bone poses, do the batch export, and add the animations but my evolver character in Unity deforms very weird because I think I am missing facial bones like you mentioned in the evolver file.

    I know how to export the xml in FaceFX Studio Free and integrate into Unity, its just the Maya step with the proper files that are confusing me.

    Where do you find the facial bone rig option?

    Thanks!
     
  22. facefx

    facefx

    Joined:
    Jan 18, 2010
    Posts:
    116
    The facial bone rig is an evolver purchase option. The FaceFX controller only supports bones at the moment, so you need to have the facial bone rig if you want to drive facial animations with the FaceFX controller.

    If you are using the "Darwin Default" skeleton and the facial bone rig, you should be able to generate your bone poses by importing the XML file we use in the sample in Unity with the Assets->Import FaceFX XML Actor option. So you don't even have to generate bone poses.
     
  23. mehware

    mehware

    Joined:
    Nov 19, 2007
    Posts:
    739
    Ah that would have made things a lot simpler. I am on evolver and looking all over where I would set Darwin Default skeleton and bone rig option. I didn't see it when i checked out/paid for the evolver character. This is exciting tho that I don't have to really worry about generating any bone poses and that it works with your actor files in Unity. I just need to find out where these download options are.
     
  24. mehware

    mehware

    Joined:
    Nov 19, 2007
    Posts:
    739
    I just noticed the
    Pro – Purchase your Avatar’s 3D Source Data with all these options but what the heck.

    What did I get on evolver doing the FaceFX option checkout at Evolver that I paid $70 for.
     
  25. IonDave

    IonDave

    Joined:
    Sep 27, 2010
    Posts:
    58
    You got the body mesh pre-rigged and skinned, plus a facial setup pre-rigged and skinned for bones. Apparently you can get these rigs "tuned" for various purposes including FaceFX, but you're not paying extra for it. The total cost is divided between the base mesh (body) and the facial rig (which is the extra part).
     
  26. mehware

    mehware

    Joined:
    Nov 19, 2007
    Posts:
    739
    Right there is a Pro – Purchase your Avatar’s 3D Source Data option and a Pre-tuned FaceFX® Avatar option. I did the facefx avatar option for my first one but the source mb file had no facial rig.

    I am trying the first option Avatar 3D Source Data but this character is taking forever to generate seems lost in space.

    - Matthew
     
  27. facefx

    facefx

    Joined:
    Jan 18, 2010
    Posts:
    116
    I think the "pre-tuned" FaceFX avatar just gives you the Ogre files that work in FaceFX Studio. That's not what you want. Get the Maya file, Darwin skeleton, facial bone rig whatever resolutions you want.
     
  28. mehware

    mehware

    Joined:
    Nov 19, 2007
    Posts:
    739
    Thanks, I talked to the people from Evolver. They had an issue with generating the character. I got the files downloaded/

    I am using the FBX. All I have to do is Import the EvolverActorFile.xml on my character and then my custom animations from the maya plugin with Import FaceFX XML Animation ?

    In the maya plugin I don't have to export the bone poses or anything correct because I am using the EvolverActorFile.xml Just generate the animation with audio and text file in the maya plugin. Then load it in FaceFX free to export it to an xml file to run in Unity as the animation right?

    Almost there!

    - Matt
     
  29. facefx

    facefx

    Joined:
    Jan 18, 2010
    Posts:
    116
    That's correct.
     
  30. mehware

    mehware

    Joined:
    Nov 19, 2007
    Posts:
    739
    I got everything to work :D. I wasn't used to the Maya art pipeline I had a scale issue which I must have overlooked even tho it was in the FaceFX Tutorial Video.

    To the Unity community. If you need talking avatars FaceFX is the way to go! You can easily create character's at Evolver.com (another service) and Import them into Unity and get them talking in no time with the FaceFX soultion.

    FaceFX is perfect for my upcoming Unity application that involves a lot of talking Avatars. I highly recommend FaceFX. I am currently using the Maya Plugin to export my custom animations onto my avatar characters. It only costs $199 USD! It's a steal for an indie.

    I am going to pick up FaceFX Studio Professional because it is a much more powerful solution. It lets you analyze audio in different languages, add custom gestures, tweak animations and much more.

    Thank's again to Doug and the rest of the FaceFX team for their outstanding support and Unity integration! These guys are top notch.

    - Matthew
     
  31. KITT

    KITT

    Joined:
    Jul 17, 2009
    Posts:
    221
    - Amen!
     
  32. facefx

    facefx

    Joined:
    Jan 18, 2010
    Posts:
    116
    As mentioned here, Autodesk has recently purchased Evolver.

    What isn't discussed adequately is that for a limited time, Evolver Professional models are being offered for free. I have just logged in and confirmed that this is indeed the case.

    Walk, do not run to evolver.com! In my book, Autodesk rocks for this move!

    BTW, to use your evolver character with FaceFX, our samples are created with the Darwin Default skeleton, Facial Bone rig, male height. So you will have the smoothest time using our sample rig if you have those settings too.

    Stay tuned for FaceFX 2012 with emoticons, Mandarin Chinese Czech support, full body animation preview, and lots more!
     
  33. MikeUpchat

    MikeUpchat

    Joined:
    Sep 24, 2010
    Posts:
    1,056
    Just seen this go up on youtube, Evolver face morphs in Unity. I have megafiers so I think I might try a comparison between bones and morphs.
     
  34. facefx

    facefx

    Joined:
    Jan 18, 2010
    Posts:
    116
    It should be very straightforward to drive the morph targets with FaceFX curves. After all, we were treating "bone poses" exactly like they were morph targets. As an added bonus, the range of expression is better and the morphs are created by evolver automatically. The "gotchas" are that you will still need bone poses to rotate the head and eyes, and the mouth and teeth morphs are separate (as you can see in the video), but this shouldn't be a problem.

    Are you going to try to drive the morphs with FaceFX curves? Let me know if you need any help. Anyone else out there using this system? The FaceFX 2010 sample content used a morph evolver rig and it looked very good.
     
  35. MikeUpchat

    MikeUpchat

    Joined:
    Sep 24, 2010
    Posts:
    1,056
    I will be looking at using facefx to drive the system got the teeth linked up to some of the morphs so doing a Shout for example works nicely now. I just noticed that the author of the morph system has a fully multithreaded and optimized version in the Asset store so will get that later. Seems using morphs makes life a little easier for some things such as easier transfer of animations between evolver characters. I will have a chat with the morph system author and see if can do anything with your output.
    Mike
     
  36. grfxman

    grfxman

    Joined:
    May 28, 2009
    Posts:
    309
    Ok, After reading all the posts I'm a bit confused.

    _1 If I purchase the Maya Plug-in and use Unity Pro, Can I make "my own" audio clips function in games I build?

    _2 Does this also work with Web-Player Unity Games?

    Thanks
     
  37. pkid

    pkid

    Joined:
    Jul 10, 2009
    Posts:
    201
    The facefx site has a unity demo that plays in browser using the unity web player so it must work with web player games.
     
  38. grfxman

    grfxman

    Joined:
    May 28, 2009
    Posts:
    309
    Ok But can I use my own sounds with just the Maya Plug-in and Unity?? :)
     
  39. pkid

    pkid

    Joined:
    Jul 10, 2009
    Posts:
    201
    My understanding is yes, if you buy the $200 plug in or the full $2000 studio package you can use your own sounds.
     
  40. facefx

    facefx

    Joined:
    Jan 18, 2010
    Posts:
    116
    That's right. The evaluation plugin is "phrase-locked", but when you purchase the plugin you can analyze any audio/text you want. The tutorial below shows the phrase-locked pipeline. It would be exactly the same with a license to the plugin, but you could analyze your own files.

    http://facefx.com/content/facefx-evaluation-unity-maya-evolver
     
  41. grfxman

    grfxman

    Joined:
    May 28, 2009
    Posts:
    309
    Thanks for letting me know. I'm changing my avatars so we can use this. :)
     
  42. facefx

    facefx

    Joined:
    Jan 18, 2010
    Posts:
    116
    FYI, new integration files are posted here. I've updated to Unity 3.4 and dropped support for the javascript FaceFXControllerScript (both javascript and C# projects can use the C# controller script). There were a few issues fixed with audio synching, so try the latest controller if you were having problems with that.

    Thanks and let me know if you are running into any issues!
     
  43. facefx

    facefx

    Joined:
    Jan 18, 2010
    Posts:
    116
    FaceFX 2012.1 has been released, which includes the ability for the plugins to save to our XML format. This simplifies the pipeline for customers who have purchased the $199 plugin. There is a new video tutorial posted below.



    The 2012.1 evaluation can be downloaded from:
    http://facefx.com/page/no-save-evaluation
     
    Last edited: Nov 7, 2011
  44. cephaslr

    cephaslr

    Joined:
    Nov 27, 2011
    Posts:
    3
    The new update looks good. I just purchased my copy about two months ago and I would like to take advantage of the new body animations feature. Would you be able to post some steps or refer me to some documentation on how to add body animations to my characters? I am currently using evolver characters and mixamo animations so steps involving those tools would be a plus. Thanks again!
     
  45. mehware

    mehware

    Joined:
    Nov 19, 2007
    Posts:
    739
    I am using evolver and custom mocap for body animations.

    I love FaceFX Pro :)

    I am still on version 2010, Is there way to do facial expressions? Smile, frown, idle (non talking face animation) ?

    - Matt
     
  46. facefx

    facefx

    Joined:
    Jan 18, 2010
    Posts:
    116
    Hi Matt,

    We're glad you like FaceFX! You can definitely animate expressions in FaceFX 2010.

    Expressions are just like lip-synch targets, except the curves aren't generated automatically by the audio so you have to animate them manually. You can do this from the curve editor or using a workspace slider.

    Are you having trouble making the target? For an evolver character with the "facial bone rig", you can create your own bone poses with the Max/Maya plugins. Of course if you like the targets available in the FaceFX Unity Demo sample content and your skeleton is also Darwin Default, you can just transfer the bone poses onto your character using the "bonepose" command or the Unity integration itself like we talk about in the video tutorial.

    For a more complicated "idle" animation or even a longer smile/frown animation that can't be contained in a single pose, you just need to create a new animation in FaceFX Studio. Select "Do not generate phonemes" in 2010 or "Create an empty (blank) animation" in 2012. Then create your animation by hand. Once created, you can easily insert your idle animation or expression animation into another animation from the events tab.

    If that doesn't explain things, let me know if you are using bones vs. morphs and single-pose smile/frown vs. an entire animation animation.

    The FaceFX Team
     
  47. facefx

    facefx

    Joined:
    Jan 18, 2010
    Posts:
    116
    The full-body animation feature in FaceFX Studio is still relatively new. It's main purpose is to preview full body animations while you are working in FaceFX Studio to tweak an audio-based facial animation. FaceFX uses Ogre for its viewport, so you have to export your mixamo mocap animations using Ogre Exporters. The sample content uses an evolver character and mixamo animations so the sample content is a very good reference for you.

    The FaceFX Unity Demo does not currently have any support for playing mocap animations in synch with facefx-generated animations. You can move the body with FaceFX bone poses of course, but that's not the same as playing a mixamo animation on the body and using FaceFX for the head and up. It should be very easy to add an underlying mocap animation in Unity, but it's just not something implemented for you in the sample FaceFX integration scripts. You would just need to follow the Unity animation docs.

    What exactly are you trying to accomplish? Make the character walk and talk at the same time?
     
  48. EricTheCoolDude

    EricTheCoolDude

    Joined:
    Feb 21, 2011
    Posts:
    164
    I wish more game devs would just man up and do their facial animations by hand, or at least try to develop a better system, kind of like Valves faceposer. It does it half-automatically and it looks infinitely better than faceFX, even the 2004 version.

    FaceFX looks like S***.
     
    Last edited: Nov 28, 2011
  49. mehware

    mehware

    Joined:
    Nov 19, 2007
    Posts:
    739
    Thanks FaceFX,

    I'll play with some expressions today. We are basing everything off of evolver and your unity-demo project. We are using Darwin Bone Skeleton and Face Bone Rig exported from Evolver's website.

    I tired doing the "blank" audio file to get idle blinks in which worked, but I'll delve deeper into FaceFX today to play with the editor to make expressions the proper way.

    Thanks for your help.

    - Matt
     
  50. KITT

    KITT

    Joined:
    Jul 17, 2009
    Posts:
    221
    @EricTheCoolDude: It's not my job to defend Facefx software, but it is a very robust solution for interactive facial animation, serious games in game cinematics. And besides, it has nothing to do with "manning up". You wanna try walking into any of these studios and telling them to "man up"?

    I would have to disagree with you and say Facefx IS the S*** ;)