Skinned morph targets / blend shapes in Unity

Hi,

While working on our next game, the company I work for (Omek Interactive) needed to use blend shapes / morph targets in our animation system. Unity lacks support for them, and the user-posted solutions in the forums don't have support for doing so in skinned meshes, and they don't have support for animating these values. They also don't support meshes with multiple submeshes.

We implemented a system that supports :
- Blend shapes for skinned targets (only skinned targets for now)
- The blend weights can be animated or controlled manually
- Support for meshes with multiple submeshes.
- Fast import pipeline from Maya.

Runtime system :
The attached unity package contains everything you need to use the system.
There are three ways to use it :
1) Manual mode : Attach the "Skinned Morph Targets" script to the object that should be animated, and populate the list of morph targets with the morph meshes.
2) Semi-automatic mode : (This relies on the pipeline) - For an object exported after running our maya scripts (see the attached Cube.fbx/mb), click the object on the scene and go to Morph->Prepare Maya Blend Shapes. It will add the script component and link the submeshes automatically.
3) Fully-automatic mode : (Also relies on the pipeline) - Add the MayaMorphPreparer script to an empty gameobject, and set the object as the Target Object. The "Link Animations" flag toggles whether animations will be controlled automatically or not.

Demo :
The demo attached as part of the unity package contains method #2, but all three are possible in it.
The two animations that are valid in this demo are RotAnim and TransAnim, which demonstrate that the blend shapes are compatible with different types of animations.
If you disable the "Blend Weight Animation Connector" script on the cube object, you will be able to manually control the blend shapes in the demo.

Animation import system :
The attached MayaPipeline.zip file contains our solution to the Maya->Unity bridge. It automatically creates meshes for each blend shape and nodes containing the blend weight for the animations. To use it, run the two functions createMorphTargetMeshes and createMorphTargetLocationAnimations with the designated object selected, and it will create all the data needed in Maya. You can then use the normal .FBX exported to create the FBX. I attached the cube from the demo (with its blend shapes and animations) in the zip file as well so you can test on that. The Cube.fbx from the unity package is just the cube.mb after both functions have been applied to it and it was exported to Maya.
This means that the export process takes just 10 seconds longer than usual and you get full blend shape animations. Notice that if you play the animation in Maya and in the demo you get the exact same visual result - true WYSIWYG.


We chose to use the BSD license for this contribution to the community, I don't think there should be any problems with it. (If there are, feel free to comment about it).

Of course, you can modify the runtime/export scripts to fit your needs. I didn't explain all the naming conventions, but the code is documented well enough to understand.

Enjoy!

 

Sounds really amazing ! I'll try it soon.

 

Sounds awesome!!!, hopefully a kind Unity user will also port this to Blender soon too.

For Maya users, this rocks Noman.!!

Time to do some frame-rate tests now ;-)

 

I did write this with FPS in mind, and big parts of the process are optimized (mainly understand exactly what has to be morphed and not morphing anything besides that). This will always have an FPS hit when comparing to bone animation, because it can not be skinned in shaders on the GPU (I believe that this is the reason that Unity doesn't support vertex animation)
There are some unity operations which I believe cause buffer copies behind the scenes, and perhaps some Vector3[] fetches should be done once and stored in local buffers instead of retrieved from the mesh each frame.

Obviously the only place worth optimizing is SkinnedMorphTargets.cs. Performance comments about that are welcome.

As for porting to blender, the exported data is like this :
For a blend shape / morph target called ABC, you need to create a mesh called BlendShape_ABC and an object (in Maya we used locators, I don't know what the parallel thing in blender is, basically a node) called BlendWeight_ABC.
BlendShape_ABC needs to contain the mesh baked into the blend shape position,
BlendWeight_ABC needs to be animated, and have its local Y translation be between 0 and 1, stating the weight of the blend shape each frame of the animatino.
You then export the entire thing (Mesh + BlendShape_XXX + BlendWeight_XXX) as one FBX, and you're good to go.
I think that this is enough information to write a version of the export script for other software.

 

Any plans to support 3DS Max with this? Maya is great and all but if you have been using Max for your whole life then you would prefer to keep using it.

 

Sounds good, look forward to seeing some examples done with this system

 

See my previous reply. To support 3DS max (or any modeling tool) all you need to do is to create a script that prepares a model for FBX exporting like I did in the .mel script in MayaPipeline.zip. The entire maya script is around 40 lines and doesn't do that much (clones meshes, bakes morph targets into them, and creates dummy nodes with their animated weights), so it should be relatively easy to write parallel ones for other software packages.

Have a look at my previous post, the runtime code and the sample FBX that comes with the demo, and it shouldn't take more than a few hours (at the most) to port the pipeline code to other packages.

 

@Noman, thanks for the detailed description of how to possibly port this to Blender as well. If I get a working Python script for Blender, I'll try to post it here(as Matrix also mentioned for 3DS Max). Again awesome work, and thank you for sharing!!!

 

This forum rocks, and so does Noman.

 

sounds good!

 

Hey Noman,
Since you understand the workings behind morphing, do you know anything about lattices?

 

I never used lattices, but from reading about them on the internet it sounds like baking them into new meshes (like blend shapes) is possible, so it can be done.

By the way, the fact that I use "Prepare Maya Blend Shapes" and "MayaMorphPreparer" doesn't mean that you have to use FBX's exported from Maya, but that I rely on the BlendShape_XXX and BlendWeight_XXX conventions, so don't be alarmed by those classes' names.

 

Baking them kinda loses the point of FFD or lattices.

 

I need to learn some scripting...i use XSI in my pipeline, and my projects need morph targets for the hero models... i could probably just constrain the bones to the controllers... a la MGS 4

 

I'd be willing to pay a bounty for anyone who ports this to Blender, 2.5 preferably. I'd do it, but my Python-Savvy is very very rusty now.

 

Hi there, total amateur here, this looks like a really great tool...definitely thanks for sharing with the community. However, I can't seem to get it working? I tried the cube.mb and loaded up the mel script, executed the script with the cube selected....then exported as fbx, but the result is totally different to the fbx in the package and seems like nothing happened?

Sorry to be a bother but did I miss a step somewhere along the line? Would be much appreciated if you could give me some pointers thanks!

 

Hi, Nomad!

I tryied to make a similar script last month, but there was a trouble:
- during import .fbx files, unity creates mesh with different number of vertices, depending on how are the tangents and normals set. This makes impossible to build proper blendshapes for hipoly meshes, or for meshes where morphs builds sharp edges.

How do your pipeline deals with that kind of trouble?

(sorry for my english, i'm from Russia)

 

Paper - I'll try to put up a small tutorial (w/screenshots) on how to do the exporting. For one of the scripts (the one that creates the blend shapes) you need to select the main object before running the function.
Also, running the .mel script doesn't execute the two functions, it just declares them. You have to run them (I created a menu button for each one)

vtornik23 - We encountered the problem as well. There is a paragraph somewhere in the unity docs (forgot where, sorry) explaning what unity does when it imports an fbx file and what can cause the vertex counts to differ between the modeling software and the unity runtime. You can use that to understand why you're getting different poly counts.

 

Thanks Noman, that would be greatly appreciated! I kinda figured that was what I was doing wrong...I couldn't find the menu buttons though...maybe I'm not looking in the right place...sorry I'm not that familiar with maya

 

So, if I paid a programmer, would they potentially be able to use this code and get a character with morph target animation on it's face working in Unity? or is that a long way off?

I would like to export morph-target animation but I don't know any programming. Should I switch to another game engine? Are there any that can do it?

 

Ste74 check the wiki and search the forums, there are a few morph solutions available for Unity.

 

Thanks SpookyCat, I hope I'm not getting on your nerves asking about this with no programming knowledge. I trawled the forums and the wiki for days before posting questions. It seems like it's so nearly there, that I'm confused why everybody's not really excited and jumping all over it?

I couldn't find any example/ mention of a skinned character with morph targets working. The examples I have seen of morphs in Unity are of very basic geometric shapes bending and curving with not much mixing between different target shapes apart from your face test, which is very impressive - and I was trying to find out if there are limitations in this technique that prohibit it being used on a skinned character, exporting facial animation baked into the fbx, before I spend money on hiring a programmer to try and then finding it was always unachievable. thx

 

hey Norman

i was spending some time with a friend programmer trying to achieve that and your solution working really well, i did some simple tests but for now its working perfectly

the solution of exporting the blend shape animation in the fbx is awesome, it will save me a lot of time, thanks a lot! =]

 

Hey Noman,

We've had success with your mel script, but there are a few things that we cannot solve which prevent us from successfully getting the morphs animated in Unity. The Different Blendshapes created from the mel script are exporting via FBX with no problems (aside from the vertex count issue mentioned before). We are having some difficulty getting the MorphDemoGUI to control the BlendWeights animation.

When the complete FBX is imported into Unity, the BlendWeight animation always imports as "Take 001". When adding :Take 001" to the MorphDemoGUI, the morphs do not animate during playback. We've tried exporting (via FBX) the BlendWeights locators separately and using that animation file on the Skinned object. We receive no errors when doing this, but the morph Blend Weighting is not effected. Also our BlendWeight animation also imports always as "Take 001" and we cannot separate multiple blend animations on import as shown in your cube.fbx in the Unity package file.

Unless someone else downloading your scripts has had success and can show an example, I suspect that you have used Motionbuilder or another animation program to modify the cube.mb beyond the maya default animation settings. Any light you could shed on the locator export or the way that the locators need to be rigged into your scripts would be a big help.

BTW - Thanks for releasing these tools, it should have already been included in Unity as a standard feature. Maybe this will give them a kick start!

 

@Tvidotto

Glad to hear you are using the script and its working!

@tinyutopia

We exported both animations that were made with MotionBuilder and with maya.
We used the @animName convention for animations though. (Each animation is a different .mb file, called modelName@animationName.fbx).
Perhaps if you try to go by a different route (one big animation + splitting in unity's import setting) the script might have to be slightly different.
Did you try to see if manual control (moving the sliders manually in the demo / controlling the values from the inspector) is working?

 

Ok, that's what we thought. Thanks for clearing that up.

yes, if we use the manual control, the morphs work fine. At this point we're only unable to get the locator animation connected to the Blend Weights in Unity so we can playback the morphs via the blend shape keys from maya.

 

has anyone ported this to Blender? ShapeKey support would be GREAT!

 

@zhx

I don't think so. The runtime scripts can stay the same, just the export scripts need to be rewritten. You can have a look at the sample cube that is part of the demo to see how the scripts expect the data to be packed, and write a blender script to modify the model pre-exporting to fit that data set.

@tinyutopia

I see. I don't really know how to help you. Are the BlendWeight_ANIMNAME transforms appearing in the imported mesh in unity? Can you see if they are animated in the animation view in unity?

 

This was the issue. You guys must use a different unit scale, because after we adjusted the locators to go to y0 - y100 instead of y0 - y1, we could see the results of the animation playback. Thanks for your help Noman, and for this useful tool.

 

Ah yes, we had a similar problem too. We ended up adding a small 'scaleFactor' parameter to the locator animation connector, and set it to the inverse of the "scale on import" setting in the model's import settings in unity (so, if we used 0.01 scale, we set the scale factor to 100).

(This is the correct way to solve it, as it will allow you to use the same script on models with different import settings, as long as you set the value accordingly)

I might update the unity package with our latest (and greatest) updates to the script.

 

We've encountered a bug in the script. It's necessary for us to import any FBX files with normals set to "Import" and tangents set to "None" in order for all of the FBX meshes to import into Unity with the same vertex count.

When the Skinned Morph Targets script is setup using these meshes and an animated rig, the shader which is assigned to the object renders completely black at runtime. We're sure that the bug is in the Skinned Morph Targets script, because when you turn off the script the shader renders correctly. Likewise if tangents is changes to calculate the shader renders correctly, but then as the meshes have different vert counts the morph targets do not work.

As the cube example included does not appear to have bone animation or a shader, you may have not encountered this yet. We're including a link to the package file which has our animated, skinned morph example for debugging.

Here is a direct link to the package file

 

I've noticed that all the morphs are applied as cummulitive. That is to say, if I have a morph for blinking and one for mouth open, then if I apply both @ 100%, each will cancel the other out and the final result will be 50% of each.

Is there a way to change the morph method to additive, so that you can get the full effect of each morph when applied in combination?

 

We also noticed that in Maya, morph animations are additive, and modified the script to work in that fashion. But I didn't update the package on this site yet.
I'll post an updated package in the coming days.

I'm not sure I'll be able to have a look at your problematic package. Might install unity at home to check it out...

 

@Noman, looking forward to the additive package update . Sent you a PM too

 

I don't have time to prepare a full unity package with the demos and all, but here are the latest versions of the script. It should be possible to plug and play them with the current unity package.

 

Thanks for the update.

I am getting a few errors on MorphDemoGUI.cs when these new scripts are updated.

Assets/My Assets/Scripts/MorphDemoGUI.cs(20,70): error CS1061: Type `SkinnedMorphTargets' does not contain a definition for `neutralWeight' and no extension method `neutralWeight' of type `SkinnedMorphTargets' could be found (are you missing a using directive or an assembly reference?)

Assets/My Assets/Scripts/MorphDemoGUI.cs(20,13): error CS1502: The best overloaded method match for `UnityEngine.GUI.HorizontalSlider(UnityEngine.Rect, float, float, float)' has some invalid arguments

Assets/My Assets/Scripts/MorphDemoGUI.cs(20,13): error CS1503: Argument `#2' cannot convert `object' expression to type `float'

 

Naturally, with additive blending, there is no 'neutral pose weight', so the variable is gone, so the line that is displaying it in the demo GUI should be removed as well (line 20, according to your compile errors)

 

Hello,

I've imported a fbx (see attached file) in the project, but it doesn't work after the original cube was replaced by the other. The fbx have been created with 3D Studio Max. Is the model not compatible with this project ?

 

I don't have Maya, so I can't test Norman's script out, but would it be possible to use this script to morph between two meshes while the first mesh was also animating?

 

@Noman, thanks again. This script is sooo very useful, and IMO a feature that is puzzling that Unity does already natively support.

if anyone reading this thread has the skill-set to port this script to Blender(2.5x) ".blend" files, please PM me. I'm willing to compensate said programmer for such an effort. Our team would really like to use this on our current iOS Unity project, though we use a Blender 2.5x pipeline.

I dont mean to spam, just really hoping for a Blender compatible version of such an awesome script.

 

I am willing to make a similar offer for all the Maxscripters out there!!


Lic

 

Total noob here to Unity, but so far it's amazing.

@[Norman], thanks for this great piece of work, but I'm having a little trouble getting method (2) to work. I cannot find "Morph->Prepare Maya Blend Shapes" anywhere. I have this somewhat working by running the scripts to the model in maya, then adding my object to the unity scene and run the "Maya Morph Preparer" script, but the "Skinned Morph Targets (script)" is only visible when I run the game, then it dissapears.

I have added the (3) updated scripts to the project so I could try to adjust the scale if the inspector didn't dissapear. I guess this method is runtime only?

Thanks in advance

pats

 

Still struggling with this. While trying to use method (2), I can't find "Morph->Prepare Maya Blend Shapes".

Any help would be appreciated.

Thanks

pats

 

I got the demo to work, and was very excited - nice work, Norman!

That said, I started having trouble getting this to work when I tried this myself from scratch. In my test case, I created a sphere in Maya, and then made 3 basic blend shapes from copies of the sphere.

The setup phase seemed to go okay, but I am getting an error in Unity that tells me that my base mesh and blend shapes don't have the same number of vertices. I double-checked in Maya, and all of my spheres have 58 verts.

Is anyone else having this problem, and if so, do they have any suggestions?

Thanks!

 

Hi,

First of all, thank you Noman for the providing these scripts, they are very useful to those of us who need to use facial animation in their projects. Or rather, they will be very useful once I get them to work for me.

Here is my situation - I have a character model (originally modeled in 3ds Max, now imported into Maya) with a bunch of facial and body animations. The model is pretty complex - it consists of multiple submeshes (the largest of which, the face mesh, has close to 6000 vertices), and four of the submeshes have morph targets. There is a total of about 30 morph targets per each submesh.

For now, I export the model with *one* morph target to FBX (using Maya 2011 with FBX 2011), import it into Unity, and then initialize the morph target animation manually (i.e. by adding SkinnedMorphTargets script to the model and setting the morphTargets array). However, when I try to animate the model (manually, by setting the blend weight for my single morph target), the mesh deforms incorrectly (it just fractures in a weird way).

My guess is that vertex order is the problem - i.e. I suspect vertex order in the morph target mesh and/or base mesh changed at some point in the art pipeline and are no longer the same in the two meshes. However, I have no idea where or why this would happen, or why it happens for this particular model and not for the test models like Cube.fbx. If anyone could shed some light on this issue (and hopefully propose a solution), I would be most grateful.

I should note that I also tried to export the model directly from 3ds Max and into Unity, and the same problem occurred, indicating that the cause of the problem does not lie between Max and Maya. I should also note that I made some minor changes to Noman's scripts, as they don't play very well with multiple submeshes. I don't see how any of my changes would have resulted in this problem, though.

Thank you,

Milord

 

Make sure that the vertex scale factor is set correctly, otherwise I have witnessed this sometimes results in the mesh "fracturing" like you said.

 

Are you referring to the mesh scale factor option of the FBXImporter? I had actually set that to 0.01, but even after setting it back to 1, mesh fracturing issue persists.

 

Update: I can confirm that vertices are reordered in imported meshes, and I can also confirm that vertex ordering is correct in the source FBX file. Hence, it is Unity that changes the vertex order on import. The question is - why would Unity do that, and is there a way to stop it from doing that?

 

Hi,

Unity does quite a bit when importing meshes, and we can't control all of it.Are you using my maya script to generate the blend target meshes or doing it on your own?

Also, it might be a matter of vertex duplication that it is doing. Are the vertex counts of the imported meshes equals? If not, try setting the general tangents/normals to automatic, with the angle threshold for vertex duplication being 180 degrees (ie no duplication).

 

Hi Noman,

I have experimented with different combinations. I have exported both with meshes created using your script and those prepared on my own. I have also tried using computed normals with angle threshold set to 180°, to no effect. Indeed, it seems that Unity does some optimizations on the mesh that are beyond user control.

There don't appear to be any elegant solutions to this issue. I suppose one workaround to try would be to parse and load meshes manually (e.g. from OBJ files). Not a very efficient solution, but it would be better than nothing.

Thanks,

Milord

EDIT:

Upon more thorough investigation I have determined that some vertex duplication occurs regardless of import settings. Both the base mesh and morph target mesh have 5832 vertices in Maya. This vertex count remains unchanged upon export to FBX. However, some duplication occurs after mesh import into Unity. Specifically, the number of vertices increases to 5960 for both the base mesh and morph target, respectively. That seems to be the case for both imported normals and calculated normals with 180° smoothing angle. Lowering the threshold increases the vertex count, and it becomes different for the two meshes.

However, even though vertex count is the same for 180° smoothing angle, mesh topology is still different and morphing causes the mesh to fracture.