Outline on a skinned mesh with some flat normals

Hi,

I'm trying to get an outline on a skinned robot mesh, my starting point is this type of outlining : http://wiki.unity3d.com/index.php/Silhouette-Outlined_Diffuse

The problem with this method is that it can't work properly on meshes with discontinous normals (= not fully smoothed)
If the mesh has some flat areas, it means the vertices are duplicated along edges two have 2 different normals.
So when the outline shader inflate the mesh, the triangles are moving appart from each others instead of staying joined.


But what I want is this :


In this second case, I have stored the smoothed normals in the vertex colors, to keep the flat normals intact for rendering.
With an undeformed mesh, this is perfectly working.

But when I try to animate my mesh with bones and use a SkinnedMeshRenderer, I get this result :


The vertex normals are properly modified according to the bones rotations, but the vertex colors are (of course) not affected the same way.

Here is my current modification on the original shader, it's quite simple :

Code (csharp):

1.  
2. v2f vert(appdata v)
3. {
4.     v2f o;
5.     o.pos = mul(UNITY_MATRIX_MVP, v.vertex);
6.  
7.     // modification starts here
8.     float3 col;
9.     col.xyz = (1-v.color.rgb) * 2 - 1;
10.     float3 norm = mul((float3x3)UNITY_MATRIX_IT_MV, col.xyz);
11.     // modification ends here
12.  
13.     float2 offset = TransformViewToProjection(norm.xy);
14.  
15.     o.pos.xy += offset * o.pos.z * _Outline;
16.     o.color = _OutlineColor;
17.  
18.     return o;
19. }
20.  

So my questions are :
1) how could I modify my shader to get the same transformation on vertex colors and vertex normals according to the skinning transformations ?
2) if this is not possible, what would be a better approach to get the right result ?

Thanks ^_^

 

The easiest way to do this is have two models, your main one and a second with smoothed normals to use for outlines. If the outlines will always be on you can have this be one model with two materials to simplify things a little.

The more advanced way to deal with this is to encode the smoothed normals into the model's tangents. The normals and tangents are the only thing that are updated by skinning. You have to do this in Unity using a script to either modify the mesh on import or make a copy. If you do this however you can't use normal maps.

Another method is to use smoothed normals and use a normal map to get the appearance of hard edges.

 

Hi, this is my second option yes, I'm planning to use it as a last resort.

I don't use normalmaps, but I still need tangents for another purpose... But this is good to know anyway, thanks

I've tried this option yesterday, but didn't found how to produce the right normal map.
I've created a map with flat shading on the mesh, but when I apply it to the smoothed one, the mesh surface is still somehow smoothed in a strange way, looking like some kind of reflections :

Smoothed mesh :


And with the map :


The result is :


Since I have to use a 16bits uncompressed normal maps to get a clean result (even if wrong), I think I'll switch to option 1, and create two different meshes by hand.

But if you have an explanation on how to generate the right normal map to get the expected result, I'm still interested

Thanks for your help !

 

There are plenty of tutorials on normal mapping out there on how to get the best results with this kind of geometry. Generally it's about nicely laid out UVs, careful cage generation, and sometimes "just use xnormal with mikktspace" which helps. It's also possible your normal map has the green channel flipped for what Unity wants.

If you need tangents (and normal mapping) it is actually possible to use the tangent encoding still. You can actually calculate the tangents in the pixel shader itself.
http://www.thetenthplanet.de/archives/1180
Naughty Dog does something similar in their games since Uncharted 2 or 3, as do the latest Assassin's Creed games.

But honestly just having two meshes is the easiest way.

 

Ok, thanks for the details

I'll go with the two meshes solution

 

Judging the color of your texture Your normal map is in world space. You need to bake a tangent space map. These look more blueish.

 

Yep I know.
I tried it too, but I still get the same kind of weird artifacts

Anyway, the normal mapped solution is way too heavy in my case.
If I have some time, I'll try what bgolus proposed ealier : put the secondray normals in the tangent buffer, and compute the tangents directly in the shader, and post it here.

But for now, I'll stick on the two meshes solution.

Thanks for your advice anyway