Transparency blended into Toon

I am working on a public art installation with a client where they require characters to subtly shift from a wireframe representation to a toon shaded appearance. I have found multiple posts on this forum about rendering wireframes using mesh filters and some opengl trickery, but it was my idea from the beginning to simply use a texture (which may or may not match the mesh of the character vertex for vertex; ie: imagine a low-poly mesh on a high-poly model, or vice versa).

I figure the best way to accomplish this was to combine the Particles/Additive shader and the Toon/Basic shader in some way utilizing the Blend2Textures.shader on unify: http://www.unifycommunity.com/wiki/index.php?title=Blend_2_Textures. This would allow me to write a script that adjusts the blend value based on the characters distance from the camera. Simple enough, right?

However, the trick comes in fading in the Toon pass over the Transparency pass. Why use separate passes? The Transparency pass requires culling and zwrite to be turned off so you can see the wireframe (or whatever it might be) on the other side of the object. The Toon pass you obviously want culling and zwrite on so that it doesn't look weird. It also has to be blended somehow, either with a specific Blend mode, or with some alpha adjustment within SetTexture, which I think will ultimately be the way to go. I just don't know what that is.

The image attached shows the progression from "wireframe" to toon. The additive effect of the translucent toon character with culling off will hopefully be eliminated.

TLDR:
(1) How do I apply a normalized value as a multiplier to my toon shader's alpha?
(2) How do I do that while keeping my culling/zwrite different?

And of course, here is the code:

Code (csharp):

1.  
2. Shader "Toon/Blended" {
3.  
4.     Properties {
5.         _Blend ("Blend", Range (0, 1) ) = 0.5
6.         _TintColor ("Tint Color", Color) = (0.5,0.5,0.5,0.5)
7.         _MainTex ("Underlying Texture", 2D) = "white" {}
8.         _Color ("Toon Tint Color", Color) = (.5,.5,.5,1)
9.         _MainToonTex ("Main Toon Texture", 2D) = "white" {}
10.         _ToonShade ("ToonShader Cubemap(RGB)", CUBE) = "white" { Texgen CubeNormal }
11.     }
12.    
13.     SubShader {
14.        
15.         Pass {
16.             Blend One One
17.             AlphaTest Greater .01
18.             Cull Off Lighting Off ZWrite Off Fog { Color (0,0,0,0) }
19.    
20.             BindChannels {
21.                 Bind "Color", color
22.                 Bind "Vertex", vertex
23.                 Bind "TexCoord", texcoord
24.             }
25.             SetTexture [_MainTex] {
26.                 constantColor [_TintColor]
27.                 combine constant * primary
28.             }
29.             SetTexture [_MainTex] {
30.                 combine texture * previous DOUBLE
31.             }
32.         }
33.        
34.         Pass {
35.             Blend One One
36.             ColorMask RGBA
37.             ZWrite On
38.            
39.             SetTexture [_MainToonTex] {
40.                 constantColor [_Color]
41.                 combine texture * constant
42.             }
43.             SetTexture [_ToonShade] {
44.                 combine texture * previous DOUBLE, previous
45.             }
46.         }
47.        
48.     }
49.    
50. }
51.  
52.  

 

Ok, I figured out how to apply the transparency value using the constantColor command, but I don't know how to apply both the color selected in the inspector as well as the alpha value.

Code (csharp):

1.  
2. SetTexture [_MainToonTex] {
3.     //constantColor [_Color]
4.     constantColor (0, 0.5, 1, [_Blend])
5.     combine texture * constant
6. }
7.  

Ideally I would like the transparent blending effect to work most similarly to the Alpha VertexLit with Z shader on the unify wiki: http://www.unifycommunity.com/wiki/index.php?title=AlphaVertexLitZ, so that there is no additive effect from the mesh behind it.... but that is a separate issue.

 

If you can post a unitypackage, I'll have a look at it. What exactly is the type of blending you want? If the lines are white, then additive blending works perfectly and is easy. What about for the toon version, though? You do not want additive blending on that, but instead, alpha blending, right? And I don't see a need for vertex colors. Are you just including that because of copy/paste from the particles shader, or are you really utilizing them?

 

The transparent wireframe texture (basically the underlying layer) just uses additive blending. I do want to have the option of adjusting its alpha/color with a tint.

The toon shader I want to be alpha blended, simply fading in on top of the underlying transparency layer.

I am able to apply the _Blend value when I specify the constantColor myself in the shader, but I want to be able to use the color from the inspector combined with the alpha value from the _Blend property.

Sorry, I don't know how to make a unity package.

 

The idea is to transition from wireframe to toon. That is what the _Blend property is for, so it can be accessed from a script. The rest of the Blend functionality within each pass is just stuff I've been trying out in the hopes of getting something to work the way I expect it.

If you can suggest a different route for going accomplishing this, I would love to hear it.

 

I realize that. You're okay with that blending mode? Unless it's white, then it's going to pick up whatever was already rendered.

What does that mean?

Why aren't you just using the alpha of the color instead of the Range value then?

http://unity3d.com/support/documentation/Manual/HOWTO-exportpackage.html

And again, what about my vertex colors question?

 

Well, probably not in the long run. I'll want to be able to apply different tints and alpha value to that texture.

I'm just referring to the texture that will appear when the _Blend value is set to 0. I have been thinking of it as the underlying layer because it is the first pass.

Because clearly I don't know what I'm doing. I figured it would be easier to modify a range value from a script than the color value. That's probably far from the truth, but I am being forced to learn this stuff as I go about trying to accomplish very specific goals.

I don't know what you're referring to. What is the vertex color portion of the shader?

 

Code (csharp):

1. BindChannels {...}

...

Code (csharp):

1. combine constant * primary

I have a definite solution for you using four passes, but I think it can be done in three. I don't have anything good to test with, though, so I need you to make that package.

 

I actually did a little more research and managed to get it working the way I like it (so far, though I can still imagine improving it quite a bit from what Jessy has mentioned). I have removed the vertex color bit, but had to keep the BindChannels call in place with the bing color bit. I also removed the _Blend property in favor of using the toon color's alpha value.

Code (csharp):

1.  
2.  
3. Shader "Toon/Blended" {
4.  
5.     Properties {
6.         _TintColor ("Tint Color", Color) = (0.5,0.5,0.5,0.5)
7.         _MainTex ("Underlying Texture", 2D) = "white" {}
8.         _Color ("Toon Tint Color", Color) = (.5,.5,.5,1)
9.         _MainToonTex ("Main Toon Texture", 2D) = "white" {}
10.         _ToonShade ("ToonShader Cubemap(RGB)", CUBE) = "white" { Texgen CubeNormal }
11.     }
12.    
13.     SubShader {
14.        
15.         Pass {
16.             Blend One One
17.             AlphaTest Greater .01
18.             Cull Off Lighting Off ZWrite Off Fog { Color (0,0,0,0) }
19.    
20.             BindChannels {
21.                 Bind "Color", color
22.             }
23.             SetTexture [_MainTex] {
24.                 constantColor [_TintColor]
25.                 combine constant * primary
26.             }
27.             SetTexture [_MainTex] {
28.                 combine texture * previous DOUBLE
29.             }
30.         }
31.        
32.         Pass {
33.             Blend SrcAlpha OneMinusSrcAlpha
34.             ColorMask RGBA
35.             ZWrite On
36.            
37.             SetTexture [_MainToonTex] {
38.                 constantColor [_Color]
39.                 combine texture * constant
40.             }
41.             SetTexture [_ToonShade] {
42.                 combine texture * previous DOUBLE, previous
43.             }
44.         }
45.        
46.     }
47.    
48. }
49.  

And here is a simple C# script for controlling the alpha value based on the objects distance from the camera:

Code (csharp):

1.  
2.  
3. using UnityEngine;
4. using System.Collections;
5.  
6. public class BlendShaderScript : MonoBehaviour {
7.    
8.     int fullDistance = 2;
9.     int opaqueRange = 3;
10.     Color toonColor;
11.     float alpha = 1;
12.  
13.     void Start () {
14.    
15.     }
16.    
17.     void Update () {
18.         float dist = Vector3.Distance(Camera.main.transform.position, transform.position);
19.         if(dist > opaqueRange  dist < opaqueRange+fullDistance){
20.             alpha = 1 - ((dist-opaqueRange) / fullDistance);
21.         } else  if (dist < opaqueRange){
22.             alpha = 1;
23.         } else if (dist > opaqueRange+fullDistance){
24.             alpha = 0;
25.         }
26.        
27.         for(int i=0; i<renderer.materials.Length; i++){
28.             toonColor = renderer.materials[i].GetColor("_Color");
29.             toonColor.a = alpha;
30.             renderer.materials[i].SetColor("_Color", toonColor);
31.         }
32.     }
33. }
34.  

Right now I'm SUPER happy with this, but here's a list of things that could be improved:
(1) The first pass should apply a tint color/alpha value.
(2) The first pass should not be additive so it can utilize more than just white.
(3) The toon effect should be written to the buffer before being blended, similar to this transparency shader: http://www.unifycommunity.com/wiki/index.php?title=AlphaVertexLitZ

BTW, I made a package of this but it's 80mb. I started working from within the Locomotion_System project while I was developing it, so there's a lot more than is necessary for viewing the effect.

 

I haven't read the last post yet. Just the model and textures are all I need (I know I can get that model but I don't feel like using the bandwidth for that whole tutorial just for the model). Feel free to downsample the texture to like 128 first if you want.

Edit:

What? The shader you have now shouldn't even be able to render the first pass. If you don't bind the vertex positions then the pass is useless. You don't need a BindChannels block, but unless you account for primary being black, then yeah, it will be broken if you don't change them to white, with the binding.

 

I was reading more about this just now and further simplified the code, which now also allows the first pass to be tinted properly:

Code (csharp):

1.  
2. Shader "Toon/Blended" {
3.  
4.     Properties {
5.         _TintColor ("Tint Color", Color) = (0.5,0.5,0.5,0.5)
6.         _MainTex ("Underlying Texture", 2D) = "white" {}
7.         _Color ("Toon Tint Color", Color) = (.5,.5,.5,1)
8.         _MainToonTex ("Main Toon Texture", 2D) = "white" {}
9.         _ToonShade ("ToonShader Cubemap(RGB)", CUBE) = "white" { Texgen CubeNormal }
10.     }
11.    
12.     SubShader {
13.        
14.         Pass {
15.             Blend One One
16.             AlphaTest Greater .01
17.             Cull Off Lighting Off ZWrite Off Fog { Color (0,0,0,0) }
18.    
19.             SetTexture [_MainTex] {
20.                 constantColor [_TintColor]
21.                 combine constant
22.             }
23.             SetTexture [_MainTex] {
24.                 combine texture * previous DOUBLE
25.             }
26.         }
27.        
28.         Pass {
29.             Blend SrcAlpha OneMinusSrcAlpha
30.             ColorMask RGBA
31.             ZWrite On
32.            
33.             SetTexture [_MainToonTex] {
34.                 constantColor [_Color]
35.                 combine texture * constant
36.             }
37.             SetTexture [_ToonShade] {
38.                 combine texture * previous DOUBLE, previous
39.             }
40.         }
41.        
42.     }
43.    
44. }
45.  

I'll try simplifying everything in a new project for packaging.

 

I made a new project with all of the hero assets removed (that model and it's animations is 135mb alone). I instead have made a new scene with a sphere using the same shader (not as interesting, but hey, it's there). If you use the arrow keys to move the sphere closer to the camera, it turns from the wireframe to the toon shader.

 

Problem with testing with a sphere is lack of concavity. I've got characters lying around, but a sphere is no good! 8)

 

Yeah, I realized that as soon as I made the post. :?
I tried taking a hint from that AlphaVertexLitZ shader and adding a pass in the middle with the ColorMask 0 call, but it didn't seem to make a difference at the time.

 

you should add this to the wiki, probably so that someone can see it and maybe add a version that blends to diffuse.

 

I prefer the results of alpha testing to any kind of blending, for the wireframe. So that's what I've done here. Regardless of if you want to change that, you should be putting your wireframe texture in the alpha channel of the toon texture. I also optimized/cleaned up your script, but you have to set up your variables before you hit play.

There are some other lines I might add, in the way of ZWrites, AlphaTests, and ZTests, if I knew they were actually going to have a useful effect, but in my experience, that requires testing on the target hardware. I'd most likely split this up into two materials, if I were doing this on an iOS/Android device, too, to avoid alpha testing and blending in the same render queue.

 

Really not sure what I'm doing wrong, but for some reason it just appears to show the first texture, with no transparency. When I run away from the camera it blends into an emissive wireframe tint color, with no toon effect. I'm also seeing bits of the wireframe tint color revealed underneath the wireframe texture as the character mesh is moved around.

I set the material property to the material used on my character, the transform property to my character's transform (which the script is applied to), and the camera to the main camera transform.

 

You just dragged the wireframe texture onto the Main Toon Texture (A = Wireframe) slot. Won't work. I just uploaded a new version of the shader using the label Wireframe Color instead of Wireframe Tint Color, because that's what it is.

Now that I think about it, not only did you not supply a non-wireframe texture in your package, but your screenshots don't show any. Do you even want a UV-mapped texture??

 

There is a wire-tile texture that I've been simply repeating across it, because we do not have a nicely unwrapped high-res UV texture yet (we are working on finding a plugin for 3DS Max to do this, but haven't found the perfect solution yet). Eventually we will have a low-poly generic model of a human figure with a wireframe appearance which will be used to transition into more unique characters. We're working right now on creating the morph target animations to accompany this shader.

I'm also currently not using a texture for the underlying toon shader, but rather a default white and relying on the constantColor to tint it. The art direction simply calls for a two-tone cel shaded effect.

 

How are you planning on getting these animations into Unity?

You don't need to tint white; white is 1, and "tinting" is multiplying. Hence what I did with the wireframe. You ought to fix the last pass so that it isn't a jumble of clutter for your purposes.

The wireframe still needs to be an alpha channel though; put it in something else's alpha channel so you don't waste the RGB channels or waste memory on an 8 bit alpha.

 

As FBX's, I guess. I am not handling the modeling/animating, just the scripting/shaders/interaction.

I don't understand why I would add the wireframe to the alpha channel of the toon texture... I just don't get how that would work...

 

That's not a plan, then. Unity doesn't support morph targets; only bones.

It won't work, because as you've said, there is no toon texture - I made the shader assuming you'd actually need one, because your shader included a variable with the name "Main Toon Texture". But look at what you're actually doing, first. Your texture defines where the wires are, not what color they are. With that in mind, look at what I'm doing in the first pass. I'm just rendering a solid color wherever there is white in the texture, with a variable to control the edges. You can't do that unless you're using an alpha for the wires, or Cg/GLSL, which I don't know. (But pure ShaderLab seems ample for your needs.)

So that's why you need it there. To see "how it would work", just do it. I tested it thoroughly. Put white in the RGB channels and the wire cross that you supplied in the A channel. Then see if you like the results and edit the shader accordingly so your last pass isn't full of nonsense. Post if you need help with that, of course.

 

We're not doing it in unity. We're doing it in max and capturing the morph as an animation to be played back. That is a complete aside and has nothing to do with the issue at hand... just letting you know the context I'm working in.

When I do that, I get a predominantly black character (with no transparency) with the Toon Tint Color coming through where the white lines are. When I move the character out of range it fades to purely Wireframe Color (with no transparency).

I tried what you suggested earlier, and made a PNG where there is all alpha except the lines, and that at least produces a transparency effect. However (1) the lines are very chunky, but I guess that just means adjusting the wireframe cutoff, (2) the character continues to display weird wireframe type transparency even though it is fully transitioned into toon shader mode, and (3) There continues to be underlying Wireframe Color coming out at certain points of the mesh.

This is what I'm seeing:

 

This is what it looks like when I use the white cross on black background that I was using originally. Notice the weird bits of white coming through the mesh as if it was being intersected by a mesh underneath it:

 

No, what you're describing is what happens if you use the texture you supplied. That is, white cross on black in RGB and white in A. (Using RGB for a texture makes the alpha channel white in ShaderLab)

If the slider can't get what you need, make the lines thinner in the texture, UV map differently, or take out the AlphaTest and do alpha blending instead. But as you know, you need ZWrite off in that case.

That's not my shader. I can see the animal's arm through its body. Mine doesn't do that.

And that black one isn't mine either. Do you still have the shader you made in the project? You can't have two shaders with the same name. I just used the name you gave because I figured you wanted it.

 

Sorry, but it is. I renamed your shader BlendedNew so I could switch between the two. I have checked repeatedly, and it is the most recent shader you uploaded that is presented on the 4 images I just posted.

 

You've got nothing to be sorry about; I don't consider it my shader. The only way it could be the code that I offered is if it's not behaving as intended. You mentioned that ColorMask 0 wasn't helping you before. Maybe it's a bug in your graphics driver, or Unity is broken on your OS (maybe you're on Windows and it's a DirectX issue?). I can't help you if that's the way the shader actually works for you; that can only be the result of a bug.

Here's what it looks like on my end. I edited the shader as I recommended for you to do, to eliminate a texture stage in the last pass. No difference other than a tiny performance improvement and clutter removal, if you had used a white texture before.

 

Here are screenshots with alpha at zero and 1. Also, here's a unitypackage that contains all you need, to see it, except the cubemap you already supplied, and any texture with the wireframe texture in the alpha channel.

 

Here are two workarounds for the middle pass. Just replace that pass with one of these...

Code (csharp):

1. Pass {ColorMask A}

...if it works, and...

Code (csharp):

1. Pass {Blend Zero One}

if not. (In theory the first one is faster.)

Let us know if either fixes things. Then make sure you have the latest graphics drivers. If you do, and my original shader is still broken for you, please report a bug.

 

When I open the package, make a sphere and camera, set up the material and script, then test it out... The blending works, the transparency of the wireframe works, and all the coloring works. I believe even the concavity issue regarding seeing other parts of the character show through while transitioning has been resolved (EDIT: actually, no, I can still see stuff through it). However there is still the issue of a weird wireframe effect being "masked" on top of the toon shader pass, which is most obvious when the alpha is set to 1.

I am on Windows 7, running Unity Pro 2.6.1f3, with a Gigabyte HD5870 Eyefinity x6.

 

I tried both of those instead of ColorMask 0 and didn't notice any difference. We should have the latest driver for this card as it was installed last week.

 

Okay, it took me a little while to figure it out. You have two issues. The first is that the second pass has been ignored. That causes the problem in the middle. The next problem is that your mesh is not writing to the depth buffer in the last pass. Combined with the first problem, it allows you to see the wireframe any time you have overlapping frontfaces with areas of different color, from the tooncube.

Here is a last attempt. All I'm doing here is trying to override the problem with ZWrite On in the last pass. It's not necessary on my computer, but it may be on yours. If it doesn't work, it's a serious Unity bug. (It's a bug either way because it's supposed to be ZWriting by default.) Again, change the middle pass to Blend Zero One as a last resort. There's definitely a Unity bug for the middle pass.

I'm not optimistic that this shader will make any difference. Explicitly using ZWrite On should be useless, as it's supposed to be the default. And if neither of those middle pass changes worked for you before, at alpha=1, then they won't now. You can at least take comfort in the fact that UT can make it work – I have no issues here. A bug report would probably get it fixed faster.

And lastly, although your results are consistent with the mesh not writing to the depth buffer, it may not be true. I expect that you'll get the results shown below, with the scene in this package. Let me know if that is true. I can't tell from your other screenshots, because you're using the Game view instead of the Scene View, which doesn't allow us to look at the grid, which is rendered last.

 

Unfortunately I've gone home for the weekend and won't have access to that particular machine until Monday. I can test the code on my home rig, but it has completely different components (though the same version of Unity, so it's worth a shot).

Thank you so much for the time you've put into this so far. It's been invaluable in increasing my knowledge and furthering the project.

 

Unfortunately, on my home system (Windows XP x64, Dual GeForce 9800GT's in SLI, and Unity Pro 2.6.1f3) I still see the middle pass issue occurring. I have included a screencap, and tried to include a video but the forum doesn't seem to like my MP4.

 

I'm getting the same results on the workstation at the studio. Can anyone else confirm/deny this bug?

 

You mentioned that you achieved the wireframe shader "using mesh filters and opengl trickery". I couldn't find any info on this that has worked for me. could you please elaborate on how you achieved this effect?

thank you