See through shaders that work with the deferred renderer and Speedtrees

I present to thee my newest invention: flat transparent see through shaders that make sure characters are visible through geometry and walls. Works with the deferred renderer, and speedtrees.

The Implementation is using the stencil buffer and additional transparent shaders both for writing and reading from the stencil buffer, and drawing the transparent overlays.

Basic idea behind it:
1) Every character or object that should be visible through walls needs to have an additional material(just add one to the existing renderer).

2) Every occluding object needs an additional material. If the object is static, and static batching is active, additional materials will get purged by the batching, so you need to duplicate the geometry (which can then be batched again). Speedtrees need duplicate geometry because of the way the SRT Files are build (multiple meshes).

3) I have added a float to the shaders that can modulate the brightness of the colors. If you are using a day/night cycle, and the overlay colors look to bright during the night, write a script that feeds a float to the shaders and make sure to lerp the float so it gets smaller during the darker hours.


I'll give you some gifs to show what the shaders do and the shader code... feel free to copy the code if you like it, and feel just as free to give me some feedback on how I could improve the shaders. They are the work of a horribly noobish beginner when it comes to shader coding, so most probably there are a ton of things that could be improved.

WARNING: This shaders are NOT free when it comes to performance. Altough I haven't really seen any FPS drop in my test scene, I hardly did any benchmarking with more objects in the level.
Each object that should be visible through walls needs an additional onepass shader, adding one drawcall to mostly non-batchable and non-instanciatable objects.
Each occluding object needs an 1-3 pass shader (Depending on how many different bits you want to use from the stencil buffer), which adds up to 3 additional drawcalls per object.
Then there is the duplicated geometry if you want to use speedtree trees or batching.


Speedtree Shader in action


Normal transparent shader, for all 3 used stencil bits (neutral used for fuel barrels)





So, shader code....

OccludedObject.shader: this is the shader that writes to the stencil buffer. That is all it does really. I did add a frag program and just returned a transparent pixel. guess that could be cut.
Use any bit you want. In my case, I used bit 1 for friendlies, bit 2 for enemys, bit 4 for neutral objects.

Code (csharp):

1.  
2. Shader "Unlit/OccludedObject"
3. {
4.    Properties{
5.      _StencilBit("Stencil Bit", int) = 1
6.    }
7.  
8.    SubShader
9.    {
10.      Tags
11.    {
12.      "Queue" = "Transparent"
13.      "IgnoreProjector" = "True"
14.      "RenderType" = "Transparent+10"
15.    }
16.  
17.      Cull Off
18.      Lighting Off
19.      ZWrite On
20.      Blend One OneMinusSrcAlpha
21.  
22.      Pass
23.    {
24.      Stencil
25.    {
26.      Ref [_StencilBit]
27.      WriteMask[_StencilBit]
28.      Comp Always
29.      ZFail Replace
30.      Pass Replace
31.    }
32.  
33.      CGPROGRAM
34. #pragma vertex vert
35. #pragma fragment frag
36. #pragma exclude_path:deferred
37. #include "UnityCG.cginc"
38.  
39.      struct appdata_t
40.    {
41.      float4 vertex  : POSITION;
42.      float4 color  : COLOR;
43.    };
44.  
45.    struct v2f
46.    {
47.      float4 vertex  : SV_POSITION;
48.    };
49.  
50.    fixed4 _Color;
51.    fixed _AlphaCutoff;
52.  
53.    v2f vert(appdata_t IN)
54.    {
55.      v2f OUT;
56.      OUT.vertex = mul(UNITY_MATRIX_MVP, IN.vertex);
57.  
58.      return OUT;
59.    }
60.  
61.    fixed4 frag(v2f IN) : SV_Target
62.    {
63.      return half4(0,0,0,0);
64.    }
65.      ENDCG
66.    }
67.    }
68. }
69.  

OcludedObjectHighlighting.shader: This shader needs to be put on the occluding objects. It reads from the different stencil bits, and simply draws a flat color where the objects with the OccludedObject shader put something in the corresponding stencil buffer bit.
I use 3 bits from the stencil buffer. If you do not need 3 different colors, you could cut the corresponding passes.
LightIntensity is just multiplied with the color to darken it.

Code (csharp):

1.  
2. Shader "Unlit/OccludedObjectHighlighting"
3. {
4.    Properties
5.    {
6.      _OccludedColorPlayer("Occluded Tint Player", Color) = (0, 0, 0, 0.5)
7.      _OccludedColorEnemy("Occluded Tint Enemy", Color) = (0, 0, 0, 0.5)
8.      _OccludedColorNeutral("Occluded Tint Neutral", Color) = (0, 0, 0, 0.5)
9.      _StencilBitPlayer("Stencil Bit Player", int) = 1
10.      _StencilBitEnemy("Stencil Bit Enemy", int) = 2
11.      _StencilBitNeutral("Stencil Bit Neutral", int) = 4
12.      _LightIntensity("Light Intensity", float) = 1
13.    }
14.  
15.      CGINCLUDE
16.  
17.      struct appdata_t
18.    {
19.      float4 vertex  : POSITION;
20.      float4 color  : COLOR;
21.      float2 texcoord : TEXCOORD0;
22.    };
23.  
24.    struct v2f
25.    {
26.      float4 vertex  : SV_POSITION;
27.      fixed4 color : COLOR;
28.      half2 texcoord  : TEXCOORD0;
29.    };
30.  
31.  
32.    fixed4 _Color;
33.  
34.  
35.    v2f vert(appdata_t IN)
36.    {
37.      v2f OUT;
38.      OUT.vertex = mul(UNITY_MATRIX_MVP, IN.vertex);
39.      OUT.texcoord = IN.texcoord;
40.    
41.      return OUT;
42.    }
43.  
44.    ENDCG
45.  
46.  
47.  
48.      SubShader
49.    {
50.      Tags
51.    {
52.      "Queue" = "Transparent"
53.      "IgnoreProjector" = "True"
54.      "RenderType" = "Transparent+10"
55.    }
56.  
57.      Cull Back
58.      Lighting Off
59.      ZWrite Off
60.      Ztest Equal
61.      Blend One OneMinusSrcAlpha
62.  
63.      // occluded pixel passes. Anything rendered here is behind an occluder
64.      Pass
65.      {
66.  
67.        Stencil
68.        {
69.          Ref [_StencilBitPlayer]
70.          ReadMask [_StencilBitPlayer]
71.          Comp Equal
72.          ZFail Keep
73.        }
74.  
75.        CGPROGRAM
76.        #pragma vertex vert
77.        #pragma fragment frag
78.        #pragma exclude_path:deferred
79.        #include "UnityCG.cginc"
80.  
81.        fixed4 _OccludedColorPlayer;
82.        float _LightIntensity;
83.  
84.        fixed4 frag(v2f IN) : SV_Target
85.        {
86.          fixed4 color = _OccludedColorPlayer;
87.          color.rgb *= _LightIntensity;
88.          return color;
89.        }
90.        ENDCG
91.      }
92.  
93.      Pass
94.      {
95.  
96.        Stencil
97.        {
98.          Ref[_StencilBitEnemy]
99.          ReadMask[_StencilBitEnemy]
100.          Comp Equal
101.          ZFail Keep
102.        }
103.  
104.        CGPROGRAM
105.        #pragma vertex vert
106.        #pragma fragment frag
107.        #pragma exclude_path:deferred
108.        #include "UnityCG.cginc"
109.  
110.        fixed4 _OccludedColorEnemy;
111.        float _LightIntensity;
112.  
113.        fixed4 frag(v2f IN) : SV_Target
114.        {
115.          fixed4 color = _OccludedColorEnemy;
116.          color.rgb *= _LightIntensity;
117.          return color;
118.        }
119.        ENDCG
120.      }
121.  
122.      Pass
123.      {
124.  
125.        Stencil
126.        {
127.          Ref[_StencilBitNeutral]
128.          ReadMask[_StencilBitNeutral]
129.          Comp Equal
130.          ZFail Keep
131.        }
132.  
133.        CGPROGRAM
134.        #pragma vertex vert
135.        #pragma fragment frag
136.        #pragma exclude_path:deferred
137.        #include "UnityCG.cginc"
138.  
139.        fixed4 _OccludedColorNeutral;
140.        float _LightIntensity;
141.  
142.        fixed4 frag(v2f IN) : SV_Target
143.        {
144.          fixed4 color = _OccludedColorNeutral;
145.          color.rgb *= _LightIntensity;
146.          return color;
147.        }
148.        ENDCG
149.      }
150.    }
151. }
152.  

SpeedtreeOccludedObjectHighlighting.shader: this is the same as the last shader, this time for the SpeedTrees. Most complex of the bunch... used a speedtree shader someone else posted on this forum and kitbashed it with my existing shaders.
Make sure you pick the same settings for cull and wind as you do for your normal speedtree shaders, so the vertex animations line up.

Code (csharp):

1.  
2. Shader "Unlit/SpeedTree Occluded Object Highlighting"
3. {
4.    Properties
5.    {
6.      _OccludedColorPlayer("Occluded Tint Player", Color) = (0, 0, 0, 0.5)
7.      _OccludedColorEnemy("Occluded Tint Enemy", Color) = (0, 0, 0, 0.5)
8.      _OccludedColorNeutral("Occluded Tint Neutral", Color) = (0, 0, 0, 0.5)
9.      _StencilBitPlayer("Stencil Bit Player", int) = 1
10.      _StencilBitEnemy("Stencil Bit Enemy", int) = 2
11.      _StencilBitNeutral("Stencil Bit Neutral", int) = 4
12.      _Cutoff("Alpha Cutoff", Range(0,1)) = 0.333
13.      _MainTex("Base (RGB) Trans (A)", 2D) = "white" {}
14.      _LightIntensity("Light Intensity", float) = 1
15.      [MaterialEnum(Off,0,Front,1,Back,2)] _Cull("Cull", Int) = 2
16.      [MaterialEnum(None,0,Fastest,1,Fast,2,Better,3,Best,4,Palm,5)] _WindQuality("Wind Quality", Range(0,5)) = 0
17.    }
18.  
19.      // targeting SM4.0+
20.      SubShader
21.    {
22.      Tags
23.      {
24.        "Queue" = "Transparent"
25.        "IgnoreProjector" = "True"
26.        "RenderType" = "Transparent+10"
27.        "DisableBatching" = "LODFading"
28.      }
29.      LOD 400
30.      Cull [_Cull]
31.      Lighting Off
32.      ZWrite Off
33.      Ztest Equal
34.      Blend SrcAlpha OneMinusSrcAlpha
35.  
36.        // occluded pixel passes. Anything rendered here is behind an occluder
37.        Pass
38.        {
39.  
40.          Stencil
41.          {
42.            Ref[_StencilBitPlayer]
43.            ReadMask[_StencilBitPlayer]
44.            Comp Equal
45.          }
46.  
47.          CGPROGRAM
48.          #pragma vertex vert
49.          #pragma fragment frag
50.          #pragma multi_compile __ LOD_FADE_PERCENTAGE LOD_FADE_CROSSFADE
51.          #pragma shader_feature GEOM_TYPE_BRANCH GEOM_TYPE_BRANCH_DETAIL GEOM_TYPE_BRANCH_BLEND GEOM_TYPE_FROND GEOM_TYPE_LEAF GEOM_TYPE_FACING_LEAF GEOM_TYPE_MESH
52.          #define ENABLE_WIND
53.          #include "UnityCG.cginc"
54.          #include "SpeedTreeVertex.cginc"
55.  
56.          #if defined(GEOM_TYPE_FROND) || defined(GEOM_TYPE_LEAF) || defined(GEOM_TYPE_FACING_LEAF)
57.          #define SPEEDTREE_ALPHATEST
58.          uniform fixed _Cutoff;
59.          #endif
60.  
61.          fixed4 _OccludedColorPlayer;
62.          sampler2D _MainTex;
63.          float _LightIntensity;
64.  
65.          struct Input
66.          {
67.            fixed4 color;
68.          };
69.  
70.          struct v2f
71.          {
72.            float4 vertex   : SV_POSITION;
73.            #ifdef SPEEDTREE_ALPHATEST
74.            half2 uv : TEXCOORD1;
75.            #endif
76.          };
77.  
78.          v2f vert(SpeedTreeVB v)
79.          {
80.            v2f o;
81.            #ifdef SPEEDTREE_ALPHATEST
82.            o.uv = v.texcoord.xy;
83.            #endif
84.            OffsetSpeedTreeVertex(v, unity_LODFade.x);
85.            o.vertex = mul(UNITY_MATRIX_MVP, v.vertex);
86.            return o;
87.          }
88.  
89.          fixed4 frag(v2f i) : SV_Target
90.          {
91.            #ifdef SPEEDTREE_ALPHATEST
92.            clip(tex2D(_MainTex, i.uv).a - _Cutoff);
93.            #endif
94.            
95.            fixed4 color = _OccludedColorPlayer;
96.            color.rgb *= _LightIntensity;
97.            return color;
98.          }
99.          ENDCG
100.        }
101.  
102.        Pass
103.        {
104.  
105.          Stencil
106.          {
107.            Ref[_StencilBitEnemy]
108.            ReadMask[_StencilBitEnemy]
109.            Comp Equal
110.            ZFail Keep
111.          }
112.  
113.          CGPROGRAM
114.          #pragma vertex vert
115.          #pragma fragment frag
116.          #pragma multi_compile __ LOD_FADE_PERCENTAGE LOD_FADE_CROSSFADE
117.          #pragma shader_feature GEOM_TYPE_BRANCH GEOM_TYPE_BRANCH_DETAIL GEOM_TYPE_BRANCH_BLEND GEOM_TYPE_FROND GEOM_TYPE_LEAF GEOM_TYPE_FACING_LEAF GEOM_TYPE_MESH
118.          #define ENABLE_WIND
119.          #include "UnityCG.cginc"
120.          #include "SpeedTreeVertex.cginc"
121.  
122.          #if defined(GEOM_TYPE_FROND) || defined(GEOM_TYPE_LEAF) || defined(GEOM_TYPE_FACING_LEAF)
123.          #define SPEEDTREE_ALPHATEST
124.          uniform fixed _Cutoff;
125.          #endif
126.  
127.          fixed4 _OccludedColorEnemy;
128.          sampler2D _MainTex;
129.          float _LightIntensity;
130.  
131.          struct Input
132.          {
133.            fixed4 color;
134.          };
135.  
136.          struct v2f
137.          {
138.            float4 vertex   : SV_POSITION;
139.            #ifdef SPEEDTREE_ALPHATEST
140.            half2 uv : TEXCOORD1;
141.            #endif
142.          };
143.  
144.          v2f vert(SpeedTreeVB v)
145.          {
146.            v2f o;
147.            #ifdef SPEEDTREE_ALPHATEST
148.            o.uv = v.texcoord.xy;
149.            #endif
150.            OffsetSpeedTreeVertex(v, unity_LODFade.x);
151.            o.vertex = mul(UNITY_MATRIX_MVP, v.vertex);
152.            return o;
153.          }
154.  
155.          fixed4 frag(v2f i) : SV_Target
156.          {
157.            #ifdef SPEEDTREE_ALPHATEST
158.            clip(tex2D(_MainTex, i.uv).a - _Cutoff);
159.            #endif
160.  
161.            fixed4 color = _OccludedColorEnemy;
162.            color.rgb *= _LightIntensity;
163.            return color;
164.          }
165.          ENDCG
166.        }
167.  
168.        Pass
169.        {
170.  
171.          Stencil
172.          {
173.            Ref[_StencilBitNeutral]
174.            ReadMask[_StencilBitNeutral]
175.            Comp Equal
176.            ZFail Keep
177.          }
178.  
179.          CGPROGRAM
180.          #pragma vertex vert
181.          #pragma fragment frag
182.          #pragma multi_compile __ LOD_FADE_PERCENTAGE LOD_FADE_CROSSFADE
183.          #pragma shader_feature GEOM_TYPE_BRANCH GEOM_TYPE_BRANCH_DETAIL GEOM_TYPE_BRANCH_BLEND GEOM_TYPE_FROND GEOM_TYPE_LEAF GEOM_TYPE_FACING_LEAF GEOM_TYPE_MESH
184.          #define ENABLE_WIND
185.          #include "UnityCG.cginc"
186.          #include "SpeedTreeVertex.cginc"
187.  
188.          #if defined(GEOM_TYPE_FROND) || defined(GEOM_TYPE_LEAF) || defined(GEOM_TYPE_FACING_LEAF)
189.          #define SPEEDTREE_ALPHATEST
190.          uniform fixed _Cutoff;
191.          #endif
192.  
193.          fixed4 _OccludedColorNeutral;
194.          sampler2D _MainTex;
195.          float _LightIntensity;
196.  
197.          struct Input
198.          {
199.            fixed4 color;
200.          };
201.  
202.          struct v2f
203.          {
204.            float4 vertex   : SV_POSITION;
205.            #ifdef SPEEDTREE_ALPHATEST
206.            half2 uv : TEXCOORD1;
207.            #endif
208.          };
209.  
210.          v2f vert(SpeedTreeVB v)
211.          {
212.            v2f o;
213.            #ifdef SPEEDTREE_ALPHATEST
214.            o.uv = v.texcoord.xy;
215.            #endif
216.            OffsetSpeedTreeVertex(v, unity_LODFade.x);
217.            o.vertex = mul(UNITY_MATRIX_MVP, v.vertex);
218.            return o;
219.          }
220.  
221.          fixed4 frag(v2f i) : SV_Target
222.          {
223.            #ifdef SPEEDTREE_ALPHATEST
224.            clip(tex2D(_MainTex, i.uv).a - _Cutoff);
225.            #endif
226.  
227.            fixed4 color = _OccludedColorNeutral;
228.            color.rgb *= _LightIntensity;
229.            return color;
230.          }
231.          ENDCG
232.        }
233.    }
234.  
235.    CustomEditor "SpeedTreeMaterialInspector"
236. }
237.  

If you use those shaders with the deferred renderer, don't forget to put this line somewhere in your code:

Code (csharp):

1.  
2. cameraObj.GetComponent<Camera>().clearStencilAfterLightingPass = true;
3.  

It makes sure you have all 8 bits of the stencil buffer to play with once the deferred part of the pipeline is done.


Things left to improve, or to try out (might do it given I find the time):

1) Instancing: see if instancing is an option for these shaders, at least for the occluding geometry (speed trees for example).

2) Replace the Stencil Buffer with RenderTargets: Maybe by going with an additional RenderTargets, the whole process could be moved back into the deferred shaders. This might save a lot of draw calls, and the need for duplicated geometry.

3) Cut out the overlapping parts from the stencil buffer bit when an occluded object is farther away than another occluded object, but they are overlapping from the current view direction.
Don't know how to do it, but would be mighty cool.


Now, I embarked on this knowing that I as a total shader noob would most probably bang my head against the wall for hours... but surely this would be a simple task. What could go wrong?

Turns out a lot. I am using the dererred renderer for not getting hit as hard by degrading FPS while using tons of lights. Now, given that I wanted to get flat colored shillouettes to show through occluding geometry, the stencil buffer soon sounded like the most obvious choice for my little project.
But that same stencil buffer is now (ab-)used for the deferred renderer. 3 top bits used for internal stuff, up to 4 for different light exclusion layers. Then you cannot read from the stencil buffer in deferred shaders. Couldn't get writing to it to work consistently, so in the end I guessed it would be better to move all stencil operations to the forward/tranparent parts of the pipeline.
Seems I was right.

Then I had to learn a lot about batching, and how it treats additional materials on the objects it batches (they are thrown away it seems).