Stencils + Surface shader

Can't get stencils to work with surface shaders. Works just fine with fragment shader.

Trying to create a very simple mask that just writes into stencil, and if value is present then other object wont be rendered.

Code (CSharp):

1. Shader "Custom/StencilWrite" {
2.     Properties {
3.         _Color ("Color", Color) = (1,1,1,1)
4.         _MainTex ("Albedo (RGB)", 2D) = "white" {}
5.         _Glossiness ("Smoothness", Range(0,1)) = 0.5
6.         _Metallic ("Metallic", Range(0,1)) = 0.0
7.     }
8.     SubShader {
9.         Tags { "RenderType"="Opaque" "Queue"="Geometry"}
10.         LOD 200
11.         ZWrite Off
12.  
13.         Stencil
14.         {
15.             Ref 1
16.             Comp always
17.             Pass replace
18.         }
19.         ColorMask 0
20.      
21.         CGPROGRAM
22.         #pragma surface surf Standard noshadow
23.         #pragma target 3.0
24.  
25.         sampler2D _MainTex;
26.  
27.         struct Input {
28.             float2 uv_MainTex;
29.         };
30.  
31.         half _Glossiness;
32.         half _Metallic;
33.         fixed4 _Color;
34.  
35.         void surf (Input IN, inout SurfaceOutputStandard o) {
36.          
37.             fixed4 c = tex2D (_MainTex, IN.uv_MainTex) * _Color;
38.             o.Albedo = c.rgb;
39.          
40.             o.Metallic = _Metallic;
41.             o.Smoothness = _Glossiness;
42.             o.Alpha = c.a;
43.         }
44.         ENDCG
45.     }
46.     FallBack "Diffuse"
47. }
48.  

Code (CSharp):

1. Shader "Custom/StencilRead" {
2.     Properties {
3.         _Color ("Color", Color) = (1,1,1,1)
4.         _MainTex ("Albedo (RGB)", 2D) = "white" {}
5.         _Glossiness ("Smoothness", Range(0,1)) = 0.5
6.         _Metallic ("Metallic", Range(0,1)) = 0.0
7.     }
8.     SubShader {
9.         Tags { "RenderType"="Opaque" "Queue"="Geometry+1" }
10.         LOD 200
11.         Stencil
12.         {
13.             Ref 1
14.             Comp NotEqual
15.             Pass keep
16.         }
17.  
18.         CGPROGRAM
19.         #pragma surface surf Standard fullforwardshadows
20.         #pragma target 3.0
21.  
22.         sampler2D _MainTex;
23.  
24.         struct Input {
25.             float2 uv_MainTex;
26.         };
27.  
28.         half _Glossiness;
29.         half _Metallic;
30.         fixed4 _Color;
31.  
32.         void surf (Input IN, inout SurfaceOutputStandard o) {
33.  
34.             fixed4 c = tex2D (_MainTex, IN.uv_MainTex) * _Color;
35.             o.Albedo = c.rgb;
36.  
37.             o.Metallic = _Metallic;
38.             o.Smoothness = _Glossiness;
39.             o.Alpha = c.a;
40.         }
41.         ENDCG
42.     }
43.     FallBack "Diffuse"
44. }
45.  

This doesn't work. But if I replace surface shader with fragment shader, it does work. What gives?

 

Ok, several issues.

1. Had to remove Fallback from StencilWrite shader.
2. Had to add exclude_path:deferred to StencilRead shader

Code (CSharp):

1. Shader "Custom/StencilWrite" {
2.     Properties {
3.         _Color ("Color", Color) = (1,1,1,1)
4.         _MainTex ("Albedo (RGB)", 2D) = "white" {}
5.         _Glossiness ("Smoothness", Range(0,1)) = 0.5
6.         _Metallic ("Metallic", Range(0,1)) = 0.0
7.     }
8.     SubShader {
9.         Tags { "RenderType"="Opaque" "Queue"="Geometry+1"}
10.         LOD 200
11.  
12.         ZWrite Off
13.         Stencil
14.         {
15.             Ref 1
16.             Comp always
17.             Pass replace
18.         }
19.         ColorMask 0
20.      
21.         Pass{}
22.     }
23. }
24.  

Code (CSharp):

1. Shader "Custom/StencilRead" {
2.     Properties {
3.         _Color ("Color", Color) = (1,1,1,1)
4.         _MainTex ("Albedo (RGB)", 2D) = "white" {}
5.         _Glossiness ("Smoothness", Range(0,1)) = 0.5
6.         _Metallic ("Metallic", Range(0,1)) = 0.0
7.     }
8.     SubShader {
9.         Tags { "RenderType"="Opaque" "Queue"="Geometry+1" }
10.         LOD 200
11.         Stencil
12.         {
13.             Ref 1
14.             Comp NotEqual
15.             Pass keep
16.         }
17.  
18.         CGPROGRAM
19.         #pragma surface surf Standard fullforwardshadows exclude_path:deferred
20.         #pragma target 3.0
21.  
22.         sampler2D _MainTex;
23.  
24.         struct Input {
25.             float2 uv_MainTex;
26.         };
27.  
28.         half _Glossiness;
29.         half _Metallic;
30.         fixed4 _Color;
31.  
32.         void surf (Input IN, inout SurfaceOutputStandard o) {
33.  
34.             fixed4 c = tex2D (_MainTex, IN.uv_MainTex) * _Color;
35.             o.Albedo = c.rgb;
36.  
37.             o.Metallic = _Metallic;
38.             o.Smoothness = _Glossiness;
39.             o.Alpha = c.a;
40.         }
41.         ENDCG
42.     }
43.     FallBack "Diffuse"
44. }
45.  

Now it sort of works, but I have this issue:



https://gyazo.com/daf0e28ff71cf84c2b33962867473878

As you can see, the shadowing on the object behind becomes different when behind the stencil for some reason.

Also in game mode it looks even worse:



https://gyazo.com/cdb345fa45ee697f9e9ac51ea7d64d61

There's some blue stuff where the stencil object is.

 

Disabling shadows for the light "fixes" the issue, and that's what they're doing in this sample project too:
http://forum.unity3d.com/threads/unity-4-2-stencils-for-portal-rendering.191890/

It seems like stencils and shadows simply aren't compatible. What a shame. (Enabling shadows in that other project breaks it as well.)

Resorting to render textures and custom masking instead.

 

So I can kind of explain why this is a problem if you're curious, and suggest a solution or two.

First is Unity doesn't deal with mixing deferred and stencil very well. If you want to do stencil stuff you're most likely going to want to stick to forward rendering. I'll get to why later.

The other problem is Unity is a semi-deferred renderer even when the forward rendering path is enabled. Specifically the primary directional light's shadows are rendered against a depth prepass texture. This is the same texture accessible from _CameraDepthTexture. The depth texture is generated by rendering the camera view using each shader's ShadowCaster pass, and this same shader pass is used for generating the shadow maps.

By default surface shaders, which are actually vertex / fragment shader generators, don't generate a unique shadow caster pass and instead reuse one from the Fallback shader. In fact the common fallback shader of Diffuse doesn't have a shadowcaster pass either, instead it too has a fallback shader listed ... in the end almost every shader uses the same shadow caster pass from the "VertexLit" shader.

What this means the shadow caster pass that gets used by default for the depth texture does not read or write to the stencil, and neither your "stencil write" or "stencil read" shaders ever create the needed hole in the depth texture! Instead the object in the background will get what appears to be the shadows of the object in front of it.

For the deferred rendering path something similar but different is happening. That "blue" is the camera's clear color; it's what the screen blanks to every frame before things are rendered. Stencil doesn't behave nicely with deferred Unity's surface shaders simply ignore any stencil values set for the deferred pass, Unity's official documentation on Stencils mentions that fact. By excluding deferred on your surface shader it's being rendered as a forward pass afterwards allowing it to work. But forward rendered objects are actually still rendered as part of the deferred pass in a weird way. They use that same shadow caster pass mentioned above to write into the deferred depth buffer as well as set the albedo color to black wherever those objects render. The initial gbuffer pass already rendered the ambient lighting for all deferred objects, but then the albedo color used for subsequent lights like the directional light is black so you'll end up only getting ambient lighting on objects visible through the "hole". Also the skybox will only render in areas where the depth buffer is empty.

So now you're left with a hole in the forward rendered objects showing an ambient only lit deferred object behind it and the camera's clear color everywhere else.


So, now the solution, sort of. First off switch to forward rendering. We'll come back to deferred later.
Next we need to add a shadow caster passes to the two shaders. This is easier than it sounds as the stencil write shader we don't actually want to cast shadows so it doesn't need any of the special shadow caster pass shader code, it just needs to know it needs to render, and for that we just need a second pass with "LightMode"="ShadowCaster". You also probably want to disable shadow casting on the stencil writer's MeshRenderer since even though it won't actually render anything it is still being rendered during the shadow map generation.

Code (CSharp):

1. Shader "Custom/StencilWrite" {
2.     Properties {
3.     }
4.     SubShader {
5.         Tags { "RenderType"="Opaque" "Queue"="Geometry+1"}
6.         LOD 200
7.         ZWrite Off
8.         Stencil
9.         {
10.             Ref 1
11.             Comp always
12.             Pass replace
13.         }
14.         ColorMask 0
15.    
16.         Pass{}
17.         Pass{ Tags {"LightMode"="ShadowCaster"} }
18.     }
19. }

The surface shader is even easier, it just needs "addshadow".

Code (CSharp):

1. Shader "Custom/StencilRead" {
2.     Properties {
3.         _Color ("Color", Color) = (1,1,1,1)
4.         _MainTex ("Albedo (RGB)", 2D) = "white" {}
5.         _Glossiness ("Smoothness", Range(0,1)) = 0.5
6.         _Metallic ("Metallic", Range(0,1)) = 0.0
7.     }
8.     SubShader {
9.         Tags { "RenderType"="Opaque" "Queue"="Geometry+1" }
10.         LOD 200
11.         Stencil
12.         {
13.             Ref 1
14.             Comp NotEqual
15.             Pass keep
16.         }
17.         CGPROGRAM
18.         #pragma surface surf Standard fullforwardshadows exclude_path:deferred addshadow
19.         #pragma target 3.0
20.         sampler2D _MainTex;
21.         struct Input {
22.             float2 uv_MainTex;
23.         };
24.         half _Glossiness;
25.         half _Metallic;
26.         fixed4 _Color;
27.         void surf (Input IN, inout SurfaceOutputStandard o) {
28.             fixed4 c = tex2D (_MainTex, IN.uv_MainTex) * _Color;
29.             o.Albedo = c.rgb;
30.             o.Metallic = _Metallic;
31.             o.Smoothness = _Glossiness;
32.             o.Alpha = c.a;
33.         }
34.         ENDCG
35.     }
36.     FallBack "Diffuse"
37. }

And now everything works!

Except if you try to go back to deferred using those shaders it's sadly still broken, but in slightly different way. Now the skybox shows through properly as the stencil masking is actually happening when those two objects are rendered to the depth buffer, but the deferred object behind is now either the clear color (if HDR is disabled) or just ambient lit just like before all of this! I believe what's happening is the fact your shaders are writing to the stencil buffer is interfering with Unity's use of the stencil buffer to mask areas for later lighting. I have no idea why they differ based on the HDR setting though.

So, a final crazy hack to get it working in deferred is you have to re-render any object that's deferred a second time during that gbuffer depth injection pass to reset the stencil back to 255 so the directional light will hit it.

Code (CSharp):

1. Shader "Custom/StencilDeferredReset" {
2.     Properties {
3.     }
4.     SubShader {
5.         Tags { "RenderType"="Opaque" "Queue"="Geometry+2" }
6.         LOD 200
7.         ZWrite Off
8.         Stencil
9.         {
10.             Ref 255
11.             Comp always
12.             Pass replace
13.         }
14.         ColorMask 0
15.    
16.         Pass{}
17.         Pass{ Tags {"LightMode"="ShadowCaster"} }
18.     }
19. }

Easiest way to do that is just add an additional material to an existing object using the above shader! ... or just use the forward rendering path.

 

Thanks for that very detailed explanation and samples. I couldn't quite get it to work though. I changed rendering path to forward, and copy pasted those shaders, and this is the result:

 

Not entirely sure what's happening there. It almost looks like only the in shadow part of the red box is still being rendered in the shadow buffer, but that's not possible (unless your red box is actually separate meshes per face for some reason). Try changing the stencil write's queue to "Geometry" and see if that makes a difference? It shouldn't but it's my only guess for the moment.

 

I replicated the issue. It's caused by the depth pre-pass not respecting the render queue, so the render order for the depth won't match the scene render! In my own test case I always had the stencil write object closer to the camera than the stencil read which caused it to render in the correct order in the depth pre-pass as well. This is rather unfortunate as there doesn't seem to be a way to affect that order in any way, either for forward's depth prepass or deferred's depth injection.

 

Regardless, thank you so much for the help. It's great this community has such knowledgeable users that are willing to spend time on sharing their wisdom. Thanks again.

 

Some super hacks; if you're still using 5.3 you can disable the "screen space shadows" that are causing the issue for the forward rendering path by going to Edit->Project Settings->Graphics, changing your inspector window to "debug" by right clicking on the tab, expand "shader settings" and disable screen space shadows there.

This disables the shadows being done as a pre-pass which is the cause of this pain. I actually found it to be a minor perf win for VR, but that might not apply to you, and it also disabled shadow cascades. In 5.4 the screen space shadows are easier to disable, but last I checked doing so doesn't actually make it revert to using native shadows like it does in 5.3 so it's no better than not having shadows at all.

Also if you're not doing VR you could move and scale the stencil write mask object closer to the camera every time it's rendered. It'll look exactly the same but ensure the mask is always rendered first in both the depth and the camera. Use the OnPreCull delegate and save of the original world position, scale, and rotation, move it closer, then OnPostRender move it back. Shouldn't interfere with any other script that's likely running in Update or LateUpdate. If you don't want to deal with the math of scaling and moving it while keeping it looking the same on screen you can add a child object on the camera that you temporarily reparent the mask to and scale down, then scale back up afterward and unparent from. It would be easy to move the object via a shader, but that won't solve the render order issue.

I also suggest you check out the Window->Frame Debugger. You can step through the rendering and see the order things are happening fairly clearly. That's how I worked through a lot of this. Stencils can't be previewed in the Frame Debugger, but you can see when the stencil write object gets incorrectly rendered after the stencil read object during that depth prepass.

 

Could you please upload the project files? I'm having a hard time setting up your samples. And I really need some basic stencils on Deferred rendering.

 

The short version is the deferred path doesn't really work. In some very, very carefully crafted scenarios you can make it look like it's working, but it's a pretty major hack.

Also in 5.4 it appears to be even more broken as the "StencilDeferredReset" isn't working as expected 100% of the time and I have no explanation for it beyond something appears to be broken on Unity's side.

 

@bgolus and other shader gurus...

I'm trying to adapt the shader above to work with transparency but have run into an issue.
First, I altered RenderType to "Transparent" and added #pragma alpha

This looks like it works, at least in the editor. I have a checkerboard pattern of solid and transparent sq, but when I actually run the game it only renders pixels on front of other objects. It looks like the skybox is drawn on top of everything. Any ideas?

Edit: The material queue was wrong, see comments in https://answers.unity.com/questions/1203991/skybox-not-correctly-alpha-blending-with-custom-sh.html

 

Everybody, is there any way to blur the edges?

 

Not using stencils, no. Stencils are by their very nature a binary thing. With some work you might be able to dither the stencil mask which may produce a somewhat softer appearance, but for most cases to produce a blurred version of the mesh you want to use as a mask means you already have a screen space texture that could be used to do the masking without stencils.

 

Although I still don’t understand it, thank you for giving me the research direction.