Using the Stencil buffer in a post-fx

Hi all,

I'm trying to do a graphical effect where a couple of objects in the scene write to the stencil buffer, and then a post-fx reads the stencil buffer to do a full-screen effect.

The part where the objects write to the stencil buffer is definitely working.

Let's say I just want to write green pixels where the stencil buffer is set. Here's my shader:

Code (csharp):

1.  
2. Shader "Green" {
3.     SubShader {
4.         Tags { "RenderType"="Opaque" "Queue"="Geometry+1"}
5.         Pass {
6.             Stencil {
7.                 Ref 1
8.                 Comp equal
9.                 Pass keep
10.                 ZFail keep
11.             }
12.  
13.             CGPROGRAM
14.             #pragma vertex vert
15.             #pragma fragment frag
16.             struct appdata {
17.                 float4 vertex : POSITION;
18.             };
19.             struct v2f {
20.                 float4 pos : SV_POSITION;
21.             };
22.             v2f vert(appdata v) {
23.                 v2f o;
24.                 o.pos = mul(UNITY_MATRIX_MVP, v.vertex);
25.                 return o;
26.             }
27.             half4 frag(v2f i) : COLOR {
28.                 return half4(0,1,0,1);
29.             }
30.             ENDCG
31.         }
32.     }
33. }

If I have my OnRenderImage function blitting straight from source to dest using this shader, everything works as expected - I get solid green on screen only where I wrote to the stencil:

Code (csharp):

1.  
2.     void OnRenderImage (RenderTexture source, RenderTexture dest)
3.     {
4.         Graphics.Blit (source, dest, greenMat);
5.          }
6.  

But, if I try to use an intermediate buffer, it seems to ignore the Stencil section of the pass, and just write green to every pixel:

Code (csharp):

1.  
2.     void OnRenderImage (RenderTexture source, RenderTexture dest)
3.     {
4.         RenderTexture buffer = RenderTexture.GetTemporary(source.width, source.height, 0);
5.  
6.         Graphics.Blit (source, buffer, greenMat);
7.         Graphics.Blit(buffer, dest);
8.  
9.          }
10.  

This just results in a completely green screen.

I've noticed that there's a RenderTexture.SupportsStencil method, which returns false for my buffer, but I can't find any documentation on why, or how to change it. Is that even relevant? I only want to use the main screen's stencil buffer.

Any help would be much appreciated,
Rob.

 

I Was trying to use the stencil buffer for shadows, but whenever I add an image effect it doesn't work anymore. I guess it is related to this problem. Does someone knows a workaround?

 

Im not 100% sure on this but from my experience it looks like Unity clears the stencil buffer before the OnRenderImage function call, meaning anything you have written into it is no longer there. If there is a way to prevent this Im unaware of it.

The undocumented RenderTexture.SupportsStencil suggests Unity plans to add the ability to give individual render tex's there own stencil buffer to use in image effects.

At the moment I think there's maybe only 1 stencil buffer thats gets shared with the deferred renderer, which may also be when it currently gets cleared (even if deferred is not enabled).

EIDT - Maybe this will work. Havnt tested it.

camera.clearStencilAfterLighting Pass = false;

 

Hi Rob,

I was trying actually to use the stencil buffer support and I'm working with deferred lighting active as well.

What I'm trying to do is just to mask the rendering of a sphere from the other geometry in the scene (for now testing it with solid green, so I'd expect to see green just on the sphere).

i.e.

1) Render the sphere in the stencil buffer marking pixel with reference value 1 using a write mask because seems like deferred pipeline is using some of
the bits of the stencil to do its own dirty stuff.

Stencil{
WriteMask 4 //use just the 3 lowest bits on writing
Ref 2
Comp Always
Pass Replace
}

2) Then I need for the stencil test to pass during a post fx pass (so OnRenderImage will do something). Looks like that because stencil buffer gets cleared before OnRenderImage is called I'll lost all of the contents of the bits that I've set beforehand. So basically same problem as you ...

So during postfx the shader does stencil test against ref value 2, like this:


Stencil {

ReadMask 4 //use just the 3 lowest bits on reading
Ref 2
Comp Equal
Pass Keep

}


I get green everywhere and If I don't use a WriteMask/ReadMask I won't get any result (I guess because I screw the bits the deferred pipeline uses for the light volume masking).

I guess that If what scrawk is saying is true (i.e. stencil buffer is cleared just before OnRenderImage), then I will obviously loose the ref value that I set in the 1st pass :/.

I wonder whether you solved the problem in some way ... It's driving me crazy. A feature like stencil buffer which is out there by years and years and that is

not even a special feature is so difficult to use in a rigid pipeline, like the unity one.

 

any solution to this problem?