OnImageRender() with stencil buffers of source active

Hi

I am currently trying to write an image effect that uses the stencil buffer, and this really drives me mad!

Seemingly there is no way to make this work even though I have read from people that achieved this, and there are at least two ways described on how this could work.

My Questions:


1. Lots of people saying that this cannot work with OnImageRender flatout. Seemingly they had good results using OnPostRender() and manually setting up the correct RenderTextures.

Tried many things, but I cannot get the Blit in this method to work correctly. All I get is a thin strip at the top in the Game View of the editor covered by my simple shader (which should color the stenciled parts green) colored green, which hides the lowest button bar of the UI, but not the game view.

Is there any obvious mistake I could have made that would result in such a behaviour?


2. Unity documentation itself contains this paragraph on the page about Graphics.Blit:

Sooooo. An example on this would have been to much to ask for? Anyway, I gave it a try, but all I get is a black screen... most probably because I neither tell Unity about the shader I would like to use for this quad, nor that the src RenderTexture should be somehow involved. What I currently have is this:

Code (csharp):

1.  
2.   void OnRenderImage(RenderTexture src, RenderTexture dest)
3.   {
4.       Graphics.SetRenderTarget(dest.colorBuffer, src.depthBuffer);
5.       GL.LoadOrtho();
6.       material.SetPass(0);
7.       GL.Begin(GL.QUADS);
8.       GL.End();
9.       //Graphics.Blit(src, material);
10.   }
11.  

I just faithfully reproduced what the Unity documentation told me to. I guess I am missing half the code that is needed, but without any indication on what is missing I cannot really make any progress there.

So could anyone give me any pointers as to what the Unity documentation left out?


3. Oh, and while experimenting with the OnPostRender() MEthod of doing things, I had to manually assign a renderTexture to the camera. Which I couldn't get off it again, and the RenderTexture was still stuck to the cam in the editor. Had to manually remove it in the Editor UI after every playtesting session. Why?

Is there a way to remove such a rendertexture again from the cam before the game closes?

 

Thanks for the quick answer!

I also stumbled on a post with someone suggesting CommandBuffers for that.

Now, I tried hack together a cs script to Insert into the command buffer and a shader to go with it, but I cannot really get it to work, at all. Seems like the commandbuffer does not execute anything,even if I change the shader so solid green is output instead of the RenderTexture content.
Most probably I made a very stupid newbie mistake somewhere, so could you glance through my code quickly and tell me if I made one in it?

The cs code:

Code (csharp):

1.  
2.   [ExecuteInEditMode]
3.   [RequireComponent(typeof(Camera))]
4.   [AddComponentMenu("Image Effects/Occluded Object Highlighting")]
5.   public class OccludedObjectHighlighting : MonoBehaviour
6.   {
7.       private CommandBuffer occludedObjectBuffer;
8.       private Material material;
9.  
10.       void Awake()
11.       {
12.           material = new Material(Shader.Find("Hidden/Occluded Object Highlighting"));
13.       }
14.  
15.       void OnEnable()
16.       {
17.           if (occludedObjectBuffer == null)
18.           {
19.               occludedObjectBuffer = new CommandBuffer();
20.               occludedObjectBuffer.name = "Highlight Occluded Objects";
21.               int texID = Shader.PropertyToID("_RenderTexture");
22.               occludedObjectBuffer.GetTemporaryRT(texID, -1, -1, 0, FilterMode.Bilinear);
23.               occludedObjectBuffer.Blit(BuiltinRenderTextureType.CurrentActive, texID, material);
24.  
25.               GetComponent<Camera>().AddCommandBuffer(CameraEvent.BeforeImageEffects, occludedObjectBuffer);
26.           }
27.       }
28.  
29.       void OnDisable()
30.       {
31.           if (occludedObjectBuffer != null)
32.               GetComponent<Camera>().RemoveCommandBuffer(CameraEvent.BeforeImageEffects, occludedObjectBuffer);
33.           occludedObjectBuffer = null;
34.  
35.           DestroyImmediate(material);
36.       }
37.   }
38.  

The shader attached to it:

Code (csharp):

1.  
2. Shader "Hidden/Occluded Object Highlighting" {
3.    Properties{
4.      _MainTex("Base (RGB)", 2D) = "white" {}
5.    }
6.      
7.    SubShader {
8.      Tags{ "RenderType" = "Opaque" "Queue" = "Overlay" }
9.      Pass
10.      {
11.        // render the RenderTexture out
12.        Lighting Off
13.        ZWrite Off
14.        ZTest Always
15.  
16.        CGPROGRAM
17.        #pragma vertex vert
18.        #pragma fragment frag
19.        struct appdata {
20.          float4 vertex : POSITION;
21.          float2 uv : TEXCOORD0;
22.        };
23.        struct v2f {
24.          float4 pos : SV_POSITION;
25.          float2 uv : TEXCOORD0;
26.        };
27.        v2f vert(appdata v) {
28.          v2f o;
29.          o.pos = mul(UNITY_MATRIX_MVP, v.vertex);
30.          return o;
31.        }
32.  
33.        sampler2D _RenderTexture;
34.  
35.        fixed4 frag(v2f i) : SV_Target
36.        {
37.          fixed4 col = tex2D(_RenderTexture, i.uv);
38.          return col;
39.        }
40.        ENDCG
41.      }
42.    }
43.  
44.    SubShader {
45.      Tags{ "RenderType" = "Transparent" "Queue" = "Overlay" }
46.      Pass{
47.        // Add the overlay effect
48.        Stencil{
49.          Ref 1
50.          Comp Equal
51.          Pass keep
52.          ZFail keep
53.        }
54.        Lighting Off
55.        ZWrite Off
56.        ZTest Always
57.        Blend SrcAlpha OneMinusSrcAlpha
58.  
59.        CGPROGRAM
60.        #pragma vertex vert
61.        #pragma fragment frag
62.        struct appdata {
63.          float4 vertex : POSITION;
64.        };
65.        struct v2f {
66.          float4 pos : SV_POSITION;
67.        };
68.        v2f vert(appdata v) {
69.          v2f o;
70.          o.pos = mul(UNITY_MATRIX_MVP, v.vertex);
71.          return o;
72.        }
73.  
74.  
75.        half4 frag(v2f i) : COLOR{
76.          return half4(0,1,0,1);
77.        }
78.        ENDCG
79.      }
80.    }
81. }
82.  

Oh, and I get a ton of these errors every time I do changes and these get built:

Any reason for that? I guess something I added to the CS file is blowing up, but I am not sure what it is, only that it started when I created these files.

 

Thanks again for the answer!

I tried to implement the changes you proposed, but I still not get this code to work.

I checked the Frame Debugger, and I installed and checked renderDoc. I do see that my command Buffer is correctly setup and executed, but I don't see anything beyond that. Is there anything in renderDoc that would tell me what the command Buffer actually does? I have a hard reading the output. I see that it seems to render something...

Now, here is the current code after modifications. I removed one of the subshaders, I don't think this shader will work the way I want it to ultimately (to overlay the current rendered image from the input image texture with semi transparent flat colors masked with the stencil buffer), but at the current point in time a settled to just try to overwrite the current image with a flat color just to prove that the shader is doing something.

The cs script:

Code (csharp):

1.  
2. using System;
3. using UnityEngine;
4. using UnityEngine.Rendering;
5.  
6. namespace SoldiersOfFortune.ImageEffects
7. {
8.  
9.   [RequireComponent(typeof(Camera))]
10.   [AddComponentMenu("Image Effects/Occluded Object Highlighting")]
11.   public class OccludedObjectHighlighting : MonoBehaviour
12.   {
13.      private CommandBuffer occludedObjectBuffer;
14.      private Material material;
15.  
16.      void Awake()
17.      {
18.         material = new Material(Shader.Find("Hidden/Occluded Object Highlighting"));
19.      }
20.  
21.      void OnEnable()
22.      {
23.         if (occludedObjectBuffer == null)
24.         {
25.            occludedObjectBuffer = new CommandBuffer();
26.            occludedObjectBuffer.name = "Highlight Occluded Objects";
27.            int texID = Shader.PropertyToID("_RenderTexture");
28.            occludedObjectBuffer.GetTemporaryRT(texID, -1, -1, 0, FilterMode.Bilinear);
29.            occludedObjectBuffer.Blit(BuiltinRenderTextureType.CurrentActive, texId, material);
30.            occludedObjectBuffer.Blit(texID, BuiltinRenderTextureType.CurrentActive);
31.            GetComponent<Camera>().AddCommandBuffer(CameraEvent.BeforeImageEffects, occludedObjectBuffer);
32.         }
33.      }
34.  
35.      void OnDisable()
36.      {
37.         if (occludedObjectBuffer != null)
38.            GetComponent<Camera>().RemoveCommandBuffer(CameraEvent.BeforeImageEffects, occludedObjectBuffer);
39.         occludedObjectBuffer = null;
40.  
41.         DestroyImmediate(material);
42.      }
43.   }
44. }
45.  

The shader code:

Code (csharp):

1.  
2. Shader "Hidden/Occluded Object Highlighting" {
3.    Properties{
4.      _RenderTexture("Base (RGB)", 2D) = "white" {}
5.    }
6.  
7.    SubShader {
8.      Tags{ "RenderType" = "Transparent" "Queue" = "Transparent" }
9.      Pass{
10.  
11.        /*Stencil{
12.          Ref 1
13.          Comp Equal
14.          Pass keep
15.          ZFail keep
16.        }*/
17.        Lighting Off
18.        ZWrite Off
19.        ZTest Always
20.        Blend SrcAlpha OneMinusSrcAlpha
21.  
22.        CGPROGRAM
23.        #pragma vertex vert
24.        #pragma fragment frag
25.        struct appdata {
26.          float4 vertex : POSITION;
27.        };
28.        struct v2f {
29.          float4 pos : SV_POSITION;
30.        };
31.        v2f vert(appdata v) {
32.          v2f o;
33.          o.pos = mul(UNITY_MATRIX_MVP, v.vertex);
34.          return o;
35.        }
36.  
37.        //sampler2D _RenderTexture;
38.  
39.        half4 frag(v2f i) : COLOR{
40.          return half4(0,1,0,1);
41.        }
42.        ENDCG
43.      }
44.    }
45. }
46.  

Now, I played around with the RenderQueue settings, tried to make the shader an opaque one, all to no avail. I see the CommandBuffer being active in the Frame Debugger, but the final image is not affected by the shader at all.

At the current point in time I suspect my shader is not working for some reason, or that the material is applied wrongly. What should I do next to find the root problem?


EDIT:

If I change the cs code so the result is blitted to "(RenderTexture) null" (the screen), I get something being overlayed with a green flat color like intended, but it is only the top bar of the play mode window containing the format drop down and the flag buttons like "maximize on play"... what the...?
Is this part of the window also kind of drawn by the camera? Why does the RenderTexture "null" end up only affecting that part of the window?

The changed part of the code:

Code (csharp):

1.  
2.   void OnEnable()
3.   {
4.      if (occludedObjectBuffer == null)
5.      {
6.         occludedObjectBuffer = new CommandBuffer();
7.         occludedObjectBuffer.name = "Highlight Occluded Objects";
8.         int texID = Shader.PropertyToID("_RenderTexture");
9.         occludedObjectBuffer.GetTemporaryRT(texID, -1, -1, 0, FilterMode.Bilinear);
10.         occludedObjectBuffer.Blit(BuiltinRenderTextureType.CurrentActive, texID, material);
11.         occludedObjectBuffer.Blit(texID, (RenderTexture)null);
12.         //occludedObjectBuffer.Blit(texID, BuiltinRenderTextureType.CurrentActive);
13.         GetComponent<Camera>().AddCommandBuffer(CameraEvent.BeforeImageEffects, occludedObjectBuffer);
14.      }
15.   }
16.  

 

That did the trick. Now I see my shader affecting the image.

I do not get the RenderTexture blitted to the shader correctly however, or I have some error in my shader somewhere. All I get when I am trying to output just the renderTexture content in the shader with tex2D(_RenderTexture, i.uv) is the default color the property was set to.

I guess I now need to sit down and really dig into this shader code.


Thanks Tim-C, you helped me a ton!


EDIT:

Worked on it some more, and cannot get it work. When I blit to the temporary RT, and back to the cameraTarget WITHOUT using the material, I get the original image shown as expected.

As soon as I use the material during the blit, I get a black screen. I checked the shader on a cube, and there it works fine if I set the texture manually.
So either the shader has no access to the RenderTexture written, and instead outputs a black color or the shader code does not work as expected when used with blit instead of assigning it to 3D geometry.

Any idea what I could try next? Are there good examples for similar shaders that I could look at?

 

That still does not work. I removed the property block, I still do not get the shader to output the RenderTexture. There must be something else wrong with my shader code.

This is my current code:

Code (csharp):

1.  
2. Shader "Hidden/Occluded Object Highlighting" {
3.  
4.    SubShader
5.    {
6.      Tags{ "RenderType" = "Opaque" }
7.      LOD 100
8.  
9.      Pass
10.      {
11.        CGPROGRAM
12.        #pragma vertex vert
13.        #pragma fragment frag
14.  
15.        #include "UnityCG.cginc"
16.  
17.        struct appdata
18.        {
19.          float4 vertex : POSITION;
20.          float2 uv : TEXCOORD0;
21.        };
22.  
23.        struct v2f
24.        {
25.          float2 uv : TEXCOORD0;
26.          float4 vertex : SV_POSITION;
27.        };
28.  
29.        sampler2D _RenderTexture;
30.        sampler2D _RenderTexture_ST;
31.  
32.        v2f vert(appdata v)
33.        {
34.          v2f o;
35.          o.vertex = v.vertex;
36.          o.uv = TRANSFORM_TEX(v.uv, _RenderTexture);
37.          return o;
38.        }
39.  
40.        fixed4 frag(v2f i) : SV_Target
41.        {
42.          // sample the texture
43.          fixed4 col = tex2D(_RenderTexture, i.uv);
44.          return col;
45.        }
46.        ENDCG
47.      }
48.    }
49.    Fallback Off
50. }
51.  

EDIT:

Interesting... adding some additional lines to the command buffer reliably made the Unity Editor crash every time I started up game mode with that script attached (it actually also crashed during the compile of this script).

All I added where the following lines trying to get the screen content into the shader in some alternative ways:

Code (csharp):

1.  
2. Texture2D myTexture2D = new Texture2D(Screen.width, Screen.height);
3.   myTexture2D.ReadPixels(new Rect(0, 0, Screen.width, Screen.height), 0, 0);
4.   myTexture2D.Apply();
5.   material.SetTexture("_RenderTexture", myTexture2D);
6.  

Placed that before the blits in the hopes that if the shader couldn't read from the blit target, that the shaders local property _RenderTexture would at least be filled with a Texture 2D copy of the RenderTexture content.

Yeah, I guess these shouldn't be used in a command buffer?


EDIT 2:

After thinking more about the problem, I concluded that a Posteffect / command buffer based solution would most probably not work well for my problem, as I still need to do Ztests to make sure only objects BEHIND occluding objects are highlighted. If I could get access to this information in a commandbuffer / posteffect IDK...

But seeing how I struggled getting the shader for this command buffer solution to work, and how I had better success with transparent shaders using the stencil buffer in the past, I decided to switch to transparent shaders for now.
As soon as I made sure to both stencil write and read only from transparent (thus forward rendered) shaders, my solution started working.

As soon as I have optimized it more, and got some weird errors sorted out I will post about how I got it to work.
For now, I will no longer continue to work on my command buffer solution.


Thanks for the help anyway. Learned a lot along the way.

 

*bumped*

@Tim-C I'd be curious to know if you have ever actually accessed a stencil buffer from a shader that was fired by a CommandBuffer.Blit()? Because after working 36 hours on this I can 100% confirm that... I haven't been able to do it. You are the ONLY official looking person who's commented in any thread about accessing stencil buffers from a CommandBuffer, so I'd really appreciate it if you could actually run a test and see if you can actually do this. It was actually this thread that sent me on my 36 hour journey to trying to read a stencil buffer during a CommandBuffer.Blit(). So... you are kind of the reason I have been beating this dead horse this long, trying to get this to work. I will say that I HAVE been able to access the stencil buffer from a shader fired from a GRAPHICS.Blit(), but not from a CommandBuffer.Blit(). Again, if you could check this I would faint with appreciation.

 

*bump*

 

I got it working, see: http://forum.unity3d.com/threads/ha...uffer-to-copy-with-commandbuffer-blit.432503/

 

I take it back. I didn't get it working.

@Tim-C This doesn't work in the latest version of Unity (5.4.1f1). It just doesn't. Absolutely no texture is passed from the camera to the shader's _MainTex. Or in the very least, texID is gonna be white, black or grey after the above code finishes (along with the screen, since you copied it to the screen too).

The issue appears to be that Blit wants to run a "grab pass" to get the texture from the screen. I don't know if this is a new thing that's happening in the latest version of unity, or not. But this "grab pass" seems to be required before it can get the texture data from the screen. If you do NOT specify a custom shader, it does a "Grab" according to the frame debugger, and it works. But if you specify a shader, like what you did in your code above, it doesn't do the grab, and hence it passes nothing to _MainTex, and you get a white, black or grey screen.

However, if you render the camera directly to a RenderTexture, then bingo, the grab pass is no longer necessary, as that grab pass seems to only be needed by Blit if it is Blitted from the screen. But since it's coming from a rendertexture, no grab pass needed, and it sucessfully passes the texture to the _MainTex. And so you'd figure "Good, problem solved!", but no.. problem not solved at all. Because now the stencil doesn't work. I'm only theorizing here, but I'm guessing that either the stencil isn't saved by the camera into the rendertexture, OR Blit isn't passing the stencil data to _MainTex, OR the shader itself doesn't look for the stencil to be in _MainTex, but perhaps the shader looks for the stencil to be in the SCREEN, in which case it doesn't find any stencil data in the screen because the camera is rendering to a render texture! Ugh.

Anyway so my point is:

1) If you Blit from the screen, to a RT. with a custom shader, the RT is all white, black or grey. It doesn't work.
2) If you Blit from a camer's renderTexture, with a custom shader, it passes the texture, but the stencil doesn't work.

Been trying to get this to work for over 5 days now, and I'm pratically spamming the graphics forum and nobody seems to have any knowledge on this subject there. Hopefully someone here can me with this here.