[ShaderLab + Multi-Material] Making previous material invisible

I've been trying to figure out how to make a shader to do this the whole weekend but I can't do it.

Say that I have a cube with a material that uses the Standard shader. A texture is being shown.

Now what I want to do is add a second material to this cube, which uses my own shader that renders the cube invisible.

So: The second material should (with its shader) make the pixels written by the previous material invisible.

I have to be able to do this with multiple materials without modifying the previous material or the shader that the previous material uses.

All I've been able to do is make the cube completely black. I've tried a billion different blend combinations, nothing is right. Granted I'm new to ShaderLab but it feels like I should have been able to figure this out... I should just be able to write a transparent color, right? Throw away anything that was previously in those pixels?

Please help me!

 

No that's not really possible like that. The GPU doesn't keep a layered document like photoshop in its memory.

The easiest way to make something invisible is to just not render it. There is a per pixel option using grabpass, but I would not advice that because of performance. So the question is, if you don't want to render it, why are you rendering it?

 

What I'm ultimately is trying to achieve is a mask that I want to be able to use on any GameObject (using two shaders).

My plan is to put a GameObject in front of another. The one in front will have 1 material using my first shader. It marks everything it renders into the stencil buffer and then, well, not actually render anything.

Then on the GameObject behind (it would have 2 materials, the 2nd material using my second shader) I planned to read the stencil buffer and keep the pixels that were marked while setting ever other pixel as 100% transparent. This would create the type of mask I want.

Would this be possible somehow? Let me know if I need to explain what I'm trying to achieve better.

 

You can mark the stencil buffer and then skip pixels where the stencil buffer is marked. Rendering the pixels and then removing them is also possible by using render targets or grabpass, but it's just very inefficient.

I'm afraid you'll just have to adjust (all) then shaders that needs this to incorporate the stencil buffer rejection.

 

Wait, now I'm a little confused. You make it sound like it's possible in the first paragraph and then in the second paragraph you make it sound like it's impossible? Which one is it?

Also, if this can't be done with shaders is it possible to do it in some other way? For example by changing the area that OpenGL is allowed to draw before the GameObject is drawn to the RenderTexture?

 

It's possible, but a waste of resources. So it's not a very good choice.

 

Do you know how? I must be able to do this, doesn't matter if it's not a good choice.

If nothing else please tell me solely for the purpose of me learning something new.

 

Well, you'll have to use two camera's that are essentially the same. The first one renders the background/other objects. When this camera is done, you make a copy of the backbuffer to a RenderTexture. (Or you render directly into a RenderTexture.)

Then with the second camera you render the objects you are talking about. You'll now be able to remove them partly by restoring pixels from the RenderTexture. You do have to make sure both camera's use the same depth buffer.

 

Ouch, that doesn't sound good at all. Very costly, especially if several masks are used.

I was hoping that something could be done with:

Code (csharp):

1. MonoBehaviour.OnRenderObject()
2. MonoBehaviour.OnPreRender()
3. MonoBehaviour.OnPostRender()
4. MonoBehaviour.OnRenderImage()
5. MonoBehaviour.OnWillRenderObject()

And then by using: http://docs.unity3d.com/ScriptReference/GL.html

I did try to make a solution from that but it didn't seem like Unity called any of those MonoBehaviour-methods directly before and directly after actually rendering the GameObject. It would have been really, really useful if they did because then you could do all kinds of tricks.

In my case I had hoped to be able to use GL.Viewport in order to make a rectangular mask.

 

Is it possible to skip the Stencil {} block in a SubShader? For example in Properties {} I have:

Code (csharp):

1. Properties {
2.   _MyNumber("My Number", Int) = 1
3. }

And then in the SubShader I have:

Code (csharp):

1. if ([_MyNumber] != 0) {
2.   Stencil
3.   {
4.   Ref [_MyNumber]
5.   Comp Equal
6.   Pass Keep
7.   }
8. }

The code above won't work. But it should give an idea to what I want:
+ If _MyNumber is 0 the Stencil buffer should be ignored, all pixels are written.
+ Otherwise a pixel should only be written if the number in the stencil buffer equals _MyNumber.

I hope this is possible somehow? Then I think I will be able to do what I want.

Btw when is a Stencil Buffer reset (all values set to 0)? Is it when the current camera has finished rendering? The current frame is done (all cameras have rendered)? Does the GameObject's layer have anything to do with it...? It think it might but I'm not sure...

Sorry for asking a lot at once here.

 

Hm, I may have solved it:

Code (csharp):

1. Properties {
2.   _MyNumber("My Number", Int) = 1
3.   _MyComp("My Comp", Int) = 0
4. }
5. ...
6. Stencil
7. {
8.   Ref [_MyNumber]
9.   Comp [_MyComp]
10.   Pass Keep
11. }

If _MyComp is 0 the Comp will become "Disabled", if _MyComp is 3 the Comp will become "Equal". So now I can turn off masking for those with a _MyNumber of 0. Neat!

(The integer values can be found in "UnityEngine.Rendering.CompareFunction".)

As for when the Stencil Buffer is reset to all 0... I'm not sure but from what I can find it seems like it's when Unity has finished rendering the entire frame? GameObject layers or the cameras have nothing to do with it? If you (or anyone else) can confirm this or correct me I would be grateful.

New question: What is WriteMask and ReadMask used for in Stencil Buffers and how are they used? I haven't been able to figure this out yet, it looks like they are used for enabling several stencil buffers...? Some of which are used by Unity, which ones? (It also seems like Unity's "lightning" stencil buffer for example is not reset to 0 every frame... Guess it shouldn't be a problem as long as I stay away from Unity's ReadMask/WriteMask, if only I knew which ones those are.)

 

Like you already found out, you can switch the stencil settings using shader settings. And you need to go to Comp Always to disable the stencil rejection.

They are binary masks used when reading or writing to the stencil buffer. I normally keep them at the same value as ref. The stencil buffer is usually (but not always) 8 bits wide. If you write a 1 to the stencil buffer and then a 2 with WriteMask set to the default of 255 (or 11111111 binary) you'll end up with the value 2:

00000000 (Initial)
00000001 (Write 1 with mask 255)
00000001 (Result)
00000010 (Write 2 with mask 255)
00000010 (Result)

Often you want to combine multiple masks in the stencil buffer. In that case just set the WriteMask to the same value as Ref. Then if you write 1 and 2, you'll end up with 3. Then if you check for 1 with ReadMask 1 it will be equal and if you check for 2 with ReadMask 2 it will also be equal.

00000000 (Initial)
???????1 (Write 1 with mask 1)
00000001 (Result)
??????1? (Write 2 with mask 2)
00000011 (Result)