Creating a plane that would prevent anything from being rendered in front of it

Is it possible to do what's described in the topic?

If not, is it possible to create two shaders, of which the shader A would just pass everything through but the shader B wouldn't render anything except what goes through both shaders A and B?

 

Or can I create a clipping mask that would say "don't render anything in front of this"?

 

It's possible with a clip plane, but you'd have to adjust the shader you're using to clip based on the clip plane.

Code (csharp):

1.  
2. clip(dot(pos_world, clip_plane.xyz) - clip_plane.w);
3.  

That skips pixels that are on the wrong side of the plane defined by clip_plane. (xyz = plane normal, w is distance from origin.)

The shader A and B combination is also possible using the stencil buffer. Shader A could mark pixels it renders in the stencil buffer and shader B can skip unmarked pixels.

Code (csharp):

1.  
2. // A: Mark
3. Stencil {
4.     Ref 1
5.     WriteMask 1
6.     Pass Replace
7. }
8.  
9. // B: Skip
10. Stencil {
11.     Ref 1
12.     ReadMask 1
13.     Comp Equal
14. }
15.  

 

I haven't really messed around with the CGPROGRAM stuff, but wouldn't that only affect the objects that have this code in their shader, not other objects? I can't go adding this clipping shader to all objects as they need to have their own shaders.

Yeah, I was thinking that a pair of stencil shaders would allow this! Somehow I just can't get that to work due to my limited understanding of shaders and how to write them. In order for the shader A to appear transparent, I added the transparent tag:

Code (csharp):

1.  
2. Tags { "Queue" = "Transparent" }
3.         Pass {
4.             Stencil {
5.                 Ref 1
6.                 WriteMask 1
7.                 Pass Replace
8.             }
9.             Blend Zero One
10.         }

But this is probably a wrong way to make it transparent in this context?

In any case, thanks a lot for trying to help!

 

Well, of course. There is no general clipping plane solution in Unity as far as I know. You'll have to make clipping plane versions of all shaders. You can assign them automatically using replacement shaders. (I've done the exact same thing in order to render planar reflections.)

You didn't mention anything about shader A being transparent before. Just that shader B should only render where shader A has been rendered before.

 

Yeah, I just thought that it could be somehow possible to manipulate the rendering order and the way things are rendered in a way that would have the same end result. Like for example: if a shader could somehow set certain pixels "final". If the shader would set certain pixels final, it wouldn't matter if objects in front of it tried to render something in front. But I guess this just simply isn't possible using Unity?

Ahh, I didn't know about that feature! So you can set it so that the replacement shaders apply to only one camera and other cameras will render those shaders with their original shaders? This is starting to sound like a valid possibility! A couple of questions though:

1. Since setReplacementShader only takes one shader (and a tag), it seems to me that it's not possible to do "versions of all shaders" using that, but instead I'd have to choose a common "neutral" replacing shader for those. Or did I misunderstand something related to that?
2. So even though the replacement shader would allow to automatically set the shaders for the objects in terms of rendering with a camera A, I guess it's not possible to automatically "append" this clipping logic to an existing shader?

Question 2 is mostly an issue of convenience I guess, although regarding the built-in shaders I'm not sure how to recreate them with this clipping added. Question 1 is about how good the end result can be. With some trickery I think I could get it so that in a majority of situations things would work quite alright, but I don't think I can entirely avoid small glitches caused by that.

Ah, maybe I wasn't being clear about that. Hard to explain something like this sometimes. By "passing through" and "going through both shaders A and B" I meant the stuff that's behind them.

Any way, thanks again! Really good to know about the replacement shaders!

 

I think it would be best to further explain what you want with some graphics. As far as I can judge now, using the stencil buffer will probably be an easier approach than using an actual clipping plane. And in these cases you can usually reduce the amount of additional shaders needed, by using a RenderTexture instead.

And on the questions. Replacement shaders can replace all shaders, by calling it multiple times. They are replaced on the base that the value of the tag is the same. On built in shaders that is a bit difficult, because you can't manipulate the tags in them. (Adding a clipping version is actually quite possible after making some copies of files, but you can't adjust the originals.)

You can't really append the clipping to a shader, so you do need specific variants of every shader.

 

Thanks for the clarifications!

My need is that one camera shouldn't render anything in front of a plane, but this plane can be in an arbitrary angle, so using the regular near clipping threshold won't do. What should be and what shouldn't be rendered can have any kinds of shaders. If I understand stencil shaders correctly the A + B shader solution won't actually be a working solution for this, since isn't it so that an object X that is between the shaders A and B will also carry the stencil Ref from A and thus be rendered?

RenderTextures have also crossed my mind, but I don't think that can be a solution here either as that texture would also include the things in front of the plane I'd want to use for clipping, and even so, there would probably be projection related issues.

All in all, if, by any means a Stencil shader solution would do this, that would of course be awesome and the best way to go. The second option would be that replacementShader option, I think, although it would probably be a somewhat imperfect and troublesome option. The third option is that I'd do a huge amount of mesh cutting, cloning and disabling renderers. There is some cutting involved in options one and two too, but this would require a lot more, which I think would be not only cumbersome implementationwise, but also be computationally more heavy than I'd want this to be.

 

I often see questions where it's about rendering some object B through an object A even though object B is farther away. In those cases I recommend using the stencil buffer. In your case it does seem a basic clipping plane case. You can do a trick to only render in front of a plane, but not behind it. For that I think clipping plane variants of shaders is the only way.

 

Yes, I already have that: not rendering behind a plane. For that it sufficed that the shader just had a "blend zero one". I don't see that as the only way though, just the most simple and effective one. I mean in addition to using that clip() function (which does fall under the "variants of shaders" umbrella), there is the option of mesh cutting too, like there is for the "not rendering in front of plane" problem. For the far plane there's just no need for that complexity.

In any case I was hoping that there would be an almost as simple method for the front, and now that it's starting to seem that it's not possible, it feels somewhat weird, considering how versatile and full of possibilities the engine is in general.

Now, assuming that A is not an option, considering the two secondary options of:
B) using shaderreplacement(s)
C) doing a lot more cutting etc.

I'm starting to lean towards B even more than yesterday, so thanks again for bringing that to the table. Although more complex to implement and maintain than the hypothetical (and still hopefully somehow possible ) solution A, solution B could in principle result with a pretty perfect result, as long as the source codes for the built-in shaders are available (as they seem to be from the Unity web site) so that I could attach tags to them. C would probably have more performance issues and due to that I'd probably have to do some compromises with some effects I'm planning and still end up with more potential hicups. In addition, considering my further goals in my project, although originally sort of closer to my comfort zone, option C would ultimately most likely become quite a nightmare to maintain.

 

I did some testing now with the clipping method, and it works pretty sweet (as I put it in a surf (Input IN, inout SurfaceOutput o) method). But looking at this, how do I exactly pass the external clip_plane to the shader?

 

In this case it's a global thing, so I'd use a global variable, Shader.SetGlobalVector.

 

Awesome! I first thought I could give those as shader properties too, but that apparently isn't supported by RenderWithShader. But the GlobalVector works pretty much equally conveniently as giving properties with renderWithShader would. Thank you very much for helping out in all of this!

Starting to look good - with the exception that I haven't done any versioning of the existing shaders yet, which can be seen in the visuals pretty easily.

But even before going to shader versioning, there's one (two-part) problem left: even though the clipped objects aren't rendered:
- the shadows that they cast are rendered.
- the unrendered objects still occlude other shadows from being rendered.

I haven't still figured out how to fix that. I found this thread, but with that code I only managed to remove the shadows at the cost of getting the objects themselves back.

 

You'll need a clipped version of the shadow shader too. (I don't use any of the built in shaders, but have a complete set of my own shaders that also includes a clipped version of the shadow casting shader.)

 

Shadow shader? Do you mean a part of this shader or really a separate shader? I thought all renderers only have one shader each. So do they normally share one global shadow shader? Something that has a tag so that I could replace that with the camera's renderWithShader?

 

Sorry for the bad terminology, it's not a shadow shader, but the shadow casting pass in the shader. How to adjust it depends on what type of shader format you use. (vertex/fragment or surface.)

For vertex/fragment you can look at the standard shader to find the shadow casting pass:

Code (csharp):

1.  
2. // ------------------------------------------------------------------
3. //  Shadow rendering pass
4. Pass {
5.     Name "ShadowCaster"
6.     Tags { "LightMode" = "ShadowCaster" }
7.  
8.     #include "UnityStandardShadow.cginc"
9. }
10.  

In surface shaders you usually inherit it from your fallback shader, but you can also specify it explicitly.

 

Ok, I'll try that, but first I needed to upgrade to Unity5 as Unity4 didn't have that .cginc in the built-ins. The problem just is that now I'm getting this error for a .dll I've made:

Missing method .ctor in assembly ....../......, type System.Runtime.Versioning.TargetFrameworkAttribute​

I rebuilt the .dll with Unity5 dll's, but that didn't help. Well, that doesn't really have anything to do with the topic of this thread in general, but I'll just try to fix it now somehow.

But I did some further testing in Unity4, and it made me again question what the shader fallbacks precisely do. The only thing I needed to make the shadows disappear using the clipper shader was to just set Fallback off (but then they removed also the shadows of the unclipped parts of the geometry, so that's really not a great solution). But the Unity documentation says:

- It basically says “if none of subshaders can run on this hardware, try using the ones from another shader”.
​

and

"A fallback statement has the same effect as if all subshaders from the other shader would be inserted into its place."
​

These comments seem misleading to me considering the effect of changing the fallback here, eg. to that "Fallback off". Isn't it so that in case the system is able to process the first subshader, the latter ones aren't used? So, if my subshader does successfully do the clipping, why does it still proceed to the later fallback subshaders? Is it because my subshader doesn't have any passes? Or is it that fallbacks are handled separately for each render queue or something like that?

What was also really confusing is that making the change "Fallback off" to my world clipping shader, which was only used by camera X, renderers that had another clipping test shader attached to them were suddenly rendered on camera Y through other geometry. Changing fallback for shader A should never affect anything just using shader B, right? Uh.

 

Yes, it misses the shadow caster pass. So it renders the clipped shader for the main/color pass, but for the shadow caster pass it resorts to using the version from the fallback. So you'll need to make your own shadow caster pass that includes the clipping. Just follow the fallbacks and you should be able to find the standard shadow caster pass somewhere in the vertexlit shaders. Then copy it and modify it to include the clipping.

 

I have to ask yet another terminology clarification. You're talking about a pass, but isn't a pass a part of a subshader? Or is the functionality so that it first takes one pass type and goes through all subshaders until it finds a definition for this kind of a pass under one of the subshaders, then does whatever is defined in that subshader for that pass, and then does the same thing for next pass, starting from the very first subshader? So pass 1 could use shader C, after which pass 2 could use eg. shader B, which is before shader C? If this is so, it's weird, as in the documentation I see no mentions about this. All it says is " When Unity has to display a mesh, it will find the shader to use, and pick the first subshader that runs on the user’s graphics card."

So since this clipping subshader does run, what exactly determines what pass it's used for? I mean if it now runs for the basic geometry, why isn't it run for the other passes?

Btw. I still have the "Missing method .ctor in assembly" problem introduced by Unity5 (made a separate thread for that). But I wanted to get past it so I ditched the whole .dll and all references to it for the time being, so I could test how the project otherwise works now. There's been another change for the worse! Now suddenly my basic clipping plane

Code (csharp):

1.  Pass
2.          {
3.            Blend Zero One
4.          }
5.      

suddenly doesn't work properly any more. At certain angles it just renders stuff through it. Well, I noticed that the stuff seen through was mostly using shaders that are now classified as legacy, and changing those fixed them for the most part, but not entirely! How come has the functionality of such a simple shader changed like this? And why does it seem to work so arbitrarily? Earlier it really did not let ANYTHING show from behind. Has the very basic blending somehow changed in Unity5? And for the worse?

I also tried what you posted earlier:

Code (csharp):

1. //  Shadow rendering pass
2. Pass {
3.   Name "ShadowCaster"
4. Tags { "LightMode" = "ShadowCaster" }
5.  
6.  
7.   #include "UnityStandardShadow.cginc"
8. }
9.  

but it complains about a syntax error for the #include line. And if I comment the include line, the tag "lightmode" = "shadowcaster" just simply makes the textures look sort of "interlaced", in other words having dark stripes all over. What's happening there?

I was hoping that upgrading to Unity5 would help with the issues, but unfortunately it just created more of them.

 

I never had that many issues upgrading to Unity 5, but there are some pretty massive changes in the rendering pipeline.

The subshader/pass ordering has multiple uses and I also don't know the Unity behaviour 100%. A shader can have multiple subshaders, which can define different quality/requirement levels. (With the highest quality/requirements at the top.)

A subshader can then have multiple passes. By default, all passes are rendered, but it's also possible to manually select the pass. The passes are also used to differentiate between different stages in the rendering pipeline. I assume it's not the name, but the LightMode tag that actually switches between passes. If the shader is used in a base forward pass it needs to give a whole different output than in a deferred pass.

The #include should actually be inside the CGPROGRAM block. The part I posted earlier is by no means a complete shader pass. It just shows where to look in the standard shaders for the ShadowCaster pass.

 

Well, now my first goal is to get to the point where I was before upgrading to Unity5. I have this extremely simple shader:

Code (csharp):

1.  Shader "foobar"
2. {
3.      SubShader
4.      {
5.          Pass
6.          {
7.            Blend Zero One
8.          }
9.      }
10. }

This worked in Unity 4 but now only partially. Any ideas why or how to fix this? I would understand it better if it wouldn't work at all, but it works like with an 80% accuracy: only some items are rendered through it and only at certain positions / distances / angles. All of it seems so random: I move the camera to one direction and suddenly 2 objects appear, I move more and they disappear. What's more is that a more distant item randomly appears in front of the more closer item. What the heck can cause all that? It doesn't seem to make any sense, yet there of course has to be some underlying cryptic weird logic behind it.

Apparently one needs to define both a "vertex program" and a "fragment program" inside the cgprogram (unless it's outside a pass(?)), eg. something like this:

Code (csharp):

1.  
2.         float4 vert(appdata_base v) : POSITION{
3.             return mul(UNITY_MATRIX_MVP, v.vertex);
4.         }

Well, by putting return float4(1.0, 1.0, 1.0, 1.0); there I'm able to set a color for the part that I don't even want to see, so that's sort of something. But still can't ditch the shadows. And what's really confusing is that by playing around with the frag and vert "programs" (?) it seized to affect just the rendering but also the behaviour in game! It seemed like it affected collision! I though shaders are just about rendering! How deep does this rabbit hole go?!

 

It seems to me like a simple shader set to write depth at the near plane would be a far easier solution.

Code (ShaderLab):

1. Shader "Custom/Blocker" {
2.     SubShader {
3.         Pass {
4.             Tags { "Queue" = "Background" "LightMode" = "Always" }
5.             Cull Off
6.             ColorMask 0
7.             CGPROGRAM
8.  
9.             #pragma vertex vert
10.             #pragma fragment frag
11.             #include "UnityCG.cginc"
12.  
13.             float4 vert(float4 v:POSITION) : SV_POSITION {
14.                 return mul (UNITY_MATRIX_MVP, v);
15.             }
16.  
17.             fixed4 frag(out float depth SV_Depth) : SV_Target {
18.                 depth = _ProjectionParams.y;
19.                 return fixed4(1.0,0.0,0.0,1.0);
20.             }
21.  
22.             ENDCG
23.         }
24.     }
25. }

 

Yes, you need to make an actual shader. Just setting the blending is not enough. I think you'll need to dive a bit deeper into shader programming.

 

Thanks for chiming in! I'm not sure if your shader script was meant to address the problem mentioned in the topic (not rendering or seeing things in front) or the Unity5-introduced problem of not having a working back clipping plane either. In any case I tried that (just fixed the missing colon), but without the "depth = _ProjectionParams.y" it seems to behave exactly the same way as my even more simple shader that only has the blend. In other words only almost doing what I want. And the depth = _ProjectionParams.y just seems to move the clipping plane to be at the same distance from the actual plane (that has the material with the shader attached) as the camera is. So it just moves the clipping to the wrong place.

 

What is an "actual shader"? Isn't it an actual shader if it doesn't cause compilation errors? And isn't it an actual shader if it (used to) work? Even now it almost works, and not in any way worse than the one with CGPROGRAM that bgolus posted.

 

Ah, I missed that you wanted to render stuff beyond it, and only exclude things between the plane and the camera. My shader should prevent anything (that does a ZTest) from rendering anywhere that shader is on screen.

 

No worries. After all, in Unity5 that has become a problem too! Even if removing the "depth = _ProjectionParams.y;" it still has the same problem of randomly letting stuff through that my original shader has. Wanna try it out? Just throw some objects in the scene and move the camera around to different angles, positions and rotations. In some places it works, but sometimes things just shine fully through the blocking shader.

So the situation is:
1. Original front-clipping problem half-solved: should just get the shadows to clip too.
2. New problem has arisen with the back-clipping being problem in Unity 5.

For number 1, the thought of manipulating the vertices with the vert function could work in some cases I guess (eg. by discarding the verts in wrong places or somehow projecting them on the plane if that's possible), but in most cases it would just make things broken. So still pretty clueless about that.

For number 2: it's now been tested with a simple GCPROGRAM version and my original very simple blending shader: both have the same seemingly random behaviour, which just seems broken no matter how you look at it. Even with shaders being hard to figure out, usually a single shader works consistently. This just seems random.

 

I checked Unreal Editor to find out if it would have more possibilities regarding this, but it seems that it's actually the opposite. Unreal Editor doesn't have the culling options for cameras that Unity has, which I need, and it doesn't seem like there's an alternate methodology to do this.

So anyone? Any clue as to why the simple blending doesn't work in Unity 5 like it used to in Unity 4? Or how to properly block anything from being rendered behind a plane?