Sprite Renderer Mask on Specific Sorting Layer

Hi guys,

I'm making a 2D game that zombies come out of the ground. The visuals are similar to Plants Vs. Zombies. There are 4 lanes that spawn zombies. I'm having trouble to find a good solution for that. There are 3 sprites that I'm working it: the background, the tomb and the zombie. I could simply get piece of the ground image and put above the zombie, but that would not be a great solution, because I want to change the position of the tombs and change the background image. I would have to slice the background for each situation. The obvious solution was to mask the zombie part that is beneath the ground. I found some mask shader on the Internet, and for all that worked on Unity 5, they masked everything that was above the mask. I could not configure the shader to select the sprites that I wanted to mask or to specify any parameter that limited the layer that the mask would affect. Here are the images of the game to illustrate what I'm saying:

This is two zombies getting out of the tomb. The Tomb on the center of the image is on the Sorting Layer "Tomb1", the zombie on the Sorting Layer "Zombie1", the tomb beneath in on the Sorting Layer "Tomb2", and the zombie "Zombie2".

When I apply the mask at the Sorting Layer "Zombie1", it masks everything above, including the "Tomb2" and "Zombie2". My question is if it is possible to limit the mask to have effect only in the Sorting Layer that the mask is included. The shader that I'm using is this:

Code (CSharp):

1.  Shader "Custom/MaskTest" {
2.      Properties
3.      {
4.          _MainTex ("Base (RGB) Alpha (A)", 2D) = "white" {}
5.          _Cutoff ("Base Alpha cutoff", Range (0,.9)) = .5
6.      }
7.  
8.      SubShader {
9.          Tags {"Queue" = "Transparent+1"}
10.           Offset 0, -1
11.          ColorMask 0
12.          ZWrite On
13.          Pass
14.          {
15.              AlphaTest Greater [_Cutoff]    
16.              SetTexture [_MainTex] {
17.                  combine texture * primary, texture
18.              }
19.          }
20.      }
21. }

I found that there is an Alpha Version of Unity with Mask Component to SpriteRenderer, but I don't have the Pro License, and this game will be release before december (the month that this feature will be released, according to Unity Road Map). I appreciate for reading this far.

 

Thank you very much!! It worked!!! The keyword that I was missing was "Stencil". I didn't know the existence of this buffer and its use for 2D games. I found this forum thread: http://forum.unity3d.com/threads/is-it-possible-to-clip-a-sprite-with-another-sprite.254144/ and with a little googling and your tutorial I could make it work.
I'll just check the shaders and share with you all.

 

Hi! Here are the shaders that solve this "problem". First, the shader for the SpriteRenderer of the objects that will masked.

Code (CSharp):

1. Shader "Sprites/Stencil Draw In Mask"
2. {
3.     Properties
4.     {
5.         [PerRendererData] _MainTex("Sprite Texture", 2D) = "white" {}
6.         _Color("Tint", Color) = (1,1,1,1)
7.         [MaterialToggle] PixelSnap("Pixel snap", Float) = 0
8.     }
9.  
10.     SubShader
11.     {
12.         Tags
13.         {
14.             "Queue" = "Transparent+1"
15.             "IgnoreProjector" = "True"
16.             "RenderType" = "Transparent"
17.             "PreviewType" = "Plane"
18.             "CanUseSpriteAtlas" = "True"
19.         }
20.  
21.         Cull Off
22.         Lighting Off
23.         ZWrite Off
24.         Blend One OneMinusSrcAlpha
25.  
26.         Pass
27.         {
28.             Stencil
29.             {
30.                 Ref 0
31.                 Comp Equal
32.             }
33.  
34.         CGPROGRAM
35.             #pragma vertex vert
36.             #pragma fragment frag
37.             #pragma multi_compile _ PIXELSNAP_ON
38.             #include "UnityCG.cginc"
39.  
40.             struct appdata_t
41.             {
42.                 float4 vertex   : POSITION;
43.                 float4 color    : COLOR;
44.                 float2 texcoord : TEXCOORD0;
45.             };
46.  
47.             struct v2f
48.             {
49.                 float4 vertex   : SV_POSITION;
50.                 fixed4 color : COLOR;
51.                 half2 texcoord  : TEXCOORD0;
52.             };
53.  
54.             fixed4 _Color;
55.  
56.             v2f vert(appdata_t IN)
57.             {
58.                 v2f OUT;
59.                 OUT.vertex = mul(UNITY_MATRIX_MVP, IN.vertex);
60.                 OUT.texcoord = IN.texcoord;
61.                 OUT.color = IN.color * _Color;
62.                 #ifdef PIXELSNAP_ON
63.                 OUT.vertex = UnityPixelSnap(OUT.vertex);
64.                 #endif
65.  
66.                 return OUT;
67.             }
68.  
69.             sampler2D _MainTex;
70.             sampler2D _AlphaTex;
71.             float _AlphaSplitEnabled;
72.  
73.             fixed4 SampleSpriteTexture(float2 uv)
74.             {
75.                 fixed4 color = tex2D(_MainTex, uv);
76.                 if (_AlphaSplitEnabled)
77.                     color.a = tex2D(_AlphaTex, uv).r;
78.  
79.                 return color;
80.             }
81.  
82.             fixed4 frag(v2f IN) : SV_Target
83.             {
84.                 fixed4 c = SampleSpriteTexture(IN.texcoord) * IN.color;
85.                 c.rgb *= c.a;
86.                 return c;
87.             }
88.         ENDCG
89.         }
90.     }
91. }

And the shader for the mask:

Code (CSharp):

1. Shader "Sprites/Stencil Mask"
2. {
3.     Properties
4.     {
5.         [PerRendererData] _MainTex("Sprite Texture", 2D) = "white" {}
6.         _Color("Tint", Color) = (1,1,1,1)
7.         [MaterialToggle] PixelSnap("Pixel snap", Float) = 0
8.         [MaterialToggle] ShowTexture("Show Texture", Float) = 0
9.     }
10.  
11.         SubShader
12.         {
13.         Tags
14.         {
15.             "Queue" = "Transparent"
16.             "IgnoreProjector" = "True"
17.             "RenderType" = "Transparent"
18.             "PreviewType" = "Plane"
19.             "CanUseSpriteAtlas" = "True"
20.         }
21.  
22.         Cull Off
23.         Lighting Off
24.         ZWrite Off
25.         Blend One OneMinusSrcAlpha
26.  
27.         Pass
28.         {
29.             Stencil
30.             {
31.                 Ref 1
32.                 Comp always
33.                 Pass replace
34.             }
35.         CGPROGRAM
36.             #pragma vertex vert
37.             #pragma fragment frag
38.             #pragma multi_compile _ PIXELSNAP_ON
39.             #pragma multi_compile _ SHOWTEXTURE_ON
40.             #include "UnityCG.cginc"
41.  
42.             struct appdata_t
43.             {
44.                 float4 vertex   : POSITION;
45.                 float4 color    : COLOR;
46.                 float2 texcoord : TEXCOORD0;
47.             };
48.  
49.             struct v2f
50.             {
51.                 float4 vertex   : SV_POSITION;
52.                 fixed4 color : COLOR;
53.                 half2 texcoord  : TEXCOORD0;
54.             };
55.  
56.             fixed4 _Color;
57.  
58.             v2f vert(appdata_t IN)
59.             {
60.                 v2f OUT;
61.                 OUT.vertex = mul(UNITY_MATRIX_MVP, IN.vertex);
62.                 OUT.texcoord = IN.texcoord;
63.                 #ifdef SHOWTEXTURE_ON
64.                 OUT.color = IN.color * _Color;
65.                 #else
66.                 OUT.color = IN.color * _Color;
67.                 OUT.color.rgb = 0;
68.                 #endif
69.                 #ifdef PIXELSNAP_ON
70.                 OUT.vertex = UnityPixelSnap(OUT.vertex);
71.                 #endif
72.  
73.                 return OUT;
74.             }
75.  
76.             sampler2D _MainTex;
77.  
78.             fixed4 frag(v2f IN) : SV_Target
79.             {
80.                 fixed4 c = tex2D(_MainTex, IN.texcoord) * IN.color;
81.                 if (c.a < 0.2) discard;
82.                 c.rgb *= c.a;
83.                 c.a = 0;
84.                 return c;
85.             }
86.         ENDCG
87.         }
88.     }
89. }

I created a copy of this mask shader changing the shader name to "Sprites/Stencil Mask Restore" and Stencil Ref value to 0. Then, I created the materials for the shaders mentioned. Finally, set up the scene as @sandboxed explained. First the mask sprites, then the sprites to be masked and finally the restore mask sprite. I put the last one as a child of the mask sprite, to easily fit one over the other, and set the appropriate material.
Here is the result, thanks to @sandboxed

 

Just wanted to thank you for posting the shaders code, it works great and saved me a lot of trouble

 

The game is available at Google Play: https://play.google.com/store/apps/details?id=com.funbites.thestarvingdead
Hope you like it.

 

Thanks for sharing the shader.

Could anyone point me in the direction of getting this to work with alpha blending?

 

I think you just have to copy and paste the shader that you want. Just add this part to the shader and follow those steps.

1. Stencil
2. {
3. Ref 0
4. Comp Equal
5. }

 

Nope. Hopefully 5.5. Masking multiple layers is a must, please Unity let's not wait until 6.

 

joaobsneto - Thanks, Thanks, Thanks !

 

Thank you for posting the shaders, joaobsneto, and for everyone who helped!

 

Hi, I used this shader and it is working but it doesn't have soft edges. Is it possible to use the alpha channel to get soft edged blending rather than the all or nothing threshold?

 

I don't think so... but I'm not a shader expert. I think that stencil doesn't support alpha blending. Am I right @sandboxed ?

 

5.4 is out and it seems like there is no masking for SpriteRenderer still.

 

yes no masking still.

 

In the Unity Roadmap it says...

@sandboxed what are "Workflow concerns"?
When do you think Sprite Masking will be available?
Thank you

 

Another solution I'm using, with unity 2017.1's SpriteMask feature:

Code (CSharp):

1. using UnityEngine;
2.  
3. namespace DCGame.UnityFeaturePatch
4. {
5.   [RequireComponent(typeof(SpriteMask))]
6.   public class SpriteMaskRangeFixer : MonoBehaviour
7.   {
8.     private static int index = 0;
9.     public SpriteRenderer[] targets;
10.     public SpriteMask mask;
11.  
12.     private void Start()
13.     {
14.       mask = GetComponent<SpriteMask>();
15.  
16.       foreach (var target in targets)
17.       {
18.         target.sortingOrder = index + 2;
19.         target.sortingLayerName = mask.sortingLayerName;
20.       }
21.       mask.isCustomRangeActive = true;
22.       mask.backSortingOrder = index + 1;
23.       mask.frontSortingOrder = index + 2;
24.  
25.       index += 2;
26.  
27.       if (index >= 1000)
28.       {
29.         index = 0;
30.       }
31.     }
32.   }
33. }
34.