Modifying Shadow Map to correspond to vertex displacement in a fragment Shader

Hello Everyone,

I have this fragment shader which does displacement of vertices based on camera distance, I also use custom projections from camera inside the shader and not built in unity matrix mvp. I implemented the unity macros to make frag shader cast and receive shadows using the macros from autolight.inc (TRANSFER_VERTEX_TO_FRAGMENT etc..). The shadows display but they are using the old projections and do not follow my displacement either. I started dwelling inside the hlsl and autolight include files to find out if it is possible to change this , but I'm getting a bit lost on how unity does shadow projections internally. So i was wondering if anybody knew(or even if there was a way) to have shadows follow vertex shader displacement and/or use custom projections.

Thank you for helping.

Adbar

 

Ok, So i figured out that the the shadow caster/shadow collector passes are in the fallback shader. So now I removed the fallback and added my own custom shadow collector and shadow caster passes with my custom vertex modifications in my v2f program, problem is now is that even tho both of those passes features the same vertex modifications and same projections in the v2f , when i pass it down to transfer to the frag program(using transfer shadow collector and transfer shadow caster macros), my shadows are in the default projection and do not follow the displaced geometry either, should i be doing my deformation in a certain space? Im kinda stuck at the moment with this : (.

AdBar

 

Ok , so here is my shader code, as you can see I implemented the shadow caster and shadow collector passes down below, now i broke down the macros to see what it does internally, and it does some reprojection using the mvp unity matrix, most of the examples use v.vertex to move the shadows, but I am only using o.pos. Might be a trivial question, but what is the difference between o.pos and v.vertex?, they both mean the position of the vertice no? The shadows right now are being projected back into regular space with no bending and not using my custom projection/ Im really stuck right now, I cant figure out why this doesnt work , Really need help with this. Thx again, any help would be much appreciated.

Code (csharp):

1. Shader "Unlit With Shadowsv2" {
2.     Properties {
3.         _Color ("Main Color", Color) = (1,1,1,1)
4.         _MainTex ("Base (RGB)", 2D) = "white" {}
5.         _QOffset ("Offset", Vector) = (0,0,0,0)
6.         _Dist ("Distance", Float) = 100.0
7.     }
8.    
9.     SubShader {
10.         Tags {"Queue" = "Geometry-10" "RenderType" = "Opaque"}
11.         Pass {
12.             Tags {"LightMode" = "ForwardBase"}
13.  
14.             CGPROGRAM
15.                 #pragma multi_compile BEND_ON BEND_OFF
16.                 #pragma vertex vert
17.                 #pragma fragment frag
18.                 #pragma multi_compile_fwdbase
19.                 #pragma fragmentoption ARB_fog_exp2
20.                 #pragma fragmentoption ARB_precision_hint_fastest
21.                 #include "UnityCG.cginc"
22.                 #include "AutoLight.cginc"
23.                
24.                 struct v2f
25.                 {
26.                     float4  pos         : SV_POSITION;
27.                     float2  uv          : TEXCOORD0;       
28.                     LIGHTING_COORDS(1,2)
29.                    
30.                 };
31.                 float4 _QOffset;
32.                 float _Dist;
33.                 sampler2D _MainTex;
34.                 float4x4 viewMatrix;
35.                 float4x4 projMatrix;
36.                
37.                 v2f vert (appdata_tan v)
38.                 {
39.                     v2f o;
40.                     //o.pos = mul( UNITY_MATRIX_MVP, v.vertex);
41.                     #ifdef BEND_ON  // IS BEND ENABLED
42.                         o.pos = mul ( _Object2World , v.vertex);
43.                         o.pos = mul ( viewMatrix , o.pos);
44.                         o.pos = mul ( projMatrix , o.pos);
45.                         float4 vPos = mul (UNITY_MATRIX_MV, v.vertex);
46.                         float zOff = vPos.z/_Dist;
47.                         vPos += _QOffset*zOff*zOff;
48.                         o.pos = mul (projMatrix, vPos);
49.                     #endif
50.                     #ifdef BEND_OFF
51.                         o.pos = mul( UNITY_MATRIX_MVP , v.vertex);
52.                         float4 vPos = mul (UNITY_MATRIX_MV, v.vertex);
53.                         o.pos = mul (UNITY_MATRIX_P, vPos);
54.                     #endif
55.                     o.uv = v.texcoord.xy;
56.                     TRANSFER_VERTEX_TO_FRAGMENT(o);
57.                     return o;
58.                    
59.                 }
60.  
61.                
62.                 fixed4 frag(v2f i) : COLOR
63.                 {
64.                     fixed atten = LIGHT_ATTENUATION(i); // Light attenuation + shadows.
65.                     //fixed atten = SHADOW_ATTENUATION(i); // Shadows ONLY.
66.                     return tex2D(_MainTex, i.uv) * atten;
67.                     //return tex2D(_MainTex, i.uv);
68.                 }
69.             ENDCG
70.         }
71.  
72.         Pass {
73.             Tags {"LightMode" = "ForwardAdd"}
74.             Blend One One
75.             CGPROGRAM
76.                 #pragma multi_compile BEND_ON BEND_OFF
77.                 #pragma vertex vert
78.                 #pragma fragment frag
79.                 #pragma multi_compile_fwdbase
80.                 #pragma fragmentoption ARB_fog_exp2
81.                 #pragma fragmentoption ARB_precision_hint_fastest
82.                 #include "UnityCG.cginc"
83.                 #include "AutoLight.cginc"
84.                
85.                 struct v2f
86.                 {
87.                     float4  pos         : SV_POSITION;
88.                     float2  uv          : TEXCOORD0;
89.                     LIGHTING_COORDS(1,2)
90.                 };
91.                     float4 _QOffset;
92.                 float _Dist;
93.                 sampler2D _MainTex;
94.                 float4x4 viewMatrix;
95.                 float4x4 projMatrix;
96.                 v2f vert (appdata_tan v)
97.                 {
98.                
99.                         v2f o;
100.                     //o.pos = mul( UNITY_MATRIX_MVP, v.vertex);
101.                     #ifdef BEND_ON  // IS BEND ENABLED
102.                         o.pos = mul ( _Object2World , v.vertex);
103.                         o.pos = mul ( viewMatrix , o.pos);
104.                         o.pos = mul ( projMatrix , o.pos);
105.                         float4 vPos = mul (UNITY_MATRIX_MV, v.vertex);
106.                         float zOff = vPos.z/_Dist;
107.                         vPos += _QOffset*zOff*zOff;
108.                         o.pos = mul (projMatrix, vPos);
109.                     #endif
110.                     #ifdef BEND_OFF
111.                         o.pos = mul( UNITY_MATRIX_MVP , v.vertex);
112.                         float4 vPos = mul (UNITY_MATRIX_MV, v.vertex);
113.                         o.pos = mul (UNITY_MATRIX_P, vPos);
114.                     #endif
115.                     o.uv = v.texcoord.xy;
116.                     TRANSFER_VERTEX_TO_FRAGMENT(o);
117.                     return o;
118.                 }
119.  
120.                 fixed4 frag(v2f i) : COLOR
121.                 {
122.                     fixed atten = LIGHT_ATTENUATION(i); // Light attenuation + shadows.
123.                     //fixed atten = SHADOW_ATTENUATION(i); // Shadows ONLY.
124.                     return tex2D(_MainTex, i.uv) * atten;
125.                     //return tex2D(_MainTex, i.uv);
126.                 }
127.             ENDCG
128.         }
129.         Pass {
130.         Name "ShadowCaster"
131.         Tags { "LightMode" = "ShadowCaster" }
132.        
133.         Fog {Mode Off}
134.         ZWrite On ZTest LEqual Cull Off
135.         Offset 1, 1
136.  
137.         CGPROGRAM
138.         #pragma vertex vert
139.         #pragma multi_compile BEND_ON BEND_OFF
140.         #pragma fragment frag
141.        
142.         #pragma multi_compile_shadowcaster
143.         #include "UnityCG.cginc"
144.         #pragma fragmentoption ARB_fog_exp2
145.         #pragma fragmentoption ARB_precision_hint_fastest
146.                
147.  
148.         struct v2f {
149.             V2F_SHADOW_CASTER;
150. //          float4  oPos        : SV_POSITION;
151.            
152.         };
153.         float4 _QOffset;
154.         float _Dist;
155.         float4x4 viewMatrix;
156.         float4x4 projMatrix;
157.        
158.         v2f vert( appdata_tan v )
159.         {
160.         v2f o;
161.             #ifdef BEND_ON  // IS BEND ENABLED
162.                 o.pos = mul ( _Object2World , v.vertex);
163.                 o.pos = mul ( viewMatrix , o.pos);
164.                 o.pos = mul ( projMatrix , o.pos);
165.                 float4 vPos = mul (UNITY_MATRIX_IT_MV, v.vertex);
166.                 float zOff = vPos.z/_Dist;
167.                 vPos += _QOffset*zOff*zOff;
168.                 o.pos = mul (projMatrix, vPos);
169.             #endif
170.             #ifdef BEND_OFF
171.                 o.pos = mul( UNITY_MATRIX_MVP , v.vertex);
172.                 float4 vPos = mul (UNITY_MATRIX_MV, v.vertex);
173.                 o.pos = mul (UNITY_MATRIX_P, vPos);
174.             #endif
175.             //TRANSFER_SHADOW_CASTER(o)
176.              o.pos = mul(UNITY_MATRIX_MVP, v.vertex);
177.               o.pos.z += unity_LightShadowBias.x;
178.             return o;
179.         }
180.  
181.         float4 frag( v2f i ) : COLOR
182.         {
183.             SHADOW_CASTER_FRAGMENT(i)
184.         }
185.         ENDCG
186.     }
187.    
188.     // Pass to render object as a shadow collector
189.     // note: editor needs this pass as it has a collector pass.
190.     Pass
191.     {
192.         Name "ShadowCollector"
193.         Tags { "LightMode" = "ShadowCollector" }
194.        
195.         Fog {Mode Off}
196.         ZWrite On ZTest LEqual
197.  
198.         CGPROGRAM
199.         #pragma vertex vert
200.         #pragma fragment frag
201.         #pragma multi_compile BEND_ON BEND_OFF
202.         #pragma multi_compile_shadowcollector
203.  
204.         #define SHADOW_COLLECTOR_PASS
205.         #include "UnityCG.cginc"
206.  
207.         struct appdata {
208.             float4 vertex : POSITION;
209.         };
210.  
211.         struct v2f {
212.             V2F_SHADOW_COLLECTOR;
213.         };
214.        
215.         float4 _QOffset;
216.         float _Dist;
217.         float4x4 viewMatrix;
218.         float4x4 projMatrix;
219.        
220.         v2f vert (appdata v)
221.         {  
222.             v2f o;
223.  
224.             #ifdef BEND_ON  // IS BEND ENABLED
225.                 o.pos = mul ( _Object2World , v.vertex);
226.                 o.pos = mul ( viewMatrix , o.pos);
227.                 o.pos = mul ( projMatrix , o.pos);
228.                 float4 vPos = mul (UNITY_MATRIX_MV, v.vertex);
229.                 float zOff = vPos.z/_Dist;
230.                 vPos += _QOffset*zOff*zOff;
231.                 o.pos = mul (projMatrix, vPos);
232.             #endif
233.             #ifdef BEND_OFF
234.                 o.pos = mul( UNITY_MATRIX_MVP , v.vertex);
235.                 float4 vPos = mul (UNITY_MATRIX_MV, v.vertex);
236.                 o.pos = mul (UNITY_MATRIX_P, vPos);
237.             #endif
238.             //TRANSFER_SHADOW_COLLECTOR(o)
239.             o.pos = mul(UNITY_MATRIX_MVP, v.vertex);
240.             float4 wpos = mul(_Object2World, v.vertex);
241.             o._WorldPosViewZ.xyz = wpos;
242.             o._WorldPosViewZ.w = -mul( UNITY_MATRIX_MV, v.vertex).z;
243.             o._ShadowCoord0 = mul(unity_World2Shadow[0], wpos).xyz;
244.             o._ShadowCoord1 = mul(unity_World2Shadow[1], wpos).xyz;
245.             o._ShadowCoord2 = mul(unity_World2Shadow[2], wpos).xyz;
246.             o._ShadowCoord3 = mul(unity_World2Shadow[3], wpos).xyz;
247.             return o;
248.         }
249.  
250.         float4 frag (v2f i) : COLOR
251.         {
252.             SHADOW_COLLECTOR_FRAGMENT(i)
253.         }
254.         ENDCG
255.     }
256.     }
257. }

 

Bump.

Nobody has any idea how to deform shadows at all? i changed my code to affect vertex position instead of pushing it all into the fragment, and still nothing, these shadows wont move. I even set it so that i go back to regular model space with vertices so that it can do its thing and no movement at all... and how is World2Object * Object2World NOT the identity matrix?!?! Im about to give up on the idea of manipulating shadows.... seems like its impossible

 

Sorry to hear you're stuck. I know the feeling :-(

Unfortunately I don't know how to solve you're problem. Just wanted to chim in and ask, if you've considered making your own shadows?

It's not that hard. Set up a camera, that is your light. Render a depth image (with replacement shaders that return the distance to the camera).
Now inside your shader, get the matrix of your light (well your camera), transform your vertice into this space, check if it's further away then what you get from the depthmap. If so it's in shadow.

The downside.. it's all custom from there (no more lightbaking / cast shadow.. etc)
The updsie.. well last resort?

 

Thank you very much for the idea, I am making progress, I've gotten the shadows to at least project on the bent objects, they just arent projecting properly which is to be expected since its calculating the light data after the vertice modification, I would need it to calculate shadows BEFORE I do my vertice changes, which is gonna be another huge task in itself for me. If all goes to hell and I cant do it, Ill look into rendering to depth and try this. Have you implemented anything like this before? How does it perform versus the built in shadow casting.

 

Maybe a shot in the dark. I haven't done any Surface shaders. But have you tried doing it with Vertex/Fragment programs?

Yes, I'm using my own shadow casting lights on a game I'm currently working. I use them because I need shadows on prerendered images (I use a depth texture to determine the point in space).
I haven't compared them to the native shadows, I did notice that they use less draw calls. I assume, it's because the shadows are very fixed, no fade out to depth, no update according to camera position etc.
Also I have scripts that turn them on and off on the shader. So I can have several lights with shadows in scene and only the objects in the lights frustrum calculate the shadow... as I said it's all custom.. but very controllable.

 

I got it to work! weeeee. I was doing it wrong at some places, so to resume :

- I send the vertex into the view space provided by unity, do my changes, then send it back using THE inverse of the mv space(since unity doesnt provide it you have to transpose the UNITY_MATRIX_IT_MV)
-once ur done, send the fragment position into clip space using MVP and return it
-for the shadow caster pass , in my case, I needed the light to calculate BEFORE bending , so in the shadow caster pass you put the TRANSFER_SHADOW_CASTER macro BEFORE everything then you do the same vertex deformation. DO NOT modify the fragment after transfering shadow caster,just return it
-now for the shadow collector pass its basically the same thing, but the TRANSFER SHADOW COLLECTOR before the vertex modification, but this time you have to send the fragment position back into clip space afterwards using MVP(due to the shadows needing to bend as well)

and voila! shadows that conform to deformation, this is just using the default unity projection, but that part is very simple to change. Hopefully this will help anybody that runs into a similar problem, doing shaders inside unity can sometimes be very frustrating. But when you get it work , its all good

 

Actually I spoke too soon, when I use my custom projection it doesnt work, doing MUL (customProjectionMatrix, UNITY_MATRIX_MV) is COMPLETELY different from doing mul (UNITY_MATRIX_P, UNITYMATRIX_MV), even if i do ZERO changes to the customProjectionmatrix from camera, all im doing is SETGLOBALMATRIX and set it at camera.projectionMatrix, can someone enlighten me as to how UNITY_MATRIX_P is calculated? because it is definetly not taken from the camera projectionMatrix ><.

 

Did you ever manage to get this to work? I'm tackling the same problem, getting shadows to cast/collect correctly for a vertex displacement shader.

 

No sorry, I used some other trickery to correct my clipping plane issue, which was the reason why i was using custom projections. I never really got around to look into it. It would be nice to know how these projection matrices are calculated. If you were referring to just casting and collecting shadows without custom matrices then yes my post above tells you how to do it. GL

 

This is probably too late, but the correct way to multiply your model view projection matrix is actually the inverse order
multiply the Projection with the View, and the ProjectionView with the Model matrix. It fixed a problem to me even tho it was visually correct, the clipping planes gets messed up otherwise.
The problem i have now, is that i'm trying to write on to the depth buffer with that matrix. Its working for the shader (clearing the depth buffer, then rendering the object) but for some reason, the shadow casting does not use the same depth buffer, I'll try to make the shadowcast/collector passes, see if it solves my problem :/

 

@LeoTS, any luck? I was hoping to implement something like this. Would you mind sharing your implementation if you did get it working.

 

Yeah, a lot of time has passed since i posted that, and i actually got it working.
What i tried to do is to render my first person weapon in front of every object on screen, but without using an other camera, and receiving (but not casting) shadows in the scene, while being able to even change the first person object FOV separatedly.
So what i did:
Created 3 projection matrices, one with the custom projection (fov and near/far clip planes) for rendering the ambient pass (i got a custom ambient shader), one with the custom fov but the unity's camera near and far (thats important, to render the unity's lighting passes base and add, and using unity's shadows properly), and one with the near and far clip planes depth offset, so i can use transparent objects on the first person layer as well.
In the object shader, the first pass clears the depth with the depth offset matrix (multiply the unity's model, view and the custom projection matrix), then the ambient pass uses the custom fov/near/far parameters (you can have really small near clip planes so your objects can get close to the camera without clipping), and then the unity's lighting passes using the custom fov but the default near and far clip planes, so we get the shadow map coords right, and also writes the to depth so it respect transparent objects.

It was pretty tough to get it working, with many failed attempts, but it's working wonderfully, and its pretty much the same way other games in other engines like Battlefield 3 and 4 did.

The scene is in deferred mode, but the first person objects needs to be forward, if i used deferred on them, the shadows would get messed up since unity shadows in deferred is done in screen space, and the objects have a separated fov (covering different parts of the screen, even with the same model transformation)

I also had to overwrite the unity's Camera-Depth shader, so it would use my custom projection matrix on the weapon, by creating a new render type "CustomMatrixOpaque"

 

Wow, that's a lot to digest. I'll have to try that implementation myself. I'm going to need to work on my shader knowledge. Thanks for the info.

 

@LeoTS, I wasn't quite sure what you meant by this part. Could you elaborate?

 

Sure, so unity uses a hidden shader to get the camera depth texture, that is used in to calculate the shadow map and other effects. You can get the shader source from the unity built in shader source pack "Camera-DepthTexture.shader".
In that shader you can see many subshaders, each with a different "RenderType" tag. So you can copy the opaque version and name it as something like "RenderType"="CustomOpaque", so you can use the matrices you defined on your script. As long as you use Shader.SetGlobalMatrix, every shader will receive the matrix you set. Then just drop the edited depth shader on the project, and make sure you put it on Resources or assign it on an object, to get properly included on a build.