Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

[Advanced] Stencil shadows

Discussion in 'Shaders' started by hippocoder, Apr 23, 2014.

  1. hippocoder

    hippocoder

    Digital Ape

    Joined:
    Apr 11, 2010
    Posts:
    29,723
    Hiya,

    In a fit of noob, I haven't managed to figure out the process of using stencil shadows in my game. The game is 2.5D (side view perspective) and the camera will never be in the volume, and the shadows can simply be directional.

    Is there any advice or quick tips you have for this particular setup? Unity's own shadows are way too slow. I tried out Gustav's shadow volume toolkit and it works at a very high speed vs Unity's shadows on device however his technique isn't compatible with runtime generated geometry.

    So any tips/tricks/source is welcome and thanks :)
     
    CharlieSamways likes this.
  2. jvo3dc

    jvo3dc

    Joined:
    Oct 11, 2013
    Posts:
    1,520
    I've used some stencil shadows before, but not in Unity. The result is here on youtube. (The tree shadows are not stencil shadows of course.)

    On the shader side things are not that difficult. If you're not using the reversed depth trick (commonly known as Carmack's reverse) You render two passes. In pass one you render the shadow volumes with normal z-test and without z-write. You disable the culling, so you can count how many times you go in and out of the volume. You set stencilpass to incr and the stencilmask to 1. That way the first bit will flip between 0 and 1 each time you enter or leave the shadow volume. If the final value is 1, you're in the shadow. You'll also need to disable color writes.

    In the second pass you set stencilfunc to equal, stencilmask to 1 again and stencilref to 1 on a full screen shader. This shader then only runs for pixels that are in the shadow. It can be convenient to set stencilpass to decr in this case to reset the stencil buffer to 0 after the shadow has been drawn.

    On the model side of things you have to make a version of your shadow casting object that can be stretched according to the light position. This means that you'll have to insert a quad (two triangles in reality) on each original edge and a triangle on each original vertex. This inserted geometry has zero surface area initially, but allows the model to stretch without introducing artifacts or gaps in the shadow volume. The stretching itself can be done in the vertex shader of the first shadow pass.
    Code (csharp):
    1.  
    2. float3 stretch_direction = normalize(pos_world - light_pos);
    3. if (dot(stretch_direction, normal_world) > 0.0) {
    4.     pos_world += stretch_direction * stretch_amount;
    5. }
    6.  
     
  3. hippocoder

    hippocoder

    Digital Ape

    Joined:
    Apr 11, 2010
    Posts:
    29,723
    Thanks, that's incredibly helpful :)
     
  4. jvo3dc

    jvo3dc

    Joined:
    Oct 11, 2013
    Posts:
    1,520
    You're welcome. Some original documentation on the subject is on wikipedia:
    http://en.wikipedia.org/wiki/Shadow_volume

    The technique I mentioned is essentially the exclusive-or method. I recently used it to mark volumes into a rendered scene, which worked fine.

    When you have it working, the trick is to minimize the stretch_amount to minimize the fill rate. Using the depth pass method above is generally faster than the depth fail (Carmack's reverse) method. With the restriction that you need to know whether the camera is inside the shadow volume in the depth pass method.