Search Unity

Stencilless masking alternative for smooth edges in UI

Discussion in 'Shaders' started by Thomas-Mountainborn, Aug 27, 2015.

  1. Thomas-Mountainborn

    Thomas-Mountainborn

    Joined:
    Jun 11, 2015
    Posts:
    501
    Hi there,

    As with everyone else who's using masks in UI stuff, I'm annoyed with the terrible aliasing caused by the stencil buffer approach. If I apply a post process AA on the UI camera to smooth the jagged edges, all text will be blurred, so this is not a good solution. I could split text and images into separate layers so each can be rendered separately, and mask both elements simultaneously on both layers - but this is just not a very sustainable way of working.

    I was just writing this thread to ask if any of you had a decent solution, when I had the brainwave of rendering the masks to a render texture and then using said texture to set the alpha of the masked graphics. This approach still requires two cameras however, and you can't see the masks in edit time, which is still very annoying (unless there is a way to have a camera rendering to a render texture also render at edit time?). At least you can mask both text and graphics simultaneously with a single mask object.

    I also had the following ideas, but I couldn't get any of them to work because of my inexperience with shaders. I'd love to hear from you if these are feasible, or if you have better implementations for smooth UI masks.

    1. Read the neighbouring pixels' stencil value and fade the alpha based on that. The problem seems to be that you can't read that stencil buffer as a texture. Is that correct, or is there a way after all?
    2. Only render the graphics over the mask using blend modes. The issue is that I can't seem to completely cull the source graphics no matter the blend mode - is this because there is always a value to blend with?
    3. Use a grab pass in the masked graphics after rendering the mask first, and use that information to set the alpha value of the masked graphics. The problem here is again that everything else is already in the frame. Is there a way to only have the mask in a grab pass?