Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Cel Shading w/ Edge Detection

Discussion in 'Shaders' started by IntelligentDesign, Jun 21, 2014.

  1. IntelligentDesign

    IntelligentDesign

    Joined:
    Jun 13, 2014
    Posts:
    51
    Hey all, I'm new to writing shaders (and Unity in general), and I'm looking to build a cartoonish shader for a game I'm starting. And while there's a wealth of information on how to create shaders with ramp-based lighting (a la Team Fortress 2), I have not yet found a good method for handling the outlines. One method I tried involved modifying the Diffuse/Outline shader on the Unity wiki, but the result is very aliased and not quite what I'm looking for. I also found out that Unity has an edge detection script, but unfortunately post processing is only available in Unity Pro and as a college student I can't really drop 1500 bucks on a whim. I'd also prefer to have the effect be shader controlled vs camera controlled anyway.

    Ideally I'd like something similar to the effect outlined here: http://willychyr.com/tag/edge-detection/ (which takes inspiration from here: http://www.thomaseichhorn.de/npr-sketch-shader-vvvv/).

    Is this possible in Unity Free? And more importantly, is it possible for a newcomer to Unity who only has 2 years of programming in general?
     
  2. kebrus

    kebrus

    Joined:
    Oct 10, 2011
    Posts:
    415
    What you want is indeed the edge detection screen effect, even the example you gave screams screen effect

    Not only you have a better control of the effect, it also performs very well, if you use a shader that tries to draw black lines over given angles you'll quickly see how the effect breaks down depending on the shape of the object since the shading ignores the depth of it, there's also the object shader that renders the object a second time bigger to render a black outline that not only looks ugly it doubles your drawcalls

    So, yeah, unity pro :|

    It isn't hard if you give it a serious R&D, and if you go unity pro you can always use the existing unity edge detection script
     
  3. IntelligentDesign

    IntelligentDesign

    Joined:
    Jun 13, 2014
    Posts:
    51
    So I decided to at least start a trial of Unity Pro. And the edge detection script is actually pretty nice. I'm curious if there's a way to modify it to draw lines other than black (I'm guessing the best option is thru a variable in the inspector).

    I've also redownloaded the Strumpy Shader Editor - it was kind of useless before since I couldn't view the model previews without Pro, but it certainly makes writing shaders a bit easier. However, now I have a new wrinkle: I've written a decent ramp shader for my toon effect, but I'd like to also include specular highlights and and rim lighting (similar to the TF2 shader). My problem is that both of these calculations require the light direction vector and the view direction vector, but the light vector doesn't seem to exist anywhere in Strumpy and the view vector is labelled as being unusable in the Lighting Graph (which I know can't be true because when writing shaders by hand I've created custom lighting functions via

    Code (JavaScript):
    1. inline fixed4 LightingCustom (SurfaceOutput s, fixed3 lightDir, fixed3 viewDir, fixed atten)
    2. {
    3.     ...
    4. }
    If anyone knows how I should deal with this I'd really appreciate it. Also, if it helps I can attach an image of my lighting graph as it is right now
     
  4. IntelligentDesign

    IntelligentDesign

    Joined:
    Jun 13, 2014
    Posts:
    51
    Also, is edge detection not possible with deferred rendering? I'd like to have multiple light-casting shadows and a sobel effect, but it seems they can't exist simultaneously
     
  5. hippocoder

    hippocoder

    Digital Ape

    Joined:
    Apr 11, 2010
    Posts:
    29,723
    imo strumpy should be removed from the store - it's far outclassed by Shader Forge, and strumpy really isn't updated any more.
     
  6. IntelligentDesign

    IntelligentDesign

    Joined:
    Jun 13, 2014
    Posts:
    51
    Never having used it, Shader Forge does look like a great piece of software, but it costs about $80 more than I'm willing to spend - college student budget and all. However, after a few hours of messing around I think I can make Strumpy so what I'm looking for.

    My next problem is that, knowing nothing about the differences btw Forward and Deferred Rendering, Unity's edge detection scripts don't seem to work with deferred rendering on. To my inexperienced eye, deferred seems like the way to go since I obviously want multiple shadow-casting lights (as well as point lights which can cast shadows) and that seems to only be possible with deferred rendering. Does anyone know how to get edge detection and deferred rending to play nice together, or if there's some workaround I just don't know about?
     
  7. kebrus

    kebrus

    Joined:
    Oct 10, 2011
    Posts:
    415
    You might be doing something wrong with your project, the edge detection script should be working in both forward and deferred rendering, are your using any other screen effects?

    I just tried it here with a clean project and it works, the only way i see it breaking is if you have another screen effect before or after this one that changes the depth of the camera which the edge detection uses

    cheers
     
  8. jRocket

    jRocket

    Joined:
    Jul 12, 2012
    Posts:
    699
    FYI, you can get toon highlights(sharp edged highlights) by putting the specular calculations through the same way you would do the regular toon ramp shading. ie tex2D (_Ramp, float2(specular)).rgb;

    If you just want TF2 style specular and rim lighting, there's already examples of how to do that in the documentation.
     
  9. IntelligentDesign

    IntelligentDesign

    Joined:
    Jun 13, 2014
    Posts:
    51
    Thanks kebrus, I realized the issue was that I use 2 cameras: a camera which only renders the gun the player is holding (for culling purposes) and the main camera. Once I applied the script to the gun camera instead of the main camera everything played nice
     
  10. IntelligentDesign

    IntelligentDesign

    Joined:
    Jun 13, 2014
    Posts:
    51
    Also, I posted this question in the Answers section and got the following reply:
    I've tried the expanded back faces method, but wasn't happy with the result. What I would like to try is the geometry shader, but I'm not entirely sure how to implement it. Would it be a script that goes on the camera? And what issues might I run into trying to port it into something Unity can understand?
     
  11. kebrus

    kebrus

    Joined:
    Oct 10, 2011
    Posts:
    415
    The "expanded back faces" method is the one i talked previously which not only looks ugly but it also doubles your draw calls because it needs an extra pass for the outline for each object

    That geometry shader doesn't look like a geometry shader at all since it actually uses screen space stuff to do edges. In fact i couldn't understand if the technique was even a geometry shader or simply a screen shader that creates quads, in either way you couldn't do it with unity free because of the whole depth part

    If you really can't afford unity pro it's better if you stick with a simple rim light shader that renders steep angles black, and maybe a combination with the "expanded back faces" technique
     
  12. IntelligentDesign

    IntelligentDesign

    Joined:
    Jun 13, 2014
    Posts:
    51
    Well I'm doing the 30 day trial of Pro, after which I plan on buying a 1 year subscription to Pro (someone on Unity Answers pointed out that since I'm in college I can get the license for $150, which is awesome). That said, if anyone knows a method of adapting that screen shader for Unity I'd really appreciate it - the outlines it shows are amazingly clean and the exact effect I'm going for. Since it gets a bit cut off in the quote I included above, the link is http://prideout.net/blog/?p=54.

    From what I can tell with my limited understanding, the code in that article is sent directly to OpenGL (which I figured out from the glDraw call) and written in C++. I'm not sure then if it's possible to adapt this code to Unity, or if anyone can knows of a shader with a similar look to it. I know I've seen people on the forums talking about modifying shaders from the Nvidia shader library for use in Unity, but those are written in HLSL or CgFX. If I'm getting any details wrong about this I apologize - as I've said I'm still in school for this and what I have studied only covers the very basics involved in all this