Programmable rendering pipeline? Why not?

Hi there,

I looked over some of the features outlined in Crytek's new engine and took notice of their "Directional Occlusion" shader.
Meanwhile I've looked over Unity's rendering pipeline in the documentation and have been been fiddling around with an ambient occlusion shader.

The thought occurred to me: Why are we rendering ambient occlusion overtop of the scene at the very end, when ideally it ought to be multiplied with the ambient layer in the deferred rendering pipeline? It also occurred to me that directional occlusion could be implemented easily in a similar way, per light, using the light direction value as input to the very same AO shader - and that'd put Unity on par with CryEngine 3 as far as lighting is concerned.

I wish it were possible to access and customize the deferred rendering pipeline at the point when lighting is calculated, so we could implement our own lighting effects. Perhaps you could write lighting shaders and drag them onto lights, similar to how you can with materials and cameras.
In fact it'd be nice if we could write our own pipeline from the ground up, with input and output nodes like how it is done in Substance's editor or in most CG movie renderers.

I presume Unity's business strategy with it's engine is to be largely driven by user-created content (asset store) so it wouldn't be a huge stretch to say that one ought to strip down Unity until it is almost a bare GUI and then let users build it back up again. What's already there is awesome, but having therein the same level of control that we have over Unity's standard assets would make it that much more awesome.
And likewise, letting users create assets that are as powerful as Unity itself by giving us more control over the back end would make the asset store that much more powerful. It'd give Unity that much more of an upper hand in emerging on top as the best engine in the world.
After all, even though companies like Crytek and Epic have millions of dollars to produce engines, it doesn't compare to thousands of educated, talented people driven by their passions. I can already think of a few such people who have released gems on the asset store.

 

Because Unity employs graphics hardware for rendering and the programmability of common graphics hardware is limited.

Also, you might want to check out http://unity3d.com/support/documentation/Components/SL-SurfaceShaderLighting.html and http://unity3d.com/support/documentation/Components/SL-ShaderPrograms.html .

 

I'm not sure how carefully you read or thought about what I wrote. As far as I'm aware, any given rendering pipeline is written in software which then tells the hardware what to do. Specifically, deferred lighting is just passing a render texture (API call; software) with normals, depth, and specular factors to a shader (software) which then calculates lighting and adds the results of each light together, adds ambient (shader; software) and multiplies it with an albedo render texture (API call and shader; software). This is all shader work with software-controlled render calls and software-controlled switches between render textures. Hardware takes over when software tells said hardware to render geometry to those render textures or apply shaders. If I'm mistaken please correct me and I'll shut up but as far as I know this is basic engine knowledge and entirely programmable.

Custom lighting models as described in the link you gave me are only employed in the forward rendering pipeline. Forward isn't an ideal place to introduce effects like ambient occlusion, directional occlusion or indirect lighting while a deferred pipeline is.

Ambient occlusion - a shader applied to an ambient term render texture.

Directional occlusion - a shader which multiplies an occlusion term to the computed lighting result using the light direction as input. (Possible alternative or suppliment to the soft and hard shadowing in the light's drop-down.)

Indirect lighting - a shader samples albedo * lighting using a gathering method similar to ambient occlusion and adds it to the lighting term.

Current deferred pipeline:
1. Mesh rendering passes - renders normals, depth, specular terms, and albedo in a forward manner onto multiple render textures.
2. Lighting passes - calculates lighting for each light and saves the result to a temporary render texture (presumably with alpha reserved for specular).
3. Shadowing - multiplies shadow factor calculated for the current light.
4. Add to sum - adds the result of this (and all other) light(s) into a accumulated Lighting render target.
5. Ambient light - adds ambient to the lighting texture.
6. Diffuse and Specular - multiplies albedo with Lighting.rgb, adds Lighting.a.
7. Forward rendering of materials that don't support deferred.

Modified deferred pipeline:
1. Mesh rendering passes - renders normals, depth, specular terms, and albedo in a forward manner onto multiple render textures.
2. Lighting passes - calculates lighting for each light and saves the result to a temporary render texture (presumably with alpha reserved for specular).
3. Shadowing - multiplies shadow factor calculated for the current light.
4. Directional occlusion - calculates and multiplies DO factor for the current light.
5. Add to sum - adds the result of this (and all other) light(s) into a accumulated Lighting render target.
6. Ambient Occlusion term - references normals and depth to calculate AO.
7. Ambient light - adds ambient * AO to the lighting texture.
8. Indirect lighting - gathers samples from the lighting and albedo render textures to compute indirect lighting, adds it to lighting render texture.
9. Diffuse and Specular - multiplies albedo with Lighting.rgb, adds Lighting.a.
10. Forward rendering of materials that don't support deferred.

 

On http://unity3d.com/support/documentation/Components/SL-SurfaceShaderLighting.html I'm reading:
"half4 LightingName_PrePass (SurfaceOutput s, half4 light); This is used in deferred lighting path. "

It might be possible to include the "directional occlusion" here. (Or in the functions for directional lightmaps.)

I agree that reusing the normal and depth buffer (that was computed for deferred rendering) for the purpose of screen-space ambient occlusion would have benefits. (Also some disadvantages, e.g. the implementation would be limited to deferred rendering.)

I wonder whether it would be possible that you write your own deferred rendering pipeline within Unity using multiple passes with shader replacement?

 

That is already the case if i'm not mistaken.

It definitely is. The shaders used in internally for the pre pass are included in the buil-in shaders file, yo you could also have a look what unity does on the shader side. Our you could probably change the files and use them as a replacement.

 

What I think he meant was the other buffers that get combined... specifically the lighting texture, before ambient light is added. There should be a way to intervene and multiply the ambient color with something per pixel, so only ambient light is affected.

As for being limited to deferred - you could use the regular ambient occlusion methods we use now as a fall back.

Here's what directional occlusion looks like: http://s3-eu-west-1.amazonaws.com/kayru-s3/dssdo/figure2.png

 

There is some options for customization by editing the cg-Includes and BuiltIn shaders, but at some point your're limited by the current rendering setup. Some more flexibility is planned and I had a lengthy talk to Tim Cooper (@stramit) at the Unite Nordic '13 on things like adding a callback "OnPrepassRender()", ConfigurableLights that allow you to pass custom variables and set shader KeyWords per light and many other things.

In the meantime I'm working on a customizable render path replacement that might be sold on the AssetStore later this year.

PS: I successfully implemented directional occlusion per Light in the Prepass-Shader - see my tweets @ManuKman ...