Physically based shading – open source framework

hi there,


i have been working on a physically based shader framework for a while and now would like to share the first preview with you as i plan to release it as open source.





there are already some commercial solutions out there available at the asset store, but none of those is free, some even break other shaders. so it might be rather difficult for the community to develop and improve unity’s rendering capabilities. and that is the whole idea behind this framework.

so any feedback, tips and tricks are highly appriciated.
and in case you find any bug just let me know.

lars


test the webplayer
http://bit.ly/1dAiHxN
download the latest package (version 0.021)
http://bit.ly/1bYmtzV


the framework is mainly based on the work of sebastian lagarde and dimitar lazarov:
http://seblagarde.wordpress.com/2011/08/17/hello-world/
http://blog.selfshadow.com/publicat...e/lazarov/s2013_pbs_black_ops_2_slides_v2.pdf

it supports forward and deferred lighting, lightmaps and lightprobes and handles gamma and linear lighting – although linear lighting is more than a key to physically based shading.


direct lighting
- specular: phong normal distribution function (in a range of 0 – 2048 so you will get much smaller and sharper highlihgts)
- visibility: schlick-smith
- fresnel: schlick-fresnel approximation


ambient lighting
- diffuse cubemap IBL
- specular cubemap IBL (gloss values stored in mip chain) with fresnel term based on roughness and specular color
both cubemaps can be sampled in gamma and linear space. support for RGB or RGBM (HDR) cubemaps.


deferred rendering
to make the shader work in deferred rendering i had to hack the "Internal-PrePassLighting“ shader, as i could not find another way yet to get the dotNL and especially the dot(h, lightDir) product. that is not as elegant as i would like to have it.
and as i had to „compress“ the spec value in the "Internal-PrePassLighting“ shader (log2) and decompress it in the lighting prepass function (exp2) in order to avoid wrong shaded spots in the center of the highlights, built in shaders will be broken (too less specular) when using deferred lighting. so i highly recommend to rebuilt all shaders using the Lux lighting functions.


ambient ibl
i did not spent much time on this part of the framework and it is pretty much straight forward using common rbgm decoding and a roughness based fresnel term on the specular distribution.
i also added energy conservation but could not find any information about whether this is needed or even correct.


inputs
i have chosen an approach suggested by sebastian lagarde and only have a few parameters to feed the shader right now:
http://seblagarde.wordpress.com/2011/08/17/feeding-a-physical-based-lighting-mode/

- diffuse color as texture rgb
- specular color as texture rgb
- roughness (alpha in specular color)
- normal map
- diffuse and specular ambient cube (most of the time passed as global textures by script)

one might miss parameters like metalness but as we store the specular color values and roughness in a rgba texture we can combine dielectrics and metals into one material potentially saving a lot of draw calls.

 

That looks fantastic! Well done! I am looking forward to try it out. Unfortunately I don't have the time to check it in detail at the moment.

 

thanks!
and i am looking forward to read what you think about it.

lars

 

My God...you did it again dude, this can be a community project if people able to customized it and adding stuff...

 

hi rea, thanks.
and in fact this is what i have in mind.

writing customized shaders should be pretty easy as there are common surface shaders more or less just relying on 2 simple includes:
- LuxLightingPhong.cginc
- LuxIBL.cginc

in fact the package already ships with a bumped diffuse, bumped specular, bumped specular transparent, bumped specular transparent cutout and a bumped specular self illuminated transparent coutout shader – all written in about 5min each…

e.g. the base bumped specular lux shader looks like this:

Code (csharp):

1.  
2. Shader "Lux/Bumped Specular" {
3.  
4. Properties {
5.     _Color ("Diffuse Color", Color) = (1,1,1,1)
6.     _MainTex ("Base (RGB) Alpha (A)", 2D) = "white" {}
7.     _SpecTex ("Specular Color (RGB) Roughness (A)", 2D) = "black" {}
8.     _BumpMap ("Normalmap", 2D) = "bump" {}
9.     _DiffCubeIBL ("Custom Diffuse Cube", Cube) = "black" {}
10.     _SpecCubeIBL ("Custom Specular Cube", Cube) = "black" {}
11.    
12.     // _Shininess property is needed by the lightmapper - otherwise it throws errors
13.     [HideInInspector] _Shininess ("Shininess (only for Lightmapper)", Float) = 0.5
14. }
15.  
16. SubShader {
17.     Tags { "RenderType"="Opaque" }
18.     LOD 400
19.    
20.     CGPROGRAM
21.     #pragma surface surf LuxPhongNDF dualforward noambient
22.     #pragma glsl
23.     #pragma target 3.0
24.     #include "LuxCore/LuxLightingPhong.cginc"
25.        
26.    
27.     #pragma multi_compile LUX_LINEAR LUX_GAMMA
28.     #pragma multi_compile DIFFCUBE_ON DIFFCUBE_OFF
29.     #pragma multi_compile SPECCUBE_ON SPECCUBE_OFF
30.  
31.     float4 _Color;
32.     sampler2D _MainTex;
33.     sampler2D _SpecTex;
34.     sampler2D _BumpMap;
35.     samplerCUBE _DiffCubeIBL;
36.     samplerCUBE _SpecCubeIBL;
37.    
38.     // Is set by script
39.     float4 ExposureIBL;
40.  
41.     struct Input {
42.         float2 uv_MainTex;
43.         float2 uv_BumpMap;
44.         float3 viewDir;
45.         float3 worldNormal;
46.         float3 worldRefl;
47.         INTERNAL_DATA
48.     };
49.  
50.     void surf (Input IN, inout SurfaceOutputLux o) {
51.         fixed4 diff_albedo = tex2D(_MainTex, IN.uv_MainTex);
52.         fixed4 spec_albedo = tex2D(_SpecTex, IN.uv_MainTex);
53.         // Diffuse Albedo
54.         o.Albedo = diff_albedo.rgb * _Color.rgb;
55.         o.Alpha = diff_albedo.a * _Color.a;
56.         o.Normal = UnpackNormal(tex2D(_BumpMap, IN.uv_BumpMap));
57.         // Specular Color
58.         o.SpecularColor = spec_albedo.rgb;
59.         // Roughness – we just take it as it is and do not bring it into linear space (alpha!) so roughness textures must be authored using gamma values
60.         o.Specular = spec_albedo.a;
61.         // we do not need and write to o.Gloss as it is not available in the Internal-PrePassLighting shader
62.        
63.         #include "LuxCore/LuxIBL.cginc"
64.     }
65. ENDCG
66. }
67. FallBack "Specular"
68. CustomEditor "LuxMaterialInspector"
69. }
70.  

i guess i wil have to add some more options like ambient occlusion which should also effect the image based ambient lighting and therefor must be incorporated into the LuxIBL.cginc too but that could easily be done using some #def instructions for the compiler.
so in the end i hope to provide a very flexible framework that will enable people to tweak or even write their own shaders and maybe share them with the community.

lars

 

Man this kind of thing is make me wonder why we don't have convoluted mipmap mode in texture importer, could be great for cubemap...

 

that is for sure…

 

Looks cool.

How are you intending to deal with shadows? I was looking into this a while back and that was a bit of a stumbling block for me. Short of picking a spot on the IBL image to be the dominant light source and removing the corresponding information from it to be replaced by a light.

 

hi farfarer,
thanks for the reply.
and you are completely right: real time shadows do not "shade" the specular ibl ambient lighting which doesn’t look very convincing.
and to be honest: i do not have i clever clou right now to get around this. maybe adding specIBL to o.albedo instead of o.Emission might already does the trick?

[edited]well, it adds shadows but a lot of other problems too.
so that is not the solution and thinking about it is pretty clear that it could never be…

lars

 

Fantastic work. In fact there is Jove free physically based shader but free ver has some limitations and no nice custom editor so your shader has chance to be better option than Jove

 

thanks breyer.
in fact jove never worked for me at all but on the other hand the commercial package ships with a complete material editor and cubemap tool as far as i can say. so it is probably not really fair to compare both solutions.
anyway, i am glad that you like it and hope that this framework will get some great contribution from the community to make it a real alternative,

lars

 

Hey guys, i somewhat interested with this topic.
Can you explain what do you mean by "deal with shadow", How did the shadow behave in PBS/PBL actually?

 

Shadows in PBS are dealt with exactly as they are now.

The issue comes from IBL - where objects are lit using an environment image. In cases like this, there isn't a set direction or point in space that light comes from (and if there is, it's usually not easily separable from the rest of the image).

So you have to somehow introduce shadows for the most dominant light source, while not excluding the light that comes in from other directions.

You could pick a point in the environment map and then either remove that light source from the environment image and add it in again using a regular light, or a secondary environment texture that you can mask using shadows.

The problem is that any way you try and do it will be a bit of a cheap hack.



For what it's worth, this is about as far as I got with IBL - mostly it was a test for writing a shader that would use equirectangular maps rather than cubemaps.

 

I had a pleasure to play a bit with early version of Lux and this is quite cool toy. First of all - this is way more "open" comparing to other engines. Modifying shaders to get PBL/IBL is very easy. And it's free ? Lars is so generous (again) .

For shadows to damp IBL term we can do it in lighting function I guess. In RTP3.1, besides of this issue I introduced damping value passed in custom SurfaceOutput struct. This way don't loose any ambient lighting when applying POM self shadowing for example. In RTP3.0 it was in fact done naive way (damping Albedo output from surface shader).

In simpliest case if you know that o.Emission comes from IBL (diff+spec) alone you can damp it inside lighting function using atten value. By default SurfaceOutput struct is passed as "in" for lighting function but we can specify "inout", too. Then when you modify Emission term in lighting function, the modified version will be applied at the end of surface shader pipeline. Of course this IS cheap trick, because we'd need real ambient occlusion mask, not only realtime shadow IBl damping. But this technique applied a tiny bit (I'd recommend to not damp completely IBL Emission term in lighting function even if atten is 0) will surely improve look. Another problem is (as usual) deferred where we don't have explicit atten value available...

Tom

P.S. We'll need some kind of cubemap importer hack like Skyshop has to not allow Unity making mipmaps for our cubes (convolution filter works much more precise and complex way comparing to just averaging 2x2 pixels for subsequent mip levels). In skyshop it's a function called in editor importer class (to "fix" screwed mip levels use Reimport option for cubemap asset).

 

I had a pleasure to play a bit with early version of Lux and this is quite cool toy. First of all - this is way more "open" comparing to other engines. Modifying shaders to get PBL/IBL is very easy. And it's free ? Lars is so generous (again) .

For shadows to damp IBL term we can do it in lighting function I guess. In RTP3.1, besides of this issue I introduced damping value passed in custom SurfaceOutput struct. This way don't loose any ambient lighting when applying POM self shadowing for example. In RTP3.0 it was in fact done naive way (damping Albedo output from surface shader).

In simpliest case if you know that o.Emission comes from IBL (diff+spec) alone you can damp it inside lighting function using atten value. By default SurfaceOutput struct is passed as "in" for lighting function but we can specify "inout", too. Then when you modify Emission term in lighting function, the modified version will be applied at the end of surface shader pipeline. Of course this IS cheap trick, because we'd need real ambient occlusion mask, not only realtime shadow IBl damping. But this technique applied a tiny bit (I'd recommend to not damp completely IBL Emission term in lighting function even if atten is 0) will surely improve look.

[edit] We actually HAVE atten available for deferred, but the problem I'm thinking of is how to treat it per object/material (as this lighting model is applied globally for everything).

Tom

P.S. We'll need some kind of cubemap importer hack like Skyshop has to not allow Unity making mipmaps for our cubes (convolution filter works much more precise and complex way comparing to just averaging 2x2 pixels for subsequent mip levels). In skyshop it's a function called in editor importer class (to "fix" screwed mip levels use Reimport option for cubemap asset).

 

Hi Lars, been trying your shader and i must said it's awesome, couple thing that i don't understand is, what do you mean by visibility in direct lighting.
and when the object are at full roughness shouldn't the specular should be gone completely?

Edit : Scratch the question about roughness and specular i guess i was wrong....sorry

 

hi there,

thanks for all the feedback.
and sorry for me having been so quite during the last days: i have been very busy refactoring the whole lighting and fixing all the bugs tom has been so kind to point me to. thank you very much for this incredible help tom!

so there is a new webplayer and a new package available and i hope the new version now can truely claim something like physically based shading…

webplayer: http://bit.ly/1dAiHxN

[edited]
get the latest package (version 0.021) here:
http://bit.ly/1bYmtzV

changelog
- deferred lighting
-- refactored
-- visibility term added
-- fresnel fixed (direct lighting): fresnel according only to ONE directional light, light position has to be passed by script
see: "Lux Bumped Specular" shader

- forward lighting
-- refactored
-- visibility term fixed

- lightmaps
-- lighting fixed when using lightmaps or lightprobes

lars

 

@ farfarer:
equirectangular maps – can you explain it a little bit more? what are the advantages/disadvantages using those? and what the heck are those at all?
and your images look just great!

@tom:
your suggestion about adding real time shadow to image based ambient reflections sounds very interesting. but if i get you right it wll only work in forward rendering, right?
things are so much easier in forward rendering (and more accurate in lux right now) but thinking about physically based shading i ask myself which render path would one choose when working on a project using such an expensive shading solution like lux?
personally i have focused on deferred / linear lighting / hdr but of course one might go with forward / linear lighting / hdr.
what do you think?

lars

 

try out the latest version: it’s much better and less buggy.

lars

 

uups, there has been one little but pretty annoying bug in the latest version:
it should be: o.mainLightDir = mul( (float3x3) _World2Object, -Lux_MainLightDir);
in the vertex function.

i have just updated the package and webplayer, please redownload it if you have already updated. sorry.

lars

 

Great project! In the demo web-player (Win7, Nvidia Quadro with DX11/GL4.x), after having hit 3 for sun rotation a couple of times, when I switch back and forth between Forward and Deferred (1 and 2), I see lots of differences not just in specular highlight (which I guess is normal) but also in not directly-lit (ambient / "in shadow") faces which don't seem to happen at all in the Forward case..

 

Got pink color on some shader with the latest version, i already try to remove the old version and reimporting the update but still got an error

 

hi rea,

thanks for pointing this out.
i have totally forgotten to recompile all shaders to see if they would work with the latest modifications: they did not.

but now i have tweaked all included shaders so they compile correctly.

get the latest package (version 0.021) here:
http://bit.ly/1bYmtzV

lars

 

hi metaleap,

i can’t reproduce what you have written. so may be you are looking at the lightmapped spheres?
lars

 

For deferred we've got only an option to get info of atten variable in internal pass, so only direct lighting diffuse/specular term can be damped there (either PBL or not). Unfortunately we don't have this option for IBL which baked to Emission term in surf function. We could eventually think about using light.a in _prepass lighting function which stores specularity strength. Relying on this we could at least damp a bit IBL spec emission term there (as diffuse IBL is mixed up with spec IBL we'd actually damp them both).

Deferred is mostly welcome for plenty of dynamic lights present on scene and this is often the case for certain types of projects (forward would be no option there).

Tom

 

See here: https://dl.dropboxusercontent.com/u/136375/img/screens/unity-weblux.png

Left is Forward (1), right is Deferred (2).

Quite the difference on the right-hand "unlit" (I presume) red-cube-face, no?

 

hi there,

thanks for the image.
but it is as i have written: the big red cubes and the orange flat box are lightmapped. just like some of the spheres.
i think i have baked dual lightmaps which are not properly supported by forward lighting.
so luckily that is nothing to worry about.

lars

 

Oh cool then I just missed that. Good stuff, I never care for lightmaps so no prob for me

 

Awesome i'll test it

cheers

 

Awesome project! It's nice to see a free IBL solution.

 

i am glad you like it.
but it is not about IBL but PBS!
IBL is just one littel part of it…

lars

 

You might want try adf.ly for the links Yeah, adf.ly-links are annoying but this way you could earn something and still keeping the Framework free.

 

hi there,

the last version of lux shipped with a "correct" fresnel for just ONE directional light when using deferred lighting.
i am still working on a correct fresnel for all lights in deferred lighting (pure cg and no surface shader...) but meanwhile i have finished a version using some kind of "faked fresnel" – well: it is not that bad and at least as good as skyshop’s fresnel – that works with all kind of lights and does not need any input by script as it simply relays on NdotV. that is much more flexible and will allow you to create indoor scenes as well as outdoor scene but just not physically correct.

so get this version here:
http://bit.ly/1b4jir6



lars

 

I hope i dont seem too ignorant here, but i have to ask.... does IBL really bring that much too the table? I mean really.... PBS should be a pretty large boost to realism alone i would think... or even more important, predictability when setting up lighting. Also, deferred rendering just seems to have so many gotchas. Forward i understand can take more horse power but its so much more flexible it seems. And cant you just render your most important lights in Forward? And possibly use vertex or spherical harmonics for the rest? Maybe I need to look at more examples.

 

IBL would allow for cube maps that have been baked in Mental Ray or V-Ray to provide photometric lighting for static objects.

 

You're right that IBL lighting model isn't strictly necessary for physically-based BRDF-driven surface shading model. But it can look extremely stunning in terms of illumination (with or without BRDF/PBS) with fairly little overhead

Indeed deferred is a bit overhyped and forward is quite fine, it's just smart to only compute illumination only once per screen-space pixel instead of many times for potentially overlapping triangle fragments. With today's early-Z GPUs, less of an issue but still. Historically, if you're doing any screen-space stuff at all (such as post-FX) and have native depth-buffer access and ideally no transparent surfaces, it makes a lot of sense to put illumination in there too. But with Unity doing it's "depth texture rendering" and then doing it's post-fx screen-space things in additional stacked render-textures, it definitely adds a lot of overhead. So for the time being myself I'm also trying to go with Forward.

 

hi there,

@sixto1972: you do not have to use IBL when using lux as you can simply turn IBL off on a per material basis.
but in fact it really enhances the visual impact and is the only possibility to add ambient specular lighting and reflections.
however one could replace IBL ambient diffuse lighting by using either "hemispheric" lighting (lerping between sky and ground color based on worldNormal) or spherical harmonics instead.

@metaleap: i am not sure if deferred is overhyped or not. but unity has definitely ranked it over forward when it comes to real time lighting.
nevertheless forward is much more flexible and allows "correct" pbs (as it allows "correct" fresnel) whereas deferred either does not support fresnel at all, has to use a faked fresnel or has to cope with pretty heavy quantization artifacts due to unity’s very slim g buffer… (not to mention: no correct colored specular highlights in deferred).
so forward will give you much nicer results and simply works with all other shaders you might use in your project.

lars

 

Is there a Github repository for these shaders?

 

unfortunately: no.
but you will find the latest version here:
http://forum.unity3d.com/threads/221985-Lux-%96-an-open-source-physically-based-shading-framework

lars

 

Physically Based Shading seems quite easy to look at - nice results here. But what are the real advantages beyond just artist workflow? I've yet to see a situation where PBS look cannot be done with a classic shader, albeit with some tweaking. I'm wondering if the performance hit is worth it.

 

A water shader is a good example...

How does the surface of a water behave? You can see through it when looking directly from above, with little to no reflections. On the other hand, at grazing angles it behaves like a mirror and you can't see anything below the surface.

To simulate these properties, you need a physically based shader in one form or another. You must simulate the fresnel effect as well as ensure energy conservation - the more of the environment is reflected, the less transparent the surface needs to become.

A classic shader will never look correct

 

Water perhaps isn't the best example though as water is classically a hand-tailored shader different to every other shader in the game. You'd probably want to go a lot further than just PBS for that.

 

hi hippocoder,


that is a good question but i do not have any answer which would be based on personal experiences as i am just a one man show. so i have to refer to what others have written about pbs.




realism
PBS will give you much more convincing results than traditional lambert or blinnphong shading: no more fights with washed out specular highlights or boring dark areas: everything is shiny!


flexibility
psb will help you to get much more constant and predictable results under varying lighting conditions: author once, use everywhere!


there are some very impressing videos showing the metal gear solid ground zeroes trailer at night and at day time:


night time
http://www.youtube.com/watch?v=ltH1eWxZutE


day time
http://www.youtube.com/watch?v=FQMbxzTUuSg
around 1:02:13


the costs
you can break down lux like most other solutions into several „independent“ parts and just pick those you want or need.


direct lighting
- normalized blinn phong: pretty cheap and absolutely necessary.
- visibility: expensive but could be dropped with only a small drop in quality
- fresnel: expensive but could be dropped also

ambient lighting
- ambient diffuse IBL: more lively ambient lighting but could be dropped
- ambient specular IBL: necessary for reflecting materials

 

This is the greatest gift to the community I've seen! Looking forward to cubemap blending and dynamic interpolation.
I have a few questions:

- I didn't notice any effect of DiffuseIBL on metallic surfaces. Is it ok? I do not fully understand physics here but reading the code gives me the sense there should be some effect in output Emission color but I don't see it.

- I'm trying to achieve bright and contrast metallic reflections like in UE4 temple demo but I can not. It seems there is some difference in metallic reflections: UE4 reflections are barely noticeable at non-metallic surfaces and very contrast at metallic

while in LUX Metalness parameter only affect reflection color (Lerp between specular color and the diffuse albedo color based on "Metalness"). What do you think should I change to achive similar effect?

 

metals do not have any diffuse, so it is absolutely ok the way it is.

hmm, lux preferred workflow is specular reflectivity not metalness (as most sample shaders support specular reflectivity). nevertheless both workflows should give you the possibility to achieve reflections like those in the temple demo.

and the lux metalness does NOT simply lerp between specular color and diffuse "albedo" – although it might look like this ;-)
what it does: on non metal parts of the surfaces it takes the specular color from the blue channel of the "Metalness (R) AO (G) Spec (B) Roughness (A)" texture (as dielectrics or non metals always have a none colored specular color which simply fits into a single color channel) whereas on the metal parts of the surfaces it takes the color from the diffuse texture and sets the diffuse to black (as metals do not have any diffuse). doing so should be physically correct.

but getting back to your question about contrast reflections: you will only get those on metal surfaces as on dielectric surfaces the incoming light is spilt into diffuse and specular and as lux is energy conserving any kind of diffuse "reflection" will lower or darken the specular reflection.

for further questions please come to the new thread:
http://forum.unity3d.com/threads/235027

there you will find the latest nes, a link to the githib repo and more.

lars