5.3 Standard PBS is Wrong Cannot Eliminate/Remove Specular

As it stands, it is impossible to achieve a matte material, which is rather embarrassing for Unity, please fix this asap.

Changing the UnityStandardConfig.cginc makes no difference to specular response from lights (or global / reflection probes for that matter)

#ifndef UNITY_GLOSS_MATCHES_MARMOSET_TOOLBAG2
#define UNITY_GLOSS_MATCHES_MARMOSET_TOOLBAG2 0
#endif
#ifndef UNITY_BRDF_GGX
#define UNITY_BRDF_GGX 0
#endif






When smoothness is at 0 (zero) there should not be ANY specular either from lights or Reflections period.

BTW. Reflection Specular (Not light specular) Can be eliminated with Standard Specular Setup shader if specular component is set to black, this is ALSO incorrect.

This issue is very bad as it makes it impossible for PBS to be used with Terrains etc. Why in gods name tie it to another third party product such as Marmoset toolbag anyway... Do not add 0.04 adjustment to smoothness curve.

Cheers,
Michael
Livenda

 

I thought everything has specular?

 

not matte materials (they do have specular however they fall to almost zero) . No need to debate this issue as it's a no-brainer. As it stands you simply cannot do a matte material. Please note this is TWO issues, one, the reflections should be cut down, two, the 'specular' from lights should be cut down.

 

Ummm this car sure looks a lot like your material up top.

 

Shouldn't you fill a bug instead? Posting in general means you want to discuss.
What you say/think about pbs does not match info I saw from the other sources.
Afaik, 0 smoothness does not eliminate highlights (not phong's specular level) and all materials are reflective at extreme angles (due to fresnel term being used).

Also see:
http://www.marmoset.co/toolbag/learn/pbr-conversion
http://www.marmoset.co/toolbag/learn/pbr-practice
https://www.marmoset.co/toolbag/learn/pbr-theory
http://forum.unity3d.com/threads/official-5-0-pbr-calibration-charts.289416/

 

Or just use a Diffuse texture, that definitely gets rid of the washed out specularity effect lol.

 

No it doesn't at all.

Like I said, Unity ADDS an artificial value of 0.04, This is easy to replicate in Unreal Engine if you have access to it. Which does NOT suffer from this issue.

I don't think you understand what I have posted, you cannot just put a ' Diffuse texture' this is a global light/PBS issue.

Like I said again, Unity ADDS an artificial value of 0.04, This is easy to replicate in Unreal Engine 4 if you have access to it. Which does NOT suffer from this issue. When roughness is at 1 (inverse of smoothness) , the ' Fresnel ' reflectivity goes to almost zero.

 

Fair enough.

 

@LIVENDA - Umm the car and that material do look a lot a like I don't know what you're looking at, both have almost the same exactly roughness, okay the car has less roughness making it more shiney, however as @neginfinity has stated - there isn't one single thing on this (planet at least) that doesn't have some form of *roughness* to it, nothing whatsoever.

So Unreal must have the wrong lighting calculations if it can ignore the laws of lighting physics which therefore makes it an inadequate product not worth using or u'd be using it instead of Unity.

And I know what you meant by the global light/PBS stuff, it isn't an issue - it's based on reality, if you don't like reality use a Diffuse and get no roughness and strictly specular map control over what you want your object to do so therefore you don't have to rely on PBR shading which to you is faulty and it's not.

Heck even a dust particle has a normal map and roughness (speaking in game dev terms here). An Aluminum can has roughness, that matte car above has roughness, heck even my water even has DYNAMIC ROUGHNESS - something that is amazing to see if you study water and learn how it works. So beautiful.

So I'm not exactly sure what you're complaining about when it's supposed to have roughness either way, if you do *not* want roughness then don't use PBR and stick to older Diffuse and Specular texture formats. There's absolutely nothing wrong with the current PBR System other than baking taking so long.

And if all else fails - just go back to Unreal if it's that much of an issue.

 

To clarify, I think some are misunderstanding what I'm trying to say. Believe me I understand that ' Everything ' has a certain 'fresnel' reflection value, even charcoal (one of the least reflective substance) has reflective propperties at high grazing angles. However, in the metallic setup standard shader, there is no way to create a pure non-reflective surface.

 

Hi Livenda, slightly off topic but are you still working on real time area lights inside unity or have you already have a plugin that works like on your youtube video?

 

Typical answer, if you don't like it go to Unreal! Jesus, I'm trying to point out an issue here. Just go ahead and try doing a terrain using the default standard shader, It is reflective, over all! not just at grazing angles. Also, BTW, using legacy Diffuse with Deferred does not eliminate the issue.

 

Use Specular instead of metallic, metallic is made purely for pure metallic materials (no paint, pure metallic no albedo texture other than from black to white range (0,255).

At least that's what I've always been told. So a black matte car would be considered Specular not Metallic, at least in terms of the PBR Shading system. Both produce minute differences but they are different.

Well you mentioned that Unreal can fix the issue, that's why I said just use Unreal.
it's a typical response because why complain when you can just use one or the other - both have their pros and cons to which is better suited for you're needs.

But about the terrain thing, why should it not have a certain roughness to it if it's supposed too?
Sorry I'm not trying to fight with you, just don't see the issue.
Why wouldn't using the old diffuse shading system on Terrain not stop specularity if you don't have it to 0? Didn't realize that 0 specularity still means you get specularity on old diffuse.

 

@LIVENDA: I'd like to point out that you'll make better argument if you post reference formula you think Unity should be using AND offending line from within standard shader. There is no real standard on PBR lighting, as far as I'm aware, so it isn't like back in the blinn/phong days where you could just remember the formula and live happily ever after, so there's no real reason to think that UE4 does it the right way.

Unity standard shader did have quite a bit of weirdness, as far as I know, maybe they broke more stuff in 5.3.

Also, UNreal 4 has different shader. In UE4 you have metallic AND specular maps available at once. In unity you have metallic OR specular, not both of them at once. At least that's the way it was in 5.1.3.

-----

Also, please consider filing a bug report. General forum is marked as "this is not support forum", so that's probably not the place to post stuff you want fixed.

 

I meant the general legacy diffuse shader not the Terrain Diffuse system. Physics of light is very complex, even the PBS formulas are rough approximations. Unity added a fudge factor of I think 0.04 to 'add' extra 'specular' light coming from global reflections. This is the issue. It adds a slight gloss/haze over everything. You need to be able to use the standard shader as it is important for other surface, however also have the option to atleast eliminate redundant reflections at 0 smoothing.

 

Don't really have the time to go look at the code at the moment, perhaps they are using N.V for the fresnel approximation, this is ok for most materials but not for rough concrete with low specularity when using pre-filtered ambient specular (The 'rim' light will be bad )

 

Out of curiosity, I've ran a diff between 5.1.3 and 5.3.1, and it goes like this:

The only reference to 0.04 I found is related to sun size in procedural skybox.

In the few places where they add small values you would get division by zero otherwise. Division by zero in a pixel shader is a bad thing.

The majority of lighting-related calculations are within UnityStandardBRDF.cginc, interestingly there are links to documents they used.

https://dl.dropboxusercontent.com/u/55891920/papers/mm_brdf.pdf

I suggest to tear apart UnityStandardBRDF and see if there are bugs in it.

Curiously, in case of specular, at least in one place results were read from precalculated texture instead of being computed.

Funny enough, the texture was named "unity_NHxRoughness" (that happens around line 432 in standard bsdf cginc), and that could theoretically be related to your issue, for obvious reason. The texture existed in 5.1.3, so if you didn't have the issue back then, it is something else.

 

unity_NHxRoughness is only in BRDF3_UNITY_PBS, which is only used when this is defined:

And I think isn't relevant because of this. (BRDF1 is highest quality down to BRDF3 for lowest quality)

Code (CSharp):

1.                     + specularTerm * light.color * FresnelTerm (specColor, lh)
2.                     + gi.specular * FresnelLerp (specColor, grazingTerm, nv);

But I'm still wondering, how much of this 'problem' is simply fresnel as it should be? I'd be interested in a comparison when looking (nearly) straight down on the plane, with the light coming from behind to eliminate fresnel being a part of this.

 

There's been a couple of papers on lighting where people have used polarized lighting rigs to sample albedo and specular separately. One I saw years ago (and can no longer find, sadly) tried a couple of obvious things with specular, then some obvious things that they thought wouldn't have specular, like towels or paper, and found they had a ton of specular. They then tested talcum powder, the material that's pretty much the standard for 100% diffuse, and found it too had a surprising amount of specular. Essentially in the real world there's no such thing as 100% diffuse with no specular, mathematical models for lighting now make assumptions for 100% reflection for all materials at "perfect" grazing angles not counting light absorption. If you think about it from a physics perspective it makes sense, the only difference between diffuse and specular is the number of times the light bounced off a surface before getting to your eye.

So, while unintuitive it's not necessarily wrong. Unity's Standard Metallic also uses unity_ColorSpaceDielectricSpec.rgb for the specular color when metallic is set to 0, which is likely where that ~0.04 is coming from as that's right in the middle of their recommended "40-75" RGB values for dielectric specular. (0.04 would be 56 if you do the sRGB conversion. 0.0409 or 0.0423 if you do 57 or 58 which are the actually average of 40 and 75.)

Now, that all said sometimes you really don't want that specular. For a game I'm working which is NPR but uses the standard shader as a base I added a little snippet of code so that zero gloss removes specular.

// clamp out spec if roughtness is 1
specularTerm *= saturate(oneMinusRoughness * 10.0);

A little dirty, but effective.

 

Where did you put that line of code, in UnityStandardBRDF ? This only works when in forward render mode, in Deferred it has no effect.

Also,

Can anyone tell me why changing these

#define UNITY_GLOSS_MATCHES_MARMOSET_TOOLBAG2 1
#endif
#ifndef UNITY_BRDF_GGX
#define UNITY_BRDF_GGX 1

Only work for forward render mode, it has no effect in Deferred. Deferred is always using GGX.

 

There's an Internal-DeferredShading.shader that handles lighting for deferred you have to modify and either have a copy of in your resources or override it in the graphics settings.
http://docs.unity3d.com/Manual/class-GraphicsSettings.html

 

Internal-DeferredShading.shader only sets up the light data though - the actual PBS function used is still UNITY_BRDF_PBS as it gets defined in UnityPBSLighting (to a function in UnityStandardBRDF).

 

Just a little update,

Managed to reduce the Ambient Specular reflections to a much more sensible value when smoothness is at zero while retaining compatibility (i.e. looks the same) at higher smoothness values.
NOTE: This is for Deferred path not forward mode. This effects the Standard (Metallic workflow) and Specular workflow. Also, this is a quick Hack! and the AMBIENT is deliberately set to zero

Now, if we can manage to reduce the Specular reflections from lights to match, that would be great. Still cannot see a way to do this... This is important as the PBS is essentially based on Dielectric materials however does not translate to other materials, it is also important to keep in mind the directional light is an infinite distance light source with parallel rays, also in 'real-life' with perfect GI you get light coming in from everywhere, that is why the current implementation is not really suitable.

@bgolus your sugestion of

// clamp out spec if roughtness is 1
specularTerm *= saturate(oneMinusRoughness * 10.0);

Only works for Forward and works fine however we need a solution for the Deferred Path, any help would be greatly appreciated.

Here is the before and after the Hack

Keep in mind, this is the 'lowest' roughness setting, so impossible to go lower with current implementation



Change line 648 from UnityStandardCore.cginc to

outSpecSmoothness = half4(s.specColor, s.oneMinusRoughness)-0.0423+(s.oneMinusRoughness*0.0423);

Cheers,
Michael

 

So if this is your fix, was the problem that dielectric materials had a specularity of ~0.04? Seeing as you lowered the specularity by 0.04. If you want fine control over specularity instead of a dielectric/metal switch there is the specular standard shader. I can imagine wanting to lower roughness to actual 0 if the GUI 0 isn't a real 0 though.

 

Oh and check the newest patch:

May be related.
http://unity3d.com/unity/qa/patch-releases

 

I've been struggling with PBR Specular since I upgraded my project to Unity 5. I'm using the Standard pipeline for my game, but there's no way I can remove Specular for my objects.

What should be the approach for something like this? (look at the bush).



Standard Specular shader with totally black (0 alpha) Specular color and 0 Smoothness. Looks the same with Standard instead of Standard Specular.

UPDATE: I 'think' this effect is most noticeable when using meshes with no normals. The only way I could fix it is by going to the Internal-DeferredShading and adding 'if (oneMinusRoughness == 0) oneMinusRoughness = 1;'.

 

True, but since Unity is mainly used for games, reality is not always the objective. I agree with Livenda here.

I've said it before in other topics. I don't mind the standard shader being PBR and always adding some reflection. In my opinion the real issue is that reflection is being added even if you write pure black as reflection color in the g-buffer in a custom shader.

If the reflection color in the g-buffer would be used as is and the standard shader just outputs the realistic reflection amount to the g-buffer everything would be fine. The standard shader would still be PBR and in a custom shader you can remove reflections completely if you want.

 

Yep, I think there's plenty of people that don't want to work on hyper realistic games but still want to use the Standard shader pipeline. Removing options/control for tuning how the game looks is always a bad choice.

 

@xDavidLeon - I haven't done the deep dive through 5.3 to find the change that made this so much more obvious, but in 5.0 I changed the internal deferred light shader to lerp to completely black specular and reflections if the smoothness was set to zero, regardless of the specular color. It's also a NPR game, so at full roughness we didn't want any fresnel.

I would say the point of the Standard shader is to be as realistic as possible, as close to ground truth as it can be. Too many options in PBR just lead to non-PBR results. It's easy enough to change the shaders to accomplish what you want.

 

I don't think diving into the built-in shaders and modifying the Internal-PrePass shader is 'easy enough'.

If you use Unity there's these options:

• Use the Deferred pipeline, that means working with the Standard shader most of the time. Creating your own surface shaders through the Standard pipeline is not documented at all. There's still no examples of custom lighting functions using the Standard pipeline. Or an example of surface shader with the Standard pipeline.
• Use the Light-Prepass pipeline. Not really fond of using a legacy feature. What's the point of using Unity 5 if you use Light-Prepass? And of course, it's less optimal than Deferred, requiring more passes per object.
• Use the Forward pipeline, and get awful performance with multiple point lights + realtime shadows.

In my specific case, I'm developing a game that uses PBR features but with the NdotL equation polarized to get a cel shading effect. Just to get that small change to work with Deferred it means having to modify internal shaders, find a way to create custom surface shaders with standard pipeline, and find a way to modify the Standard Lightmapping functions so that the resulting lightmap is cel-shaded (btw, that meant having to modify UnityStandardBRDF.cginc too, 'easy enough' i guess?). Oh, and duplicating the standard shader just to turn off culling for some objects.. For such a small change I sure had to go the extra mile. The 'you can't fully remove specular because that's not realistic' is just ignoring the issue.

This is not a rant, I just wanted to point out that the lack of flexibility of the new Standard pipeline + the lack of documentation for doing anything 'out of the default/expected uses' is what annoys me. If Unity wants all their games to look the same that's great, but I think the strong point of Unity vs Unreal (or any other modern engine) should be flexibility and not raw realism at all costs.

 

Fair, though as a point of note I modified the Internal-DeferredShading.shader which is for the new deferred pipeline, not the Light-Prepass aka "Deferred Lighting" Internal-PrePassLighting.shader. My fault for mixing terms there, I was not intending to refer to the legacy deferred systems at all.

Having documentation is good, but the Standard shader is a very complex system. I'm not sure there is a way to make it easier to edit. Something that seems like a simple change like turning the Standard shader into a cel shading shader can be a fairly significant task, as you found. One might argue the correct path would of been writing everything from scratch rather than trying to modify the Standard shader, though this is the path I took as well.

A lot of the pain comes from the interconnectedness of the cginc files and the inability to easily override them on a whole-project basis without resorting to manually replacing the files in the Editor's own CGIncludes folder. If you want to modify something way down the chain it means making a copy of everything.

 

Hey guys, sorry to necropost, but I was looking for answers and solutions to the haze I get on rough surfaces in my scene. (I'm using Alloy for my PBR) For some reason, when I crank up the ambient light from the sky, my surfaces get this weird glazing (making a rock surface with a roughness of .95) and it's damn near impossible to get rid of.

One solution I may have is setting my specularity value to .08 in Substance Painter to lower the glazing on my surfaces. There shouldn't be any glazing whatsoever on a rough surfaces. This doesn't occur in reality and even if I use a roughness of .98 I still get glazing.

Another solution is on the skybox is lowering the reflection strength. Weird, that this is an option in Enlighten. Getting reflections is the job of reflection probes... the sky shouldn't even have an option like that in the environment tab. With reflection probes, capture WHAT IS THERE and be done with it.

I'm gonna test setting this to 0 and see what happens... the skybox should just be a flat image. I'll share my findings tonight.

EDIT: So the glazing is VERY apparent if you use VERY high GI settings like an intensity of 1 and a bounce boost of 5. That is unacceptable, even at high roughness values lIke a pair of jeans or concrete. I use Unity and Unreal 4 and the glazing NEVER happens in UE4. Something has to be done because it looka weird and ugly to my eyes. I'm gonna lower the specularity on my textures to .08. As much as I like physically based rendering, this has been a pain.

 

Hi everyone. Just spent 2 days driving myself insane with a pure red surface with smoothness set to zero, and then taking a sample of the rendered image always has the blue and green components slightly raised, giving the image a pinkish appearance instead of pure red. I got down to this level of detail testing a custom terrain shader, and I finally realised that what I was seeing on the surface was a very dull specular reflection of the white light source (sun). Ambient is completely disabled and reflection intensity set to zero. Thus, in my ideal test scenario, pure red became RGB 255, 20, 20, approximately. It means that it's impossible to make a surface pure matte. It's really quite frustrating, but at least I know now what the heck has been going on. I will read above to find solutions.

 

Unfortunately there's no outSpecSmoothness in UnityStandardCore for 5.6, does anyone know how to properly hack UnityStandardCore for 5.6?

I also want to add, that I also consider this a serious issue and this approach definitely results in some really ugly visuals.

Like this.

 

I've encountered the same issue, I've adopted standard(Specular setup) instead. The specColor of metallic setup of UNITY is calculated by: specColor = lerp (unity_ColorSpaceDielectricSpec.rgb, albedo, metallic); where I believe the unity_ColorSpaceDielectricSpec is not zero for metallic=0.

 

It's a grey value about ~57/255 sRGB.

 

It's 2020 and I still don't know how to solve that issue. As soon as directional light is pretty low, stuff (especially weapons) start too lake bad and fake. Tinkering with materials doesn't help.