Vertex Color is clamped.

Hi,

I am learning some basic cg/ cgfx.

I am trying to create an accurate cg shader that displays vertex color as it would look when rendered (with the correct lighting intensities.

In Maya I have written a cgfx shader. It does VertexColor * texture * 2.

Here is a snippet of code from the cgfx shader.

Code (csharp):

1. // input from application
2. struct a2v {
3.     float4 position  : POSITION;
4.     float4 normal    : NORMAL;
5.     float2 texCoord : TEXCOORD0;
6.     float4 VertColor: COLOR;
7. };
8.  
9. // output to fragment program
10. struct v2f {
11.         float4 position        : POSITION;
12.         float2 texCoord : TEXCOORD0;
13.         float3 worldNormal     : TEXCOORD1;
14.         float4 VertColor: TEXCOORD2;
15. };

If I change in the out to frag

Code (csharp):

1. float4 VertColor: TEXCOORD2;

to

Code (csharp):

1. float4 VertColor: COLOR;

I no longer get the correct overexposure. I asked this on the Nvidia forum and a reply was that COLOR might clamp the values where as TEXCOORD just treats as data.

I have attached an image to demonstrate the result. With TEXCOORD the result is exactly the same lighting as whatever shows up in a render which is awesome!




Now my problem is, how can I do this in Unity?
I am not sure if in Unity TEXCOORD is a float 2 or something instead of a float4... becuase this is the result....




I am testing with Unitys shader from here:

http://unity3d.com/support/documentation/Components/SL-VertexProgramInputs.html

Code (csharp):

1. Shader "!Debug/Vertex color" {
2. SubShader {
3.     Pass {
4.         Fog { Mode Off }
5. CGPROGRAM
6. #pragma vertex vert
7.  
8. // vertex input: position, color
9. struct appdata {
10.     float4 vertex : POSITION;
11.     float4 color : COLOR;
12. };
13.  
14. struct v2f {
15.     float4 pos : SV_POSITION;
16.     fixed4 color : COLOR;
17. };
18. v2f vert (appdata v) {
19.     v2f o;
20.     o.pos = mul( UNITY_MATRIX_MVP, v.vertex );
21.     o.color = v.color;
22.     return o;
23. }
24. ENDCG
25.     }
26. }
27. }

Thanks!

 

Thanks for your reply Aras.

I will try to explain my problem a little better, I just started learning to write shaders yesterday so I hope I am not too confusing.

When Maya bakes vertex colors, it WILL store values greater than 1. It does not by default clamp them 0 to 1 but instead actually stores what I guess is considered the true color and intensity of the lighting in each point.


I want to make a shader that DOES NOT clamp the values but instead uses this baked vertex lighting accurately so it matches the lighting quality in a render. I can do this in Maya by changing COLOR to TEXCOORD in the out frag section of the shader as I posted above in my CGFX shader.


Now I want to achieve the same unclamped and accurate result in Unity but when I change COLOR to TEXCOORD just like I did in my CgFx shader, Unity turns the VertexColors Orange and Green like in the image.


So I am wondering how I can get unclamped and accurate baked vertex lighting with a shader in Unity?

An example would be great. Are you saying I just need to change COLOR to TANGENT? I am sorry I dont really understand your reply, I am very basic right now with shaders.

 

ah, that's a shame.

Your right, I checked in Maya and if I clamp the results 0-1 I loose the overexposure as well so it's not just Unitys shader but the mesh data is a problem too.

Edit:

Is there a reason Unity clamps on import in the mesh data? It would be great to have the option not to clamp.

 

Hmm, I think I found a workaround that works for me. I now get accurate lighting just like in a render with clamped values.

I scale my baked vertex colors by .25 so everythings is under 1.

I change my formula from

outColor.rgb = colorMap * Vcolor * 2 ;


to

outColor.rgb = colorMap * (Vcolor * 4) * 2;

eg
outColor.rgb = colorMap * Vcolor * 8;

 

@Aras is this still the case with the latest Unity?
I saw there is Vertex Color compression in the player settings, so does it mean we can have higher precision colors?
Was planning to use Vertex Colors before to paint some models and lower app size, but while Maya produced smooth color transition I was getting some ugly banding in Unity .
I wonder if it has changed now.

 

Hasn’t changed. Vertex colors are still stored exactly the same as they always have been. Vertex channel compression reduces the precision of data to save on memory usage and bandwidth, however it doesn’t actually do anything to vertex colors as they’re always stored as 4 bytes regardless of the mesh compression settings, either for the project or on the individual mesh.

However I’m a bit curious about you seeing banding since once on the GPU, the color values are converted to floating point values and interpolated at a much higher precision. The only issue is vertex colors are stored in sRGB space, so they’ll look wrong if your project is set to use linear space color unless you manually convert them in the vertex shader.

 

To be honest I never tried the shader on the device to see if the result was different as I was expecting the same.
I saw the problem and found this answer before(maybe the thread wasn't this) so I discarded the idea.
I'll need to check again. I was using the vertex colors to bake some lighting like Ambient Occlusion and the result in Unity was quite poor compared to Maya, with some abrupt changes.
I also had a custom shader to remap some detail texture using vertex color channels,similar to terrain splat maps but the low precision was giving poor results.
I need to try again and see if on device it's better.
If I understand you correctly they are converted on GPU from Unity low res value, so all the precision is lost anyway.
Color Interpolation between vertexes is ok even in Unity but if for example I had 10 vertices near each other with very subtle variation, in Unity due to rounding to lower precision format they would become all the same resulting essentially in a huge clump of polygons of the same color, while in Maya they were more harmonious and less unnatural.
It's something very subtle but make a world of difference.
See the image below for example.
Let's say the result I was getting in Unity was like the middle one of the 9 images (second row,second column) while the results in Maya were like the 3rd row first column.


My problem is that vertex color are treated as fixed values and not float.
Maya use them as float and so do most packages so there is no way to edit them correctly in advance.

Wrong color in device is ok, I can adjust them by trial and error and anyway it's much less important visually than having a flat color instead of a variation.

 

fixed vs float isn't really the issue at hand. On the PC at least, fixed values are the same as float values, and most mobile platforms fixed is the same as half. In fact "real" fixed precision is still higher precision than what Unity is storing vertex colors as. They're stored as 4 bytes, or 8 bits per channel. Aka, as 32 bit color. This is the same RGB color precision as the jpg image you posted above, or really just about anything you're working with in terms of image formats for textures. Maya likely does actually store the per vertex colors as floats which would end up being smoother. But if your mesh is so tessellated as for this to be a factor in how smooth your colors appear, then I have to wonder if your mesh is too highly tessellated. You could look to store your vertex colors in the UV channels instead. You'd have to store the RG in one UV set and the BA in another since Unity will only ever use the first two channels of a UV, but then you could have full float precision for your colors. You could even then use mesh compression on those extra UV channels to lower them to 16 bit or even 10 bits (using medium compression for the model importer's mesh compression setting) and have better precision than "real" vertex colors.

 

I don't believe this is true anymore? (Tested in 2018.4) Even though the .uv properties only return a Vector2 array, you can set/get Vector4 arrays with the Set/GetUV methods on the mesh, and the Z/W components seem to be present when accessing them in shader.

 

You can get and set Vector4 UV values, you can’t import a mesh with 3 or 4 component UVs.

 

Ah okay, wasn't sure which part of the pipeline you were referring to there. It is odd they don't import it if detected in the file.

Another option would be using the user data feature on assets in Maya to store your color info and whatever else, which can then be accessed in an importer script, which you can then apply the data to whatever UV channels you want.

 

Not a Unity issue really, but rather their reliance on the official FBX plugin, which only supports two channel UVs. FBX can support more with custom data, and it seems to use this when transferring between official Autodesk applications, but the official plugin and even the reverse engineered FBX spec only shows support for two channels.