Creating runtime normal maps using renderToTexture

Hi all,

I am able to runtime generate textures for my character using the (fantastic) renderToTexture functionality for the diffuse map, but would like to expand the 'depth' of the result on my character by runtime-generating the matching normal map as well...

Has this been done before? Currently I'm thinking of moving small 2D normal map textures that have cloth-related normal map features like cloth trims, borders, zippers and buttons... with them being of the same size and placed at the same 2D texture position as their diffuse counterparts (rendered of course in their own renderToTexture camera rendering the diffuse map)

Is this the 'recommended approach'? Are there better ways?

Any clues / hints would be greatly appreciated!!

Daniel,

 

Hi all,

Researching this further, I'm able to generate nice diffuse maps for my clothing at runtime, but even if I can generate what looks like normal maps into a 'render texture' camera, my clothing doesn't apply the bump effect to dynamically-generated render textures.

(The same setup with a statically-generated normal map will work fine... assuming that texture is marked as normal map during import)

In other words... is there a way to have a render texture camera create a usable normal map?

If not, could I use the getPixel / setPixel approach to generate a static normal map, copy the result of the render texture camera and flag that second texture as a proper normal map?

Help!!

Daniel,

 

Help! Am grasping at straws!

Would any of these alternatives work?
- Grabbing output of render texture camera, stuffing that into a properly configured normal map already connected to my mesh.
- Using some of these advanced shaders in asset store...
- Creating cloth borders as 3D cylinders, calculate 3D normal vector at each pixel, converting normal to RGB value for normal map.

Any ideas? Has anyone been succesful in using render texture to generate usable normal maps??

Thanks for any hint in proper direction!

Daniel,

 

This page seems to contain some detailed info that might be useful to you:
http://unity3d.com/support/documentation/Components/SL-CameraDepthTexture.html

 

Hi Wolfram,

Thanks for taking the time... I've tried setting that depthTextureMode to my render camera to all settings but unfortunately no change... the "normal map looking" bitmap that camera generates is still not interpreted as a normal map... similar to the effect you'd get if a normal map in Unity were not configured as a normal map... you get no 'depth' until you set them up as normal maps and I think that's the step missing here...

Am continuing my research!

Daniel

 

OK, the problem is that since some time (>=3.2 I think), normal maps are stored in a specially packed format, using two 16bit values instead of 3 8bit values (the missing 3rd value can be derived from the other two).
So what you need to do to use a simple, unpacked RGB-Normalmap-Texture as a normalmap is, use your own shader that skips decoding the 16bit-format.

In short:
- get the builtin shaders from the website http://unity3d.com/download_unity/builtin_shaders.zip
- choose the bump shader you want to use, for example "Bumped Diffuse"
- find the appropriate file containing that shader (in this case it's DefaultResources/Normal-Bumped.shader)
- make a copy of the file and import/drag it into your project
- edit the shader
- rename the shader in the first line, for example by adding "(unpacked)"
- replace this line near the end of the shader:

Code (csharp):

1.  
2. o o.Normal = UnpackNormal(tex2D(_BumpMap, IN.uv_BumpMap));

with:

Code (csharp):

1.  
2. o o.Normal = tex2D(_BumpMap, IN.uv_BumpMap)*2-1;

- select this new shader in your material

That should be it. I tested this with non-RenderTextures (by choosing "Texture" instead of "NormalMap" as the type of a standard normal map (note in this case Unity will keep complaining that this texture needs to be converted - ignore that)), and it works.

(all this assuming you're really talking about the "bluish" normal maps, not the greyscale bump maps which unity converts to normal maps)

 

Hi Wolfram,

Thanks a million for taking the time with this complex render texture stuff... (I am indeed trying to get a working bump map using a render texture created by a render camera at runtime (the bluish ones with RGB(128,128,255))

I'll research and try to implement this and will report on progress shortly.

Thanks again!

D

 

Hi Wolfram,

IT WORKS!

You rock man! Thanks again for taking the time... I think I would have given up on that one...

For those reading how to get normal maps working correctly when rendered through a 'render texture' camera at runtime (pro feature only), follow Wolfram's exact steps above and you're good to go!

Steps to get this working correctly (once you've finished Wolfram's procedure above)
1. Create a 'render texture' in your assets, and set the proper size for your needs.
2. Place a new camera you intend to use as a texture source and keep it away from all your scene's lights.
3. Set that camera's 'Target Texture' to the render texture you created in step 1. (pro feature only!)
4. Set that camera to be orthographic and of the size for your texture.
5. For each texture/shape you insert into the render camera zone...
a. Set all the shader to 'self-illum/diffuse' shader so they self illuminate... or 'unlit' as need be.
b. Set the 'Main Color' of that shader to RGB(128,128,255) (if using self illuminate)
c. Insert your bluish normal map segments to compose a starting state for your normal map.
6. On a game element, set the shader to the one you created from Wolfram's steps and point the normal map to the render texture you created in step 1.
7. Have fun moving the normal map elements in your normal map camera during run time to see the bump change at runtime while your game runs!

Totally cool stuff!!

Daniel,

 

Hi I'm having a problem with this script in Unity 4.6.

The interface just says 'Multi-object editing not supported'.
There are no controls shown for the script, nothing seems to work.

I can send a screenshot if needed-- not sure how to attach one here...

 

Minimo: What script are you referring to? There aren't any scripts mentioned in this thread.

To answer your question anyway, 'Multi-object editing not supported' means you have selected more than one object with a script attached where...well...multi-object editing is not supported for that script. Instead, select only one object, and edit the script settings in the Inspector for each object separately.

 

Hi Wolfram,

I´m trying to do the same thing that Daniel wants to, but i´m having problems following the instructions to make this works. I have a RenderTexture ,that cant be marked as Normal Map in unity, created in runtime from a particle system that drop particles like the image attached. The resulted textured is the other attached image (gotas.png).

My problem is that i cant mark a RenderTexture as Normal map in unity in runtime and I have to find a way to use that texture as Normal Map cause in doing some refraction stuff in my shaders. In that shaders if i use a Texture marked as Normal Map it works great but if i use a Normal Map texture that is not marked as Normal Map , the refraction works but it displace the image an offset proportional to the displacement value.

My question is, what is the code that i have to use to have a normal map that is not marked as normal map in unity and works in tangent space. P.D: i've tryed the "o.Normal= tex2D(_BumpMap, IN.uv_BumpMap)*2-1;" and its doesnt work.

 

For anyone who's trying to incorporate the above normal code, but getting strange results, make sure to clamp the results like this:

Code (csharp):

1. o.Normal = saturate((tex2D(_BumpMap, IN.uv_BumpMap).rgb*2.0)-1.0);

 

@ph0b0s:
Sorry for the late reply. Well, essentially you need to replace any calls to UnpackNormal() with the code I posted, in all shaders where you want to use your custom map. If that does not work as intended, you'd have to elaborate on "its doesnt work" - what EXACTLY doesn't work, do you get compile errors, does it look wrong (if so, post an image), etc.
Note that the behaviour in DX11 might be different, so you can also try to disable DX11 in the Player settings, which will then use DX9 instead.
Also note that your "Normal-ParticleNoTrans.jpg" map is incorrect. Unity uses a left-handed coordinate system, but the tangent space still has X oriented to the right, and Y pointing upwards. So to create a "bump", imagine a bluish object that is lit by a green lightsource from ABOVE, and by a red lightsource from the right. That is how you can visually check your normal maps for correctness.
If you use the wrong map format, you create physically impossible appearances, visually inverting the object's "surface", depending on its orientation.
In your "Normal-ParticleNoTrans.jpg", the green channel is inverted, making it appear as if it is "lit" from below. There are programs that require this format (e.g., 3Dmax I think), but Unity is not one of them. Either invert the green channel using Photoshop, or make sure your normal map creation tool is set up correctly (for example, in CrazyBump set "Normal map y-axis" to "up" in the preferences).

@chingwa:
Umh, no, this is not correct. All that saturate() does is clamping all components to 0..1. But this does not make sense for normals, which can (and must) have negative x- and y-components. If you use the code you posted, you will essentially disable all bump effects that "bend" to the left or to the bottom.
Your problem must lie elsewehere.
What do you mean by getting "strange results", can you post some images? It might very well be that you have the same problem as ph0b0s, i.e. an inverted green channel in your normal maps.

 

On a normal map, yes... but the above texture would be marked in the Unity editor as "texture", not as "normal"... so the shader sees it as a typical 0.0-1.0 rgb value.

You've made me doubt my results though I'm gonna go back and check over my project.

 

The texture has 0..1 rgb range, and that's what the tex2D() call returns. However, *2-1 will then "unpack" this value into a normalized Vector range, which is -1..1. This is what the "old style" UnpackNormal() did (<=Unity 3.x I believe), and where the bluish color comes from (RGB (128,128,255) = RGB (0.5,0.5,1). 0.5*2-1=0, and 1*2-1=1, so the "unpacked" Vector is (0,0,1), which is just an undistorted "neutral" normal in tangent space).

 

OK, I see what you're saying. In going back over my code I see that you are right! In my first pass I was too focused on the error I was getting with the original normal code, and when I saturated it the errors disappeared, however on closer inspection the normals also disappeared.

Below is a breakdown of what I'm tryig to do. I need the normals editable on the fly depending on what types of clothes the character is wearing (a common use-case). The normals are calculated from multiple textures that are combined with Graphics.Blit into a single normal texture. The original normal code gives me errors however... I then tried to normalize it which got rid of the errors but the lighting appears off. I then tried to manually raise the normal value from *2..0-1.0... to * 3.0-1.0... which helps fic the lighting, but... it still look off. Any advice here, Wolfram?

 

Hm, you're getting clipping artifacts somewhere. The formula itself (without the normalize) is correct, if you assign its return value directly to the o.Normal of a surface shader (as opposed to casting it to a fixed3 or something), you should be fine.

One thing that could explain what you see is that your "neutral" value is not 128,128,255. Assuming your first image are exactly the colors from your normal map, undistorted (i.e., no lighting/attenuation or similar), and assuming the .jpg compression did not mess with the color values too much, your neutral value seems to be 116,114,255. This translates to a normal of -0.0902,-0.1059,1.0, which a) is a normal pointing slightly (=8 degrees) towards the bottom left (partially explaining the body being in the shade), and b) has a length of 1.0096 instead of 1.0 (explaining the black clipping near the highlights - the bright clipping could be due to color wash-out/clipping (i.e., the sum of diffuse and ambient color is >1, and this messes up the shader when dealing with non-normalized normals)).
It is curious, though, that your formula of *3-1 almost exactly represents the true lighting conditions.

Anyway, make sure the neutral value of your generated normal map is *exactly* 128,128,255, and any deviating normals use that neutral value as a reference.

 

OK, it's looking much better now! All the base normal textures were at the correct neutral value, but once they got combined the values were shifted... thus causing the error. I ended up having writing a second Blit shader specifically for the normals without the lighting information that was in the original blit shader for the albedo and other maps. Now that the final normals are all in the proper range it works like a charm!

 

Sorry to dredge up this old thread, but this information is useful. If you want to just use Unity's unmodified Standard shader, you can instead render or convert your texture into the DXT5Nm format. Put the red channel into the target alpha channel and that should be it. If you were going to actually compress to DXT5 you should also leave the green channel as-is and set the target red and blue channel to some constant value. More info here: http://tech-artists.org/wiki/Normal_map_compression#DXT5nm

 

Hey there, anyone knows if using:

Code (CSharp):

1. o.Normal = normalize(tex2D(_BumpMap, IN.uv_BumpMap).rgb*2-1)

costs more performance than:

Code (CSharp):

1. o.Normal = UnpackNormal(tex2D(_BumpMap, IN.uv_BumpMap))

????
i'm targeting mobile, and i will use it for like 1 to 10 objects in the same place

 

The Unpacking costs more performance as it not only performes the *2-1 function that you do above, but also does an extra square root function in order to un-encode the compressed texture. I don't know how much this would actually matter in a project, but sqrt() in a shader could theoretically add up to a performance problem.

You'll have to decide for yourself which is better for your project, but using...

Code (CSharp):

1. o.Normal = tex2D(_BumpMap, IN.uv_BumpMap).rgb*2-1;

should be slightly faster.(no need to normalize assuming your texture values are within the correct range). If you REALLY need to eek out extra performance you can probably drop the blue channel from the calculation, and just perform it on the R and G channels...

Code (CSharp):

1. o.Normal.rg = tex2D(_BumpMap, IN.uv_BumpMap).rg*2-1;
2. o.Normal.b = 1.0;

Of course this only works with a non-encoded normal texture (setting the 'texture type' to 'texture' instead of 'normal'). There is an extra advantage to the above though, in that you can use the blue channel and the alpha channel of the normal texture for other effects if needed, such as a heightmap, or a smoothness map, so not only are you saving a (small) calculation in the shader, you are also potentially saving texture memory as well

 

Ok that's what i was searching for, thank you!

the heightmap and the normal map will be made during the game as standard textures, so i can flip their channels with a script, so could you please tell me which are the heightmap channels i put in .a and .b?

 

You would have to choose, and then write the shader code to use that channel yourself. The default unity behavior expects height to be a separate texture... so if you want to pack them like suggested above you have no choice but to do some shader edits.

 

Does anyone knows if something changed in this matter? I'm loading normal maps in runtime, changed textures as described http://answers.unity3d.com/questions/801670/runtime-loading-normal-texture.html - completely wrong results, changed standard shader also - the same, completely wrong effect. Dx9/Dx11 no difference, the only one properly rendering case is when texture is not runtime loaded and set as normal map.
I tried everything, no more ideas, need help you guys .
Runtime loaded and converted:
http://funkyimg.com/i/2hHnD.png
Proper look, set as normal map:
http://funkyimg.com/i/2hHnG.png
(Material on cube to exclude possible mesh irregularities)

 

I don't think anything has particularly changed. You may want to make sure that 'encode as rgbm' is set to off just to make sure no color adjustments are being made on the texture before you run it through your shader. As long as your shader is using the * 2 - 1 formula the normal should be decoded correctly.

Typical normal decode (texture marked as normal):

Code (CSharp):

1. UnpackNormal(tex2D(_NormalTex, IN.uv_NormalTex));

Manual normal decode (texture marked as texture):

Code (CSharp):

1. tex2D(_NormalTex, IN.uv_NormalTex) * 2.0 - 1.0;

In some cases (depending on project and useage) I've had to make linear adjustments to the texture like so...

Code (CSharp):

1. pow(tex2D(_NormalTex,IN.uv_NormalTex), 0.454545) * 2.0 - 1.0;

 

Yes, I also use standard shader, from 5.3.6 and there is no such lines in standard shader. What do you mean 'encode as rgbm'? Texture format is set to RGBA32.
edit: ok, I guess you mean this setting - https://docs.unity3d.com/ScriptReference/TextureImporterSettings-rgbm.html - I think it's irrevelant as I need runtime loading.

 

I see, I missed your note about runtime generation, ignore what I said about rgbm. Still, you will need to use a custom shader that does manual normal decoding, as the standard shader and all other built-in unity shaders assume the normal texture will be set as 'Normal' in the texture importer which custom encodes the normal data.

 

omg I just want to say thank you! Wolfram's answer solved it for me too

 

Unlike what is said several times in this thread, there's no need for a custom shader because of the normal map decoding ; the correct channel use is exposed in this topic : https://forum.unity.com/threads/get-external-normal-map-at-runtime.532892/ .

In short : fill the normal map texture with (1, y, 0, x) or (x, y, 0, 1), and be sure it's marked as linear. (so swap your output channels to match one of these 2 formats)