Tips for displacement maps

I spent a while getting displacement to work. There's not much info out there for this that I could find, the docs are nearly non-existant (https://docs.unity3d.com/Manual/SL-SurfaceShaderTessellation.html is about all there is, and that's just a few code snippets and no real detail), and there are a lot of gotchas. Thought I'd drop a few notes in case it helps the next person searching for it.

I tried a bit to get a vector displacement map to work, but they're a lot more expensive and I don't need them right now, so I stuck with regular displacement. (Maybe I'll put together a sample scene, but I've spent too much time on this already--maybe later...)

Quick points ("TLDR", read below if you want the "why"):

- Use xNormal. I got usable results out of it where I couldn't out of Mudbox or Zbrush.
- Set your scene to linear (Project Settings -> Player -> Other Settings -> Color Space).
- Use floating-point displacement maps. In xNormal, set "Normalization" (in "Height map" options) to "Raw FP values" and save as .EXR to get this.
- Leave displacement at 1. You don't need a magic displacement factor when you use float maps (but see below).
- If your model has a file scale on it (or you've set a scale factor), you have to adjust for this in the shader. My models are at 0.01 scale since they were exported from Maya (centimeter scale), so I have to multiply displacement by 0.01 in the shader to match. If your model explodes when you turn displacement on, check this.
- Set your displacement textures to type "Default", turn off "sRGB", and turn off compression.

Why linear?

Unity apparently won't let you sample floating-point textures without color conversions unless your scene is set to linear color space. (Project Settings -> Player -> Other Settings -> Color Space) In gamma mode, float textures are converted to sRGB when you sample them--even if "sRGB (Color Texture)" is unchecked. Unity devs claim this is "by design". Why would you design it so non-color textures have color conversions applied? If it's by design, then it's a design bug.

xNormal notes:

- If I just exported a mesh and a high-res mesh, xNormal couldn't figure it out. It would map one half of the character to the opposite side and give garbage. I worked around this by lopping off the left half of the character (in both the low- and high-res mesh) before giving it to xNormal, and this problem went away. I also had some random garbage that went away by disabling "Closest hit if ray fails".
- Make sure you have non-overlapping UVs, at least within the side of the mesh you give to xNormal. Overlapping UVs don't work for generating maps.
- xNormal will output an RGB EXR, which is a lot bigger than it needs to be. This can just be converted to a single-channel greyscale file, but I haven't chased down a tool that can do that yet.

Troubleshooting:

- Look at your displacement map after you generate it. If you're generating a map from a low-res mesh to a high-res mesh, think about how much distance separates the two meshes. If it's a human-scale character, it's probably less than 1cm. If your model is in cm scale, you should be seeing numbers less than 1 (less than .1 in my case). If I see a bright white patch with numbers like 5, I know something is wrong with the map, since there's no place where the high-res mesh is 5cm from the low-res one.
- Remember that your models may not be in the same scale as your Unity scene. I'm exporting from Maya and my characters are in cm scale, and Unity's "Use File Scale" option scales these to meters on import.
- You might have precision issues if your models are in meter scale. At cm scale I have displacements like 0.05. If I was in meter scale, that would be 0.00005. That's too small a number to store in a 16-bit float. I don't have this problem since I'm at cm scale, so I don't know if this is actually an issue or know how to fix it if it is.

Other stuff:

- Displacement maps can be stored as integer or floating-point images. If they're grey with numbers hovering around 50% grey, it's an integer map. You need to subtract 0.5 from these and then multiply them by a scale that you're supposed to get when you generate the map. You have to enter this value every single time you generate a map. That's an awful workflow. Floating-point maps are much nicer and simpler: they encode the actual object-space distance, with no displacement factors. If you load them in Photoshop, you'll see they're mostly black, with positive and negative numbers (which you can see in the info panel if you hover over the image).
- Texture compression (at least the compressors Unity uses) doesn't work well with displacement maps, even on "high", so leave it off. "Crunch compression" is probably fine (that looks like lossless archive-level compression).
- In my shader, I removed the displacement factor that was in the examples (float displacement doesn't need it) and hardcoded a 0.01 factor to adjust for model scale:

float d = tex2Dlod(_DispTex, float4(v.texcoord.xy,0,0)).r;
v.vertex.xyz += d * v.normal * 0.01f;

 

Bonus points to this forum for letting me spend an hour typing this, then saying "log in" when I click post and losing it all. Fortunately I've been using garbage-tier forum software like this for years and made a local backup out of habit, but come on, folks. That's not something that should ever happen in 2017.

Some followup info:

- Enable triangulation when you export meshes. The problem is that Unity will triangulate if you don't, and Unity and your 3d modelling software (and possibly the software you're generating maps with) won't always triangulate the mesh in the same way. If they differ, any non-planar quads will end up having a different shape. That can cause artifacts even without displacement, but they're even worse with displacement.

You can disable triangulation on import in Unity, but it doesn't work with the displacement code I've been using, and the scene view still draws tris in wireframe mode, so it's hard to tell what's happening. Also, if you disable displacement for a low-end system you'll still be triangulating at render time and end up with artifacts, so I've been avoiding that option.

- To store these as single channel textures, set "Texture Type" to "Single Channel", and--the undocumented bit--set "Alpha Source" to "From Gray Scale". (Greyscale's one word, folks...) The default with single channel will pull in the alpha channel (even if the file doesn't have one), which is rarely what you want. Setting it to greyscale will make it behave the way you'd expect. This should default to greyscale, and the label should make more sense (eg. "Single Channel Source").

Unfortunately it'll still create an alpha texture instead of a luminance texture, which means you have to modify your shader to sample the alpha channel instead of a color channel. It should use the channel that was imported or let you choose, not make you modify your shader. It should also import single-channel images as single channel by default.

- I finally got Mudbox exporting to 8-bit displacement maps to work. It's helpful, because unlike xNormal which only does fuzzy heuristics to create the map, Mudbox can map based on subdivisions, which is much more robust. Set "method" to "subdivision" and do your subdividing inside Mudbox for this. The main trick with Mudbox and fixed-point maps is that the displacement scale only shows up as part of a filename in the export completion dialog. It looks like "disp_u1_v1_g2.512.tif". The 2.512 is the displacement scale. No idea where this is documented.

 

Pro tip, use the back button (on chrome at least), to go back in time and copy your draft

 

(Of course I tried that.)

 

I came across this bug recently- it appears to be in the EXR importer, not the raw texture data, as you can read/write to RGBAHalf textures just fine in memory and get the same result you put in - but the second you save that to an EXR file, the importer kicks off and gamma's the 2.0 you wrote down to 1.375. It seems to completely ignore the importer settings.