Comprehensive Decal System (Mesh Decals on Skinned Mesh Already Working - Demo Included)

Important update.

Hello there.

I'm making a decal system, I plan to support mesh, vertex, deferred and texture decals. I've started with the hardest one in my opinion which is mesh decals on skinned meshes, and fortunately it's done already. It still needs to be threaded but it's working fine at the moment. For each skinned mesh hit by a decal, a new skinned mesh containing only the affected area is created, subsequent decals on the same object are added to the same skinned mesh so all decals in this case are drawn in a single call. You can't go more efficient that that I guess.
Here's a link to the WebGL build:
https://googledrive.com/host/0B6RP3uoh07LuRnlVUldrRzFURkk/

Another demo: https://googledrive.com/host/0B6RP3uoh07LuX1ZzdnZOUnFxNVk/
To test it, just click on the stuff, decals should appear. "R" key resets the scene. Please note that it may take some seconds to show the loading bar.

Please lemme know if the test works for you, and sorry for the horrendous monkey mesh physics .

Click the spoiler to expand tech stuff:

Spoiler

The hardest part was space conversion... the fact that skinned meshes 'snapshots' are scaled almost drove me crazy... I had to revamp a lot of code that was working fine for non-skinned meshes. I ended up making a quite cool 'viewer' for matrix operations:

The yellow box is the projector, the yellow line is the direction, the blue dots are a representation of the black wireframe mesh converted to the yellow box local space and the cyan ray is the representation of the yellow line in that space.
After doing that matrix conversions became much easier to debug. Matrix transformations are cool .

 

Hey guys, no one is interested in this? What are you guys using currently for mesh decals on skinned meshes? I see there are some texture based systems out there but that's a different beast. I wonder if what I'm doing has been solved already thus there's no interest in another solution.

EDIT:
After some optimization, you can now paint MESH decals on skinned meshes in real time:

Of course this isn't the intended usage, especially when it comes to mesh decals, but it surely shows how it's fast now . Please note that there are only 2 skinned meshes there (the base mesh, and the decals mesh). Each decal is added to the existent mesh, so rendering is extremely fast too.

 

@AlanGameDev you only gave us like 11 hours over a weekend night in the United States to see your first post. I'm pretty sure that's why no one else has posted yet.

This definitely looks promising to me. Skinned mesh support seems to be lacking in a lot of decal solutions I've seen, so if your version is well optimized, that sounds great.

I'm not too familiar with all the "types" of decals, but definitely something that appeals to me from a non-technical perspective is being able to apply a decal that seamlessly spans meshes (e.g., a large explosion impact decal). Additionally, terrain mesh support is also desirable.

 

Y U NO answer? [/joke]
Now, in all seriousness...
Thanks for your reply. Yeah, I'm using a fast AABB checking for triangles after converted to the projector matrix so triangle selection is extremely fast (btw, that mesh has 8k verts and 7.9k triangles).
I'm by no means a decal expert, but I'm more than happy to share my thoughts:

1- Mesh decals (which are basically a mesh overlay) are the classic decal type. They are the most 'general' solution in terms of decals, however, they are becoming somewhat obsolete for stuff like level details. When you need to have lots of smaller decals they are a great solution generally (when combined). Mesh generation may be slow, but since it's just a mesh like any other rendering performance generally is very good.

2- Vertex color decals, in fact are just vertex color with some shader intelligence to blend textures with the vertex color information. These are very good for skinned meshes that needs a huge area covered in decals. Creation performance is totally negligible and it's good for stuff like RTS characters for example. There may be a bit of shader overhead but otherwise rendering is pretty much 'free' too. You can have some 'detail' texture to have a much better effect and I feel using vertex colors for decals is something highly underrated and underused for skinned meshes. It's only suitable for organic objects with a somewhat uniform mesh density though (what's the case for most characters). Some cool effects could be achieve with for example a base texture and an overlay texture with 'gore' especially tailored for that mesh, so the vertex colors would 'reveal' the gore image.

3- Deferred decals are the today's standard for level details. I don't see them being used much for real time decals but nothing prevents you from using them. They generally are rendered like lights in a deferred pass. I don't think they are suitable for skinned meshes tough and the overhead of having a large number of individual decals have to be taken into account for objects like bullet holes if you plan to have many small ones on screen.

4- Texture decals are basically like texture texture painting on 3D models. You can manipulate the very 'base' texture you're using for your model or you can have an independent texture overlay only for decals. It's basically 'free' in terms of rendering but creation can be problematic. Memory usage could be a problem too since you need many independent textures. Generally you also need a clean UV map with minimal seams and plenty of space between the UV islands otherwise the decals will 'bleed' to other areas. Of course it's possible to sample each texture pixel from the 3D mesh (like 3D packages do) but not in real-time when performance is a concern, so you generally just get the texture point of the decal center and perhaps the boundaries and 'paint' the decal image over the texture. Image operations on the GPU are very fast though, but this is not something I'd really use a lot.

I plan to provide many ways to handle the decals, for example, each mesh decal could be in a separated GameObject so they could 'fade' away easily, or when you want more persistent decals then the approach in that demo is suitable. Basically each dynamic GameObject (with a mesh renderer) has its own decal mesh and decals are added to it. For persistent decals on static objects, I plan to introduce a 'chunk' system, so decals are handled in spatial chunks. That would allow them to be properly culled for example.

These days decals could be more persistent. Even in Max Payne 1 they had persistent decals and it run on systems much less powerful than any mobile device these days. Of course there was a limit on the amount of decals you could have but it was so high that you couldn't reach it (unless with some ammo cheat).

I feel that fading decals break immersion and sometimes it's a waste of resources. Tracking 1000 decals lifetimes isn't really more efficient than just having a single mesh with 1000 decals on it culled out.

I hope that helped you understand the different types of decals a little bit better and sorry for the long text .

EDIT:
Regarding terrains, I surely plan to support them, I just don't know how I would manage terrain detail transitions for mesh decals but that's another problem for the future . Maybe deferred only

EDIT2:
I need to test projections exhaustively, I was having problems with models imported from different 3D packages since their base mesh scale were very different.

 

Thanks for the detailed reply! You're obviously very knowledgeable. I'm looking forward to seeing how your asset progresses.

There are a couple other decal assets I've been watching that seem to be very active in development. One seems to support skinned meshes well (I think perhaps they are texture-based, since it doesn't create a new mesh), but I unfortunately have trouble understanding the developer, due to the language barrier. The other I've looked at doesn't seem to support skinned meshes, although it seems to have a lot of other functionality (e.g., deferred decals).

I like the idea of having just one additional mesh created for all the potential decals created for that object in your solution. That definitely seems better than the typical mesh decal solution. Painting in real time is great, too, obviously.

 

Unfortunately deferred decals aren't going to happen, for more info look here:
http://forum.unity3d.com/threads/how-is-gbuffer-handled-in-unity.418131/#post-2723432
Due limitations in Unity it's not possible to achieve a reliable and performant deferred decal system at the moment. All systems currently out there are either very limited (e.g.: only work on directly lit areas) or do some insane magic under the hood that is basically a lot of redundant checks in a (probably) much slower code what's hardly appropriate for games imho.

Until Unity changes that, mesh decals are by far the best solution for non-skinned meshes, and also they are pretty great for skinned meshes, but sometimes you may want to use either vertex or texture for them, depending on your requirements.

It's bad not to have proper deferred decals, but you also have to take into account that they have some very serious drawbacks too... mesh decals are slow to create compared to them, but the creation can be fully multi-threaded with the only drawback that in some rare cases it may take a frame to appear, what's not noticeable anyway, and considering how most hardware these days have at very least 2 physical cores, it's a sensible way to handle decals. Another thing to consider is that most game models are low/moderate on poly count.

Once you put all of that on the scale, it's not a big loss.

I'm very convinced that for skinned meshes vertex color decals are one of the best approaches. Texture decals may be appropriate in some cases, for example only for the main character, because for many enemies they won't be able to share the same texture and each one will need a separated one what's not good, especially considering that memory is a huge concern especially for mobile devices, and textures take a lot of memory.

Textures also, as I said, have a big inconvenience of UV map dependency. I've tried some things but unfortunately it's impossible to have a performant and reliable system that projects the decals over the UVs with proper transformation. If you get 4 quad points, figure the edges intersections with the mesh edges, and try to project that on a UV map in a stretched way, if one part of the quad in the mesh is in a distant UV island, lots of artifacts may appear. Doing it like 3D packages (sampling pixels based on the mesh texel density) is not suitable for real time. The only real option I could think of is just getting the center of the decal on the texture, and projecting the image there transformed with the triangle information. If the models UVs are tweaked for the decals sizes it should work perfectly, however, that's a big problem considering that 1- many people use 'generic' assets. 2- you want to use as much texture pixels as possible, and you don't want to tweak your UVs if you decide that you need bigger decals later on.

Of course an option would be to create an independent triplanar UV only for the decals. That has some serious drawbacks though like occluded areas being mapped on the same coords as others. Automatic unwrapping algorithms unfortunately are not good too since seams are important in this case. It's possible to just select all triangles affected by the decal and use them for the actual texturing though. But that's a really huge job for a not very adequate result imho.

I plan to support texture decals but I won't actually recommend them for most applications. They sure will work wonders for properly tweaked models, but the drawbacks are very serious.

 

I was contacted about this thread, so I feel I have the duty to clarify something:
I recently lost a loved one, it was very traumatic and this project is on hold at the moment. I sincerely don't know if I'll ever work on this again, or do any kind of game development. My apologies.

 

My condolences, and best wishes for you and those around you.

 

The skinned mesh decal painting looks amazing - something that Unity desperately needs built in! Have you thought of open sourcing the project?

My condolences for your loss. Hope you are doing well.