VertexDirt - Per vertex Ambient Occlusion baking

Hello,

VertexDirt is my hobby project at home, under development for a year now (well actually it could be done in weeks if I had more freetime ), and finally the alpha version is live at the Asset Store! Try it if interested.

Asset Store Link
PDF Overview
Original WIP thread
Sample Video: Spheres

VertexDirt is a Unity plugin for baking many things (like AO) to vertex colors.
The baking itself is a quick progress, but not realtime, so the method is to select some object, tune the parameters, hit the Bake button and wait for the results.

Currently, the source is ready to bake many things, but actually only the Ambient Ocllusion Tool window exists yet. This means, if you are friendly with scripting (and shading), you can easily expand the basic tool to bake more exotic things. VertexDirt includes fully documented source code.

Alpha version is for testing and feedback purposes, so feel free to comment when you had try. This is the time when I am going to spend to fix mistakes and bugs, so big new features will come after this stage.

Screenies:




Cheers,
Zoltan.

 

Any feedback guys? I would appreciate if you would write me your thoughts.

 

Amazing! Excellent work you got there! However I would like to ask about this white shade at the bottom of the model, isn't it supposed to be dark? or am I doing it wrong?

 

Thank you TVishahan. You may try to move up the building a little bit, as the vertices on the bottom corners are to close to the ground or maybe lower than it. I hope it will help. It would be really helpful if you could send me a sample project with this exact bug.

 

Alright so I move it up a little but now I don't get the occlusion from the bottom plane (or effect is not powerful enough), here is the sample scene.
[UPDATE: Sent via email]

 

Please send me via email and remove it from here. I will check it later.

 

Oh, by the way, do you plan on creating a textured shader and a lit shader?

 

Alright, after taking your advice and implementing my own (partially) colour shader, It looks great!

 

Hello!
Everything works fine until I try to build a project.
When I try to build my project, it stops it because there are compiler errors:

This error only appears when trying to build a project, and never appears at any other time.

 

Hi!
This was unfortunately my mistake, the plugin's file structure is wrong. I have fixed this bug couple of days ago, but it is not live at the store yet. If the fix is urgent to you, please write me in private.

 

New version is out, reported bugs fixed, two new sample scenes and several vertex color shaders included.

 

VD Ambient Occlusion - different parameters, different results.
If you change the sampling angle, you can modify the contrast of the results. Wide angles makes more details visible but with less contrast.
Occlusion distance controls the range of the occlusion. If you are using really small values, a specific verticle only shaded by the closest neighbours. Using large values can affect a vertex from long distances.

 

That looks amazing, can't wait to try it out!

 

Well, this is already on the Store
This picture is just a "cheat-sheet" for the good old AO function
New features still under development, pictures soon.

 

Good news everyone who experiencing plain grey results: bug fixed, new version sent to the Asset Store. Should be out soon. Sorry for the inconvenience guys.

 

Hello. I just started using this tool and it seems pretty good! I have hacked the Unity standard shader so it multiplies the diffuse colour by the vertex colour. This works really well and gives the objects a good AO look. I've managed to save the coloured meshes into prefabs so that they get used all round the scene successfully. All good.

But unfortunately I have a problem. Everything looks good when I'm running in the Unity editor, but when I make and run a build, the meshes that have been modified by the vertex dirt plugin are all messed up

It looks like it might be something to do with the enlighten light map, like the lightmap UVs are messed up or something. Do you know anything about an issue like this?

I can't take any screenshots right now because I'm at work, but I might be able to take some tonight if you think that would help.

 

hi.
Sorry for the trouble,
Would you please send me a sample project via private?
I'm not sure what the problem is, maybe you're right and enlighten conflicts with VertexDirt, so I am going to find out.

 

I'll make a small test scene tonight. Have you tested it with enlighten before?

 

Not much yet.

 

I have tested it with enlighten. Working on to fix the problem.

 

Okay, problem fixed, soon in the asset store (testing needed under different unity versions, plus the review time)
-completely rewritten VDColorHandler Component
- correct prefab control, no need of saving meshes as assets
- no mesh leaks, or missing mesh / colors anymore (rare think, but it could happened before)
- works with static / dynamic batching (could messed up enlighten results on build before, see the quote)
- works with enlighten realtime prebaking and normal baking.

 

All sounds good!

 

I'm trying to use your asset for the first time and am unable to reproduce the baking in your sample file in 5.3.4f1 without turning on legacyMode (which the code says is only intended for use with Unity 4.x projects).

When legacyMode is off, the brightest value the code bakes into a vertex is about RGB(203,203,203). When legacyMode is on, the same settings bake RGB(255,255,255) into the non-occluded vertices. I'm using a distance of 100 and an angle of 100, but the darkening happens for all values when legacy mode is off. This extra darkening in non-legacyMode is problem because it makes the baked assets not compatible with existing lighting and scenes. I'm assuming I should just go ahead and use legacyMode but the code is pretty clear legacyMode is for Unity 4.x and I didn't know if this might be a sign of deeper problems.

I'm also not finding any guidance in the docs or the code on when to use compatibilityMode, can you give any info on when one does or doesn't want compatibilityMode?

(all of this is with VertexDirt 1.59.31)

 

Hi!

Please use the legacy mode switched on in these cases. It's a bug in the code. Don't worry about Compatibility mode.
Sorry for the inconvenience.

 

Hi, just a question. I'm interested in your asset, it looks amazing, but I'm using models painted in blender using vertex colors. Is there a way to use VertexDirt without overwriting a model's original vertex colors? like, just blending the new AO over the previous vertex colors? or is the process destructive?

 

Hi Alverik,
It is "destructive" right now, but really easy to change it to something else, like multiplying in code. (I can help you with that)
Next version will be using multiplying as it was already a feature request But it has some other changes needed to be documented (writing docs is my weakness - sorry), so it is not available in the store yet.

 

Sorry for the late answer. Anyways, thanks for your response, I really want to try the asset out, but due to personal reason's/budget I'll have to wait a bit. Hopefully I'll be able to catch the next version. If the AO can be multiplied over the original vertex color, then it will really be a God send!

 

Hello - Avagy tiszteletem a kolleganak

A friend of mine showed this plugin to me and recently I bought it as well.
However I found a little error which makes our life a bit harder on our small project.

I make small group of objects, put them in prefab and use vertexdirt and save prefab.
So far so good in the editor everything is as expected.

But when the code use the prefab and place it to the level there is nothing to render.

The reason is in the asset browser when I open the prefab, all the models missing the Meshfilter's target - it's own model.
If I put the prefab to the editor it works again, but the prefab itself doesn't have them, however if I set the proper models in the prefab in the asset browser, then everything works perfectly.

However everything is lost again if I change something in the prefab and save again - and it happens a lot during development...

So my question is, could you take a look at it, why it's breaking the connection in the Meshfilter component when saved in prefab?

Also the AO works a bit funny. I have an object what doesn't generate any useable AO, not even in the deep corners. I tried many parameter variation, but nothing. I cleaned up the mesh in many ways what I know in 3D app, it seems okay in every way but still...

Well beside these it's an awesome plugin, I can only recommend!
Cheers!

 

Szia!

when the code use the prefab and place it to the level there is nothing to render
I'll check this prefabing bug, really sorry about this. Make procedural meshes and prefabs working together is an old task for me, and I almost thought I fixed all related bugs in the latest version

I have an object what doesn't generate any useable AO, not even in the deep corners
Some kind of geometry can have really strange looking AO. Please send a sample model via email and I'm going to check if I can figure out some sort of solution.

Kind Regards, Zoltan.

 

Hi, a small question prior to purchasing – is it possible to bake in runtime? Or alter the code to make it bake in standalone build? I realise there would be a lag, but technically - would it be possible or there are some Editor API involved on which the whole baking system relies?
Thanks,
Olaf

 

TheodoricSacron:
when the code use the prefab and place it to the level there is nothing to render
I have an object what doesn't generate any useable AO, not even in the deep corners
Both bugs are fixed, and available on Asset Store

made_it_app
Not tested for a long time, but is should work runtime, yes.
The main VertexDirt class is a static UnityEngine class, so its available on runtime for other scripts.

Just call VertexDirt.Dirt(Transform[]) from your script and the bake starts.
To set the properties of the Bake, fill up the VertexDirt.settings structure with your parameters.
This is a VertexDirtSettingsStruct class and you can find its parameters documented in the code of theVertexDirt class.
The VertexDirtWindow editor class is a good example for accessing VertexDirt.

And yes, the run will be freeze for the time the baking being calculated.
I hope that helped If you have further questions, please write me via email.

 

@ZFarago Thanks for the answers, on other question: does it support fragmented shaders or just surface surface (if both support vertex color)?

 

It should work with fragmented shaders. Any kind of shader wich is supports vertexcolors, will work fine.

 

Hello, I have a question prior-purchasing.

Is it possible to achieve something like this, on this kind of geometry (little buildings)? I mean, for the small details and more.


Thank you.

 

Hi!

Well, it is possible to get similar results, but need detailed geometry with good and precise topology.
I suppose it would be easier to use lightmaps instead, but of course it will have a smaller memory footprint.

I made a quick test to showcase this scenario.



I hope this helps.
Kind Regards, Zoltan

 

Hello,

Did you model that quick scene just to answer my question? I think I love this plugin even without buying it yet haha. I think this result is perfect for what I need.

One last question.

I understand when you say "VertexDirt is a Unity plugin for baking many things (like AO) to vertex colors. (...) but not realtime," that means that it is not replacing the original lightmaps? Sorry if it's a confusing question, I'm not an expert on the subject.

I have this situation.

1. A scene (standalone build) with some objects, these already have their own lightmaps or baked textures for each INDIVIDUAL.
2. The possibility of adding new objects (placeholder).
3. But if I put two objects together, even though each has its own lightmaps, there is no AO for the relationship between the two.

So, is it possible to generate that AO of both objects without overwriting the lightmap, a kind of combination?



Thank you very much. Amazing support.
Sorry my english.

 

Sure I did! I like to test VertexDirt in different cicumstances and it didn't took too mucj time either

Well VertexDirt is an "offline" generator which means the results needs to be prebaked, as it is not able to generate the AO again and again in every frame. You can use the baked results in any kind of builds in every platforms of course.

One more thing, VertexDirt is not offically supports baking in runtime, because I'm not tested it completely yet, but it works in theory (it will probably hang the game while baking)
The main VertexDirt class is a static UnityEngine class, so its available on runtime for other scripts.
Just call VertexDirt.Dirt(Transform[]) from your script and the bake starts.
To set the properties of the Bake, fill up the VertexDirt.settings structure with your parameters.
This is a VertexDirtSettingsStruct class and you can find its parameters documented in the code of theVertexDirt class.
The VertexDirtWindow editor class is a good example for accessing VertexDirt.

Probably you can update the AO in runtime with VertexDirt, but I'm afraid the updating itself will take at least a couple of seconds and hangs the game until. Other thing is the topology and placing of the buildings. Vertex AO (and not just VertexDirt) could be behave really bad sometimes in cases like that. With detailed object it should works fine, but the more vertex count the models using will lead to greater hangs while the AO updates.

You may try to do some tests with image effects too like SSAO as they can produce realtime results.

If you have further questions, please feel free to contact me in email and I give you more precise help.

 

I think it's a perfect solution for me because "VertexDirt can bake objects one by one" - I mean individual, I just read the documentation about "save baked meshes" and in any case I know it will be very helpful.

Awsome, thank you!

 

hello,

This is a great tool, Thanks.
I do have one error that i am not sure how to handle. Please see below:

Mesh.colors is out of bounds. The supplied array needs to be the same size as the Mesh.vertices array.
UnityEngine.Mesh:set_colors32(Color32[])
VDColorHandler:Awake() (at Assets/Plugins/VertexDirt/Scripts/VDColorHandler.js:19)

do you know what might be causing this? the meshes look fine, but i keep getting this error. Please let me know what might be wrong here. Thanks

 

Hi!

It looks the vertex array becomed shorter than the Vertex Color expects. If you have a mesh with a color handler and update the mesh, the verticle count can be different, depending of the mesh changes. Is the error shows in Play mode only, or always in the editor?

You may try to Rebake that particular object and check if the problem resolved.

 

@ZFarago can i ask a feature request?
vertex color channel selector where we can select to which channel the vertex ao to be applied into the vertex data?
Edit : would like to modify it but i'm not good with js :/

 

Sure thing! Will be in the next version!
Until that, just throw me a mail and I help you to modify.

 

sending you a PM

 

Hi, the tool doesn't work on my Skinnedrendered mesh. Is it normal ? The console says :
"Baking of 1 objects (0 vertices) finished in 0 seconds.
UnityEngine.Debug:Log(Object)
VertexDirtirt(Transform[]) (at Assets/Plugins/VertexDirt/Scripts/VertexDirt.js:132)
VertexDirtWindow:OnGUI() (at Assets/Plugins/Editor/VertexDirtEditor/Scripts/VertexDirtWindow.js:87)"
If I apply on a badic object, it works (so the plugin works and is well installed).
I tried also on several Skinned meshes with the same result.
Version is 1.59.11
Thanks

 

Hi! Skinned renderers not supported, sorry. I can add this feature in the future.

 

VertexDirt 1.6 in the asset store!

key improvements:

- Complete rewrite to c#
- improved baking. significally shorter baking times
- many bugs fixed

Enjoy!

 

@ZFarago
Glad to see the update to C#!

Unfortunately, new version doesn't seem to work correctly for me. I'm using Unity 5.6.3 and the bake with the following settings in your sample wheel scene results in this:



It's probably not an intended output (for some reason, green channel is the only one receiving an AO bake, and on some of my custom meshes I occasionally see pink-ish areas in the areas which should be deep black). I couldn't pinpoint the source of the problem yet, but it's probably related to some typo in Color or Color32 constructor somewhere, mixing up the channels.

Checking the issue further with a custom shader, it looks like vertex AO is only ever written to Y (green channel) while X/Z (red/blue channels) remain at 0. Range of values written to green channel is capped somewhere around middle gray and never reaches 100% even on completely unoccluded surfaces. This also has a side effect of luminance written to W (alpha channel) being even darker. Not sure where green-only color is coming from, though.

 

uhh sorry about that! Well, actually I'm a bit jealus because I've tested for weeks and never turned to Hulk to me...
I look into the problem, and post an update to the store. Until that, please throw me a mail so I can send a patch immediately once it's done.

 

Please send it to if you'll figure it out, thanks!

I've checked it a bit more, and it seems to be a problem arising from bad texture content. Just uncomment the commented lines you left in OnPostRender method of VDSampler, and you'll see this:



Just as expected, the values are always at 0 in red channel, 0 to ~0.5 in the green channel, and 0 in the blue channel for every single step. Maybe it's a problem with the shader that's used for the bake?

 

Probably, but your Control panel window also have strange enum in the "Blend to existing colors" property, I have to investigate that too.