Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

VertexDirt - Per vertex Ambient Occlusion baking [WIP]

Discussion in 'Works In Progress - Archive' started by ZFarago, Jun 2, 2013.

  1. ZFarago

    ZFarago

    Joined:
    Aug 13, 2012
    Posts:
    155
    Hi andyz,
    Thanks for the report! You are the second one who experiencing the "grey results" bug. Unfortunately I still don't know what causes the problem, but I am working on it to fix that.
    I like the idea to indicate the processing somehow... this will be in the next update.
     
  2. macdude2

    macdude2

    Joined:
    Sep 22, 2010
    Posts:
    686
    Hey, I'm also having problems with lighting indoor scenes with no back faces. It seems the occlusion only works on solid objects? Is this the case? Is it not supposed to be used to create occlusion for a corridor or something?
     
  3. ZFarago

    ZFarago

    Joined:
    Aug 13, 2012
    Posts:
    155
    Hi, It should work. Did you try different settings? If you send me a test project via email, I will try to figure out what's wrong.
     
  4. andyz

    andyz

    Joined:
    Jan 5, 2010
    Posts:
    2,251
    So does it work ok for you if you import your asset into a new project in Unity 5.1+ and rebake an object in the test scene?
    Edit: Seems the VD-INDIRECTLIGHTING shader compiles with an error
     
    Last edited: Aug 3, 2015
  5. ZFarago

    ZFarago

    Joined:
    Aug 13, 2012
    Posts:
    155
    That's correct. Actually, I have tried all versions of 5.x, as well as all versions of 4.x . Works as espected.
    I think it is not a Unity version releated bug. It is really bothers me as I can not reproduce to myself.
     
  6. andyz

    andyz

    Joined:
    Jan 5, 2010
    Posts:
    2,251
    The problem is the VD-INDIRECTLIGHTING shader not compiling - it throws an error. If I change line 24 so instead of 'sampler' it says 'sampler2D' then it's fixed.
     
  7. ZFarago

    ZFarago

    Joined:
    Aug 13, 2012
    Posts:
    155
    You're right, there is a typo in that shader. But that shader is not used by VertexDirt right now, so it shouldn't affect its behaviour.
    Anyway, thanks for the help.
     
  8. andyz

    andyz

    Joined:
    Jan 5, 2010
    Posts:
    2,251
    Oh I see yes, when I made that change it fixed the baking for a while - but now when I open it again it just bakes gray again so that's weird.
     
  9. ZFarago

    ZFarago

    Joined:
    Aug 13, 2012
    Posts:
    155
    Okay that is odd indeed, please send me an email with that project!
     
  10. jimmikaelkael

    jimmikaelkael

    Joined:
    Apr 27, 2015
    Posts:
    791
    I'm sorry about that noobish question but is there any benefit speaking of performances to use vertex baked AO vs AO maps with the standard shader (Unity 5) ?
    And what about vertex baked AO vs SSAO, how does it compare ?
     
    Last edited: Aug 5, 2015
  11. ZFarago

    ZFarago

    Joined:
    Aug 13, 2012
    Posts:
    155
    Not in a modern machine. It is more matters when targeting mobile platform. Vertex ao needs only a multiplication in the vertex program. Enlighten AO needs to be used in the fragment program, and also costs a sampler I think. Not a big deal in desktop anyway.

    Smarter people will correct me if i'm not right.

    ssao is a screen space post effect, Works in a very different way. It is quick to apply, no need any preparation, but it is expensive even in desktop.
     
    Last edited: Aug 5, 2015
  12. jimmikaelkael

    jimmikaelkael

    Joined:
    Apr 27, 2015
    Posts:
    791
    Thank you for that detailed answer!
     
  13. ZFarago

    ZFarago

    Joined:
    Aug 13, 2012
    Posts:
    155
    VertexDirt Roadmap:

    Based on the reviews I have got recently, VertexDirt will have a major update. (v1.6)
    Faster baking, and more stable working will come, as well as a redesigned control panel and some useful functions like selection sets. This upgrade will be shipped to 4.x and 5.x version of Unity.

    I'm going to aim VertexDirt to be a specialised AO baking tool. It means other planned features like color bleeding or thickness rendering are skipped for VertexDirt. Other side features could be come later and of course support and bug fixes remains. VertexDirt v1.6 coming this fall.

    After VertexDirt 1.6, I have to drop the u4.x support for future development. U4 and U5 have many differences and very difficult to get the same features working parallel. I'm going to achieve this by making a new tool(working title: VertexDevelop), as a successor of VertexDirt. The upgrade to VertexDevelop will be free for all VertexDirt users for at least a year. VertexDirt will remain for both U4 and U5, while VertexDevelop is going to be U5 exclusive.

    Cheers,
     
  14. NorthernVisionStudio

    NorthernVisionStudio

    Joined:
    Oct 18, 2013
    Posts:
    60
    Great news, ZFarago. Let me tell you about my use case, please:

    I have a very dense mesh, maybe 10M polygons. It is a car. So, the ideal situation would be to:
    - bake the car body AO into the vertex data in the editor
    - bake the ground shadow (high poly plane) during runtime to reflect changes in moving parts. Not realtime, but to trigger a calculation on demand.
    - would be ideal to be able to optimize runtime calculation, for example:
    • bake the body without the wheels
    • rebake the AO when the wheel is steered
    • during that calculation, only a spherical region around the wheel is calculated

    I realize that a new mesh asset needs to be built with the resulting mesh. So, it would have to be resubmitted to the GPU.

    My experience so far:
    - xNormal uses GPU and is fast and good quality
    - NVidia Optix can be integrated into Unity to provide AO baking services. We may do this yet so we can get raytraced reflections too.

    Thank you for your time
     
  15. ZFarago

    ZFarago

    Joined:
    Aug 13, 2012
    Posts:
    155
    Well, sounds like a complex problem. The 10M poligons already a big deal, but to get a spherical area refreshed realtime with a density like this would be impossible right now for VertexDirt. What about some screen space solution?
     
  16. NorthernVisionStudio

    NorthernVisionStudio

    Joined:
    Oct 18, 2013
    Posts:
    60
    Thanks for the quick reply. Sorry I wasn't clear enough. Not real-time. Run-time. Just to be able to calculate in the Player as well as Editor.
     
  17. ZFarago

    ZFarago

    Joined:
    Aug 13, 2012
    Posts:
    155
    I see, please note calculating this quantity of vertices run-time would take a great time. It could be done asynchronus, but more slower then. Calculating it offline in the editor would be the optimal.
     
  18. cptSwing

    cptSwing

    Joined:
    Jul 2, 2015
    Posts:
    8
    This would fit perfectly with my shader, I need the baked AO to slot into the VertexAlpha channel pretty please :)
     
  19. ZFarago

    ZFarago

    Joined:
    Aug 13, 2012
    Posts:
    155
    cptSwing: Okay then, I'm going to put this feature to VD and make a small release :)
     
  20. cptSwing

    cptSwing

    Joined:
    Jul 2, 2015
    Posts:
    8
    Great, cheers!
     
  21. ZFarago

    ZFarago

    Joined:
    Aug 13, 2012
    Posts:
    155
    Good news everyone who experiencing plain grey results: bug fixed, new version sent to the Asset Store. Should be out soon. :) Sorry for the inconvenience guys.
     
  22. SilverStorm

    SilverStorm

    Joined:
    Aug 25, 2011
    Posts:
    712
    I am very curious as to what improvements may come, because right now the main concern for me has always been the AO Resolution, I love that it is a sort of mini-beast lightmapper as well.

    So are you planning to somehow increase the resolution even on low poly objects?
     
  23. SilverStorm

    SilverStorm

    Joined:
    Aug 25, 2011
    Posts:
    712
    This is what I will say, the ONLY unavoidable reason I am still using Unity4 is because Unity5 has a bad Lightmapping solution especially in the AO department. The guys at Unity are aware of the issue but based on their Road map seem more interested in VR and WebGl stuff.

    To this day, their is no effective AO Solution for Unity5 - Screen Space AO is very bad, not accurate, introduces many artifacts, is very expensive and cannot run on all platforms.

    If I were you I would grab this opportunity to make a standalone Lightmapper and market it. Theirs no competition.
    All the screen space stuff are ass, this includes Nvidias HBAO+ and Sonic Ethers AO. Don't be fooled; I used them.

    Having said that I am very happy with Unity4 and Beast so far, although, I am beginning to see content makers slowly gear exclusively to 5. I do not plan to upgrade anytime soon unless it's worth it.

    Personally Enlighten is unusable and unstable to me and it's buggy lol.

    Here are some examples of my test renders with beast with the right settings, it's literally beast - no normal maps here and all Low Poly!
    LightMap U4.png Lightmap2.png .
    If you need any more examples or help regarding this just DO IT, DO IT, YES YOU CAN lol.
     
  24. ZFarago

    ZFarago

    Joined:
    Aug 13, 2012
    Posts:
    155
    (I have removed one post of mine above, because I was logged in with bad user sry)
    As you now baking uses the vertices positions as samples and the results are stored in the vertex color data, so the baking detail only depends on the vertex density. So to have detailed bake results even on low geometry must introduce lightmaps and other functions to VertexDirt, and currently this is not a plan, but this can change in the future. Do you thing some beast-like but simple tool can have success against enlighten?

    Alright then :) I will see what can I do. It looks possible to make a lightmapper based on VertexDirt so far, but I am not sure if I can make a fast lightmapper. :)
     
  25. SilverStorm

    SilverStorm

    Joined:
    Aug 25, 2011
    Posts:
    712
    I am more than happy to wait if the results are worth it.
     
  26. langem

    langem

    Joined:
    Mar 18, 2013
    Posts:
    73
    Hi, I just bought your awesome plugin. Works brilliant. I'm just wondering if there's an option to only use it on static objects. I don't want the dynamic objects in my scene to cause AO. Thanks and best regards, Matthias
     
  27. ZFarago

    ZFarago

    Joined:
    Aug 13, 2012
    Posts:
    155
    Hi!

    This is something missing from VertexDirt right now.
    You have to separate your scene by hand by switching off dynamic objects or move them far away. Or you can play the AO distance to exclude them if possible. Maybe in the next version :)
     
  28. PatrickBibeau

    PatrickBibeau

    Joined:
    Apr 25, 2013
    Posts:
    3
    Hi,
    Just bought your plugin (November 06th 2015) and it works great on Unity 4.6.6 but on Unity 5.2.2 i seem to have what you call the grey bug. No AO at all... Is there anything i need to do in order to fix this issue.
     
  29. ZFarago

    ZFarago

    Joined:
    Aug 13, 2012
    Posts:
    155
    Hi!

    Sorry about that. :(
    Did you just upgrade your project using VertexDirt to Unity5, or have you tried download the plugin again?
    Right now, VertexDirt have different packages to 4.x and 5.x.
     
  30. PatrickBibeau

    PatrickBibeau

    Joined:
    Apr 25, 2013
    Posts:
    3
    Im pretty sur i used Unity 5 to download the package...but i will double-check. Tests i did at work might have been done with a 4.x version in Unity 5.... but at home...probably not. Thanks for the support. I'll get back to you with the valisation results.
    Have a nice day!
     
  31. PatrickBibeau

    PatrickBibeau

    Joined:
    Apr 25, 2013
    Posts:
    3
    Well i found something interesting. If I download the package in Unity 4 and update my project to Unity 5...the plugin works just fine. If i redownload the plugin from my new Unity 5 IDE, the plugin fails to create AO.

    Then, if i re-open the same project with Unity 4, and the plugin is still the Unity 5.x one, the AO fails. And then, if I re-download the Unity4.x package, AO works just fine.

    So, for now on, i will use the Unity 4.x plugin even if I'm using Unity 5...

    Hope it helps to fix the issue!
     
  32. ZFarago

    ZFarago

    Joined:
    Aug 13, 2012
    Posts:
    155
    Thanks for the report!

    It seems I have just mixed up the versions in the last submit so thats why you experienced this. :(
    Actually I solved out how to support both 4.x and 5.x, indie/pro versions of Unity with just one variant.
    Thanks everbody who sent me a report, it was a great help to me!

    Zoltan
     
  33. DanjelRicci

    DanjelRicci

    Joined:
    Mar 8, 2010
    Posts:
    308
    Hello! Happily using your plugin since months, but I think there is one issue on some kind of situations.
    Below is a scene of our game, a lowpoly path across a canyon. Vertex AO has been applied to the whole model, but as you can see, some edges are bright despite they are quite sharp (notice the border across the ground and the cliff, or the right side of the road).
    I have played with Occlusion Distance and Sampling Angle, I have also checked for problems in the model, but I got no real clue about this.
    Thanks!

     
  34. ZFarago

    ZFarago

    Joined:
    Aug 13, 2012
    Posts:
    155
    Hi!

    Yeah I can see the problem.
    would you please send that section of the model via email? Please send it in a .unitypackage eyported from that exact rpoject, so keep the same postition and scale if possible. I'd like to play around with.
    (there are some hidden variables in VD which can control the baking more precisely, maybe one of them can fix the bad AO)
     
  35. ZFarago

    ZFarago

    Joined:
    Aug 13, 2012
    Posts:
    155
    Hello,

    new Control Panel, new features with new samples and more detailed docs!
    VertexDirt 1.6 will be ready soon!
    new_panel_01.jpg
     
  36. Ben-BearFish

    Ben-BearFish

    Joined:
    Sep 6, 2011
    Posts:
    1,204
    @ZFarago I have a 3D FBX model of an airport that is split into 3 separate meshes inside Unity: the ceiling, the airport with all the details, and the outer wall. Due to the requirements of the project I'm working on the scene needs to be all white with Ambient Occlusion everywhere.

    I'm trying to weigh the cost of using this tool vs having my artist bake the ambient occlusion and lighting into the textures of the models.

    I was wondering what are the benefits of using this tool vs having my artist do it? My artist mentioned that if they were going to bake the AO into the textures of the model, they would have to unwrap UVs on every part of the model first. Does your plugin automatically handle unwrapping UVs, or does it handle AO some other way?

    Thank you.
     
  37. ZFarago

    ZFarago

    Joined:
    Aug 13, 2012
    Posts:
    155
    I just sent you a mail :)
     
  38. angelcage

    angelcage

    Joined:
    Dec 31, 2015
    Posts:
    3
    eh.where is this AOshader?
     
  39. angelcage

    angelcage

    Joined:
    Dec 31, 2015
    Posts:
    3
    hi,Zfarago,help me pls,when i click start nothing happen.
    1.jpg 111.jpg
     
  40. ZFarago

    ZFarago

    Joined:
    Aug 13, 2012
    Posts:
    155
    angelcage: a new version will be in the store soon, until that, please ask for it in private conversation.
    A good workaround could be to download the package under 4.x, because that seems works in 5.2

    Sorry for the inconvenience
     
  41. angelcage

    angelcage

    Joined:
    Dec 31, 2015
    Posts:
    3
    it's 5.2,but it don't work.:eek:
     
  42. ZFarago

    ZFarago

    Joined:
    Aug 13, 2012
    Posts:
    155
    please write me in priv.
     
  43. J_P_

    J_P_

    Joined:
    Jan 9, 2010
    Posts:
    1,027
    Curious how this looks with low poly models. Any examples?
     
  44. ZFarago

    ZFarago

    Joined:
    Aug 13, 2012
    Posts:
    155
    Some previous posts have some screenshots, post #41, #74 for example.
    Baking AO to vertices needs evenly dense quantity of vertices on every mesh.
    You can not have too long or too big triangles compared to neighbour meshes, otherwise the shading will fail in details.
     
  45. sathya

    sathya

    Joined:
    Jul 30, 2012
    Posts:
    297
    @ZFarago
    Hi, Does it create AO if multiple objects are selected ?. And Does it remove all the vertex color which is already on mesh ? or just modifies it ?
     
  46. ZFarago

    ZFarago

    Joined:
    Aug 13, 2012
    Posts:
    155
    - Yes, it works on selection of multiple objects. The AO will be calculated on selection, but every object in the scene affects the results. Layer based filtering is under development.
    - Currently it removes/overrides existing colors. This will change in the near future.
    A new version is under development, it is quite a rewrite from scratch, and completed with a lot of other requested functions. (color channel filtering, progress bar, etc.). It just need a little bit more time.
     
    sathya likes this.
  47. sathya

    sathya

    Joined:
    Jul 30, 2012
    Posts:
    297
    Cool. Will look forward to it. Meanwhile is there a quick way to append vertex color to meshes. My meshes already has vertex color on them i do not want to replace them.
     
  48. ZFarago

    ZFarago

    Joined:
    Aug 13, 2012
    Posts:
    155
    Please write me in private.
     
  49. backwheelbates

    backwheelbates

    Joined:
    Jan 14, 2014
    Posts:
    231
    Hi @ZFarago ,

    Your toolset looks great! Just a couple of questions.

    What mesh format are the new meshes saved as? Is it possible to just overwrite the existing fbx rather than linking to another object?

    Thanks!

    Eric
     
  50. ZFarago

    ZFarago

    Joined:
    Aug 13, 2012
    Posts:
    155
    Hi Eric,

    The saving tool uses the unity's mesh .asset format to save the mesh.
    Overwrite an existing fbx is quite hard to do, and I think it would have some legal issues as the format is licenced. So Maybe not the best workflow.

    Please note, saving is optional, you don't need to do that, as the VDColorHander component is holds it for you. Most of the time it should be enough.