Assets [WIP][Sept 16th] Terrafirma - An Auxiliary Terrain Engine Replacement and Extraction Toolset

Can you clarify your add pass/highlighting comments? the second screenshot shows more contrast and color it seems, but is quite a bit slower. Or am I missing something?

 

Yay! Can't wait!

 

The first passes look exactly the same. Sorry I shouldn't have posted that so hastily, the top screen shot is my system. The bottoms is unity. The color and contrast differences have nothing to do with performance. Just shader implementation.

Basically the transparent part of the add pass was slightly tinted, and not completely transparent. It was adding a weird film-ish effect over the previous layers.

It's fixed now however, aside from the obvious tiling differences the shading is now the same. Im fixing the tiling now and should take at most twenty minutes. Once that's done i'll rebuild the demo scene and post a little later on.

Unity Top Terrafirma Bottom. Edit** Deep Profiling is on.
Added an update screenshot below this one with out deep profiling.



I should be able to smash more off the cpu aswell, the batcher is optimized but is running more then it should every frame among many things. Still taking half the time as unity so for that I am happy, and feel I can show off something functional now.

I know I probably could have waited before posting this. After the UV offset and scaling was integrated in the new shader set I built today, but figured Id show that none of the previous color/contrast issues were a concern as of now or in regard to performance.

Edit ** Deep Profiling Was on in the image Above.. Below is a better comparison.

Terrafirma Top Unity Bottom

 

It's looking great and performance is fantastic.

 

Awesome, thanks for the extra clarification. And I agree with others, looks amazing!

 

Thanks!

And sorry again for the confusion.. That was my bad.

I did a few more things out of stubbornness and fixed the uv offset/scaling issue. I'm going to get some rest for today, but I'll give the demo a final glance and post it when I wake up tomorrow.

I may or may not add in handling for frustum cull as well depending on how demanding the task is before I post the demo. Should be trivial and it would bump down draw calls quite a bit, though frustum culling unity side is already happening, the batcher doesn't take this into account as of yet. With that in mind however, I'm managing a solid 1000+ fps with out that compensation.

I'm also writing my own basemap, which will transition nicer and reduce draw calls further when using more then 4 textures.

So i'll release the demo I have prepared today either way. Ill finish the editor while working on the the two above features as well over the next few days. Once all of that is complete, i'll ensure the interface is finished while working on the documents and everything will be ready for first release.

 

First off would like to mention I fell a bit behind yesterday/today. I'm going to spend most of tomorrow/today updating thing to reflect every and upload the demo, I've actually started as of now but figure I'd mentioning that.

A few major updates as well;

I got the LOD transitions system complete, and added in frustum culling. I still need to optimize things a bit and will post "bench marks"/screenshot comparisons in a couple days. Here's a function preview though :



Also works well with batcher,



In-regards to optimizations, these are nothing major. Frustum culling is a bit taxing due to some calculations i'm doing every frame which I could be doing only when the terrain changes positions. So nothing really to be concerned about just been swamped these past few days.

 

Any chance that you look into compatibility of this system with RTP Terrain shaders? Don't know if this is a possibility given how deeply RTP integrates with the terrain system, and some functionality might be duplicate, but RTP has a big set of existing fans and users, and Tom from RTP might be happy to work with a more performant Terrain system as a base.

Your system looks promising, definitely ticking all the boxes where the old Unity terrain system is lacking.

 

Keeping an eye on this one

 

Yes, I actually have. I myself am a fan of RTP. I'm going to be implemented a few features similar to what RTP offers, when I start work on displacement mapping, but for the sake of scene compatibility as well as user preference, breath of features ect, it's something I am more than considering.

Thanks you as well, if you or anyone has any features you would like implemented definitely don't hesitate to ask.

Just a quick update as well, I'm about half way through updating the main post; Did some optimizations as well over the past hour and everything is running great with all possible layers of calculations going in the following order;

-Calculate the view frustum planes
-Initial level check with frustum culling calculations.
-Level transition finalization (Gradient Method Only)
-Stitching Check
-Level Batching
-Mesh Stack Adjustments and Rendering

Terrafirma Top Unity Bottom


Terrain Patch Divisions are the same, the batcher is doing a really great job though reducing draw calls down from 1303, to 296 With one less set pass call.

I still have more optimizations to do, for the time being though things are looking great for beta release. The main post will be updated soon as well. Ill post a link to the demo as well as video here and on the main post when I finish main posts refacing.

 

Update

I'll start off by saying there's a demo link at the bottom of this post. I'm still working on the main post and got super carried away optimizing. But for jumping the gun ten times here's something for the mean time.

This demo is the first of 5-6 I now plan to post over the next few weeks, i'll post something similar later with more settings to control patch size, transition type, level distances, ect; For now take this as a direct performance comparison when aiming for equal quality transitions between both systems;

Demo Scene Terrain


Terrain Setup
This terrain is using 2k height maps to represent 6 sq kms, the height map i generated hastily doesn't do the geometry justice and that many points is honestly overkill for this small of a region. But the set up works great for bench marking/stress testing.

Notes

-It's using the gradient method and frustum culling as well as being batched.

This differs from the first demo in a lot of was; First off the patch sizes are now set to match Unity's rather then being halved/eighth'd which resulted in more noticeable transition pop-in in order to reduce draw calls; As mentioned the patches sizes are now the same in this demo; I personally have a hard time seeing mesh transition with out the wire frames.

The gradient method is far more taxing then radial method (not noticeably) but in terms of transition quality it is the best option.

The system being displaced on the shader comes with the cost of having to handle frustum culling directly. The trade off in having to do these calculations is they reduce batching time, and level transition post processing when using the gradient method.

Should also mention there are less options then last time. This is basically a direct comparison as to how my system would handle the same scenario when attempting to emulate unity, both not compensating quality; Attempting to deliver the best interpretation of the height map with the least amount of geometry.



Performance Below :

Editor Comparison ; Demo Comparison Below

Unity Top; Terrafirma Bottom


720 x 480 Unity Top; Terrafirma Bottom


1280 x 720 Unity Top; Terrafirma Bottom


Final Notes
Fill rates kill performance as I'm sure is evident. But between examples there's a speed difference between 6.7 and 10 times, between meshers. Definitely try at both a small resolution and large one.

The shader is basically the same as the old built in alternative and differs slightly, mainly being darker and less washed out in my opinion.

Anyways; The demo is below. I'm going to continue updating the main post to reflect the new system. I figure I can get the editor for the most part finished by the end of tonight as well.

So with all the delays I will say, with the all the work I've managed to complete, release is now looking a lot closer. Yesterday I managed to get a lot of new features put in, cleaned up code and optimized everything I planned to over the past couple weeks on the side with the exception of a few things.

The mesher is now finished along with the shader; The only things i need to get to before release is one optimization to the batcher.

The system now handles the the non-instance mesher much the same as the instanced; In terms of frame rates I noticed only a 100 fps difference running between 1200 vs 1300 using the instanced method over the non; If anyone has issues running this on a non dx11/opengl 3 gpu let me know.

Free Frustum Culler
I'm also writing up a frustum culling system ill be posting here free of charge on the main post for anyone who wants a method that emulate unity's; I'm not sure what benefits will come from this but things thus far look promising. Got the get camera planes function as well as something to test points against it (Bounding box corners). As of now it's a little slower then unity. I figure I can make up for this once I move away from using unity bounds and plane objects due to get/set calls and I have all the math I need to accomplish this anyways, only needing a few vector to represent the bounds and two to represent the plane.

Would also like to mention I'll be moving code to c++; I figure I might as well push performance for those who don't wan't to dabble or modify the system, though it will still be possible to do so; I'll still include all source code as well. Basically this won't change much, i'll leave an option to handle things via c# or through the dll i'm experimenting with currently. That way if your more comfortable using c# the option will still be there to use everything as a template for anyone's future project.

Shaders
Would also like to mention most shaders should be compatible still; The only difference is you made need to modify them yourself to add #include "TerraSplatMap.cginc" vs #include "TerrainSplatMapCommon.cginc"; The front end of my shader solution is basically identical to the built-in solutions with the exception of superficial differences between labeling. The back-end is something else though, i'll be documenting this.

Demo Link :

https://drive.google.com/open?id=0B0VzDnpEwY6FaTJNSDA1WU5BT0k

 

@Murpenstien

Very impressive demo! Wasn't sure what your specs were, so I was eager to test this using VR's minimum specs (i5, 970). Went from 100-150 fps on average to 400-500! (Worth noting I had some other apps open so performance may be impacted.)

Much more importantly, even with this simple scene Unity would drop to 60 FPS when looking at the terrain from one end to the other. Terrafirma on the other hand had almost completely stable FPS.

Great work, looking forward to release!

 

Thank you ! And wow!! Guess I can say it support's vr, lol!

My specs are low tier; I'm running a five year old i7 (950 @ 3.07 ghz) and nvidea 950 gtx; For the sake of showing off here's a 4k heightmap 40km x 40km terrain i'm working on for a test scenario I'm getting ready for Lennart, (I don't know how the tagging system works. ) It doesn't even run on my system using unity.

Unity Top Terrafirma Bottom



I can't even run this set up with unity. I still need to adjust the gradient system to compensate for the greater draw distance. (40000 units) but as mentioned; it's actually run-able. Should also mention I'll be supporting height maps up to the max dimension a texture can hold.




40 000 x 40 000 meters/units (40x40 km)

I also needed to double the patch size from the previous example here; Currently the map is divided into 64x64 due to stitch mapping limitations; i'll be adjusting the stitch map to hold more patch data. I figure I can take more work off the cpu by moving things over to native code; I still have more optimization work to do in terms of culling and the a small modifications to the batcher as well. (Not a lot of work, just "complicated" stuff..)

Once everything is finished though i don't think it will ever be possible to even hit those numbers with my system, even when the patch sizes are enabled to the same size between systems.

I'll actually post this later, might make some peoples computers sad; I know mine one was when using the unity terrain, lol.

Things are looking really good at the moment though. Really glad I re-built the mesher.

 

i5-4590 and RX 480 8g. Went from average of 90fps to 400 fps on fantasic. Seems like a pretty good improvement to me.

 

Thank you, I obviously can't do much aside from batch things further to alleviate gpu load. The option will be available but at a cost in memory. How much so, i'd need to calculate. Fill rates seem to be a heavy hitter, but when running at lower resolutions windowed, i noticed a substantial difference on my machine, with unity staying roughly the same and terrafirma up to 1000 fps;

Small Update / Frustum Culler

Would also like to mention as I figured frustum culling could be optimized; Here's the stats for my culling system vs unity using functions that do exactly the same thing :

-Get Camera Frustum Planes
-Test Bounds Corners Against Them

GeometryUtility.CalculateFrustumPlanes() = LODHandler.CalculateFrustumPlanes()
GeometryUtility.TestPlanesAABB() = LODHandler.inFrustum()



It requires a new plane and bound objects which i'll provide everything needed to set up as well as a visualizer and a faster slightly-less accurate method, which wont effect anything the inside frustum. Basically it will include the odd patch outside the frustum along the edge of it, differnt from the LODHandler.inFrustum(), which requires 8 tests against 6 planes vs the new method i'm adding in which will require 1 test against 6 planes. Should also mention, there's no GC hit either.

I'll post the scripts as well as a small scene in a package either tomorrow or the next day once I move the code to it's own object. Should be great for anyone who needs a faster frustum culling solution, i wont charge for it either and will most likely continue to update it as needed. I'll post better benchmark as well. I'm just running both function once per terrain patch in the scene i'm currently modifying the system in in the screen grab above.

 

I was about to suggest a large scale terrain demo, but you beat me to it, nice! 40-60km draw distance is what I'm interested in, so it's awesome to see it already working nicely.

 

we are trying to do a scene with 4 terrains of 40x40 km each with terraforma, and real-time spawned instanced vegetation on the terrains.

 

Great demo!
I'm running a 4k monitor, 1070 GTX. I had framerates of 100-200, and was a little suprised...was hoping they would be higher.

Then I saw the itttty bitty text checkbox that was still set on Unity terrain
Great work Murpenstien!!!

 

Some stats on an even lower rig, my ASUS laptop with NVIDIA 750M and Core I7 @1280x720 i get 130 FPS on Unity terrain, 90 fps using Terrafirma, so maybe Unity terrain is a better option for older cards ?

What I don't like about unity terrain is that you can see the detail textures popping up from the basemap , while there's no transition on TF.

Eager to buy as soon as the beta is out !

 

I tested on an Intel laptop with integrated Intel HD graphics card and the results were virtually identical. I got about 21 fps with Terrafirma and about 20 fps with Unity. I also noticed that as the camera moved from left to right I could see the mesher at work at the edges of the display area. So maybe the culling isn't quite right just yet.

 

I'm going to guess integrated graphics is causing both a bottle neck, meaning the gpu is being so over worked in both cases the gains from the cpu aren't really negligible, and the edge issue. I have an older laptop, with HD chipset, so i'll check this out and see what I can do, aside from the visual error (I'll fix) not present on all the dedicated cards i've tried, filling the screen and drawing the geometry is always cost to resources. Also what version of direct x are you running?

Until my culling system is done I am using unity's at the moment with the exception that im manually calculating the bounds only when the terrain object itself moves... So that's very strange, i'll make a compensation regardless.

That's very strange, i just ran it on my girl friends laptop, nvidea 750 m core i5; Totally different on my end. I'm going to send you a pm and we'll see what's up. Thanks for the feedback anyways.

 

[QUOTE="Murpenstien, post: 3007255, member: 464587"
Until my culling system is done I am using unity's at the moment with the exception that im manually calculating the bounds only when the terrain object itself moves... So that's very strange, i'll make a compensation regardless. [/QUOTE]

When do you do the culling, could it be before the camera update?

 

It's not my normal development system since my main computer is in for repair, but I thought I'd give it a try and see how it did. I have Windows 10 on the laptop so I assume it's DX11. I'm sure the demo would do much better on my regular dev system. What I'm hoping for the most is that it improves performance on the Wii U.

 

I think Lennart is right, woops my bad... I don't think I was seeing that due to how little the frustum changes at the frame rates I was getting. Sorry about that. I'm going to get a few copy's out there with a few people who are wiling debug and give output on the system as I finish up the rest. Ill spend tonight and tomorrow getting a build to play with and I'll contact you and Frednaar. Thanks by the by way for your input. Really glad I was able to get to the bottom of that simply. xD And sorry again!

Never should assume anything.

 

In-regard to this again, I sent you that PM; I had my girlfriend send my seen screen on her end, so im not sure what's going on but im interested in finding out. I'll also post two more demos with same scene using different height maps, this situation is a little taxing, in alot of ways, im using a 4k to represent 40 km ^ 2 above and a 2k to represent 3 km ^2 here, basically it's over kill on both systems.

Had her run it on fastest using 640x720;

750 m core i5 - Unplugged/On Battery

750 m core i5 - Plugged-In/AC


Ignore the "v2", just a label i used when i started remaking the mesher.

As i said though, ill post more demos and I'm eager to work with you.

 

asus laptop gtx 970m, i7-4720HQ, 1920x1080

Unity terrain

-----
terrafirma

 

I tested it on my crappy work laptop; got interesting results.

i5-4300M w/ integrated Intel HD Graphics 4600, 1600X900

Unity top, Terrafirma bottom

 

I have tested on my laptop. (Fastest camera, everything in FPS)
Full Screen, 1920x1080, Fantastic settings.
(Laptop is MSI 60GT 2PC Dominator, i7-4810MQ@2.8GHz, 24Gb RAM)

On Battery, Intel HD 4600:
- Unity Terrain: lowest: 10, highest: 88
- No Unity Terrain: lowest: 15, highest: 30

On Battery, NVidia GeForce GTX 870M
- Unity Terrain: lowest: 16, highest: 124
- No Unity Terrain: lowest: 68, highest: 192

On Leash (charging), Intel HD 4600:
- Unity Terrain: lowest: 14, highest: 52
- No Unity Terrain: lowest: 14, highest: 30

On Leash (charging), NVidia GeForce GTX 870M
- Unity Terrain: lowest: 50, highest: 196
- No Unity Terrain: lowest: 71, highest: 368

---
Keep up the good work, this asset seems interesting. I have subscribed to this thread immediately.

 

In regard's to circumstance where both systems appear to run the same keep in mind fill rates; Both are capped at 38 ish milliseconds. You're getting a bottle neck essentially filling 1600x900 pixels;

I posted two screen shots above that capture this; The one running at 720p runs at 500 fps, and the one at 480p is at 1000 fps.

Essentially, my system isn't going to make it any less taxing actually drawing to the screen, in terms of the cost to draw a mesh with same geometry on my system vs unity is no different in terms of performance, my focus was to alleviate resources on the cpu.

For the sake of entertaining me, as I mentioned with the demo, you should try running at a much lower resolution and see if there a difference. I can't guarantee there will be considering the hardware but worth a jab. I'll post a small screen grab showing the actual processes at the bottom the bottom to for unity as well as my system running the same scenario.

Hello, thanks!

Try running the system at a lower resolution if possible, with the 870m you should see a major difference at 720x480;

Great example here though, notice how when running off the battery terrafirma isn't taking a hit? Unity puts a little more work on the gpu, while terrafirma is actually far less taxing and relies on very little cpu to do what it needs to. When the gpu is running with less power; Handling the unity mesh and drawing to the screen is crippling unity there while terrafirma is doing just fine.

I'm going to post a 40 x 40 km terrain tomorrow; Would love for every one to give that a try, i posted a screen above, and I can't even run it with out terrafirma; Like 58 000 batchs on unity at 4 fps vs 480 batches using terrafirma at 500 fps; I'll increase level transition distances but, in terms of batching, those numbers won't change.

Some Updates
Did a lot of optimizing;

My frustum culling system in the end was running at the same rates as unity, i ended up writing a new three lay system that culls dynamically rather then check against every bound that makes up the mesh;

Shader has been dramatically optimized, performance wise it's not noticeable, just requires less memory and operations; Also added a new stitch map that more compressed and supports map sizes I don't think any computer could possible handle. 256 000 000 + terrain patches;

Also reduced memory going from 30 on mono down to 17 mb and reduced mesh memory down from 4 mb down to 2.4 mb;

managed to get a lot more "meat" off the work being done by the cpu as well; Idle im registering 0.03 ms in related calls and 0.30 in rendering related work on the gpu for 32 x 32 meshes representing a 1024 heightmap; When moving since all operations are static, (The same cost no matter how fast the camera is moving/the mesh is changing), without culling optimizations against all mesh bounds, the system always stays under 1 ms on my end.

The only thing left that is costing me precious time, is the batcher. I'm going to finish that tomorrow; I wrote it really naively, and it's actually really cheap in it's current state but as it stands it's the last major resources eater..

Processes, Terrafirma top; Unity Bottom;


Note; The none deep example reflects actual performance better then deep, due to profiling over head; In that regard, we can still see where our time is obviously going when drawing the mesh;

Actual processing; Deep profiling Top, Non-Deep Bottom

Spikes on the graph occur when the camera is moving, flat line is when the camera is idle.

All of these stats come from rendering the same terrain object at the same settings; The batcher once i finish it should barely register in this circumstance; I believe with native rendering on most platforms i can remove the mesh filter set call when using the non instanced version; And frustum culling calls can be removed as well for a faster more dynamic approach as well as the gc call for getting the planes.

Most of the editor is now up and running as well, i figure ill start writing the documents next week and get prepped for a development release, following vegetation handling with in the month after.

Also going to post the 40 x 40 km today; Most systems with a decent processor should fair just fine with my system, i personally can't run it using it unity;

Also going to investigate the different numbers people seem to be reporting. Could be the fps counter, which i admittedly just copied/pasted. I'll see what's going on with a few laptops i have lying around anyways and work with a few testers to see why some are getting high increase while others not so much;

Seems really odd that it's either out right beating unity or matching it almost exactly. Considering the i7s are reporting varying figures, kinda hard to pin it to cpu.. Anyways figure all that out. I'll push out a demo with out instancing as well.. See if i get a difference there on my girl friends lap-trap i posted screens from earlier; 750 m, core i 5;

 

Did some testing, ill update this with a screen from another system; But as i suspected, after a little research fillrates. From the most credible source in the world, Wikipedia,

"When designing graphics intensive applications, one can determine whether the application is fillrate-limited (or shader limited) by seeing if the frame rate increases dramatically when the application runs at a lower resolution or in a smaller window."

This should be evident by the screens i posted along with the demo; Though there may be more to this in relation to drivers, hardware, ect, I think a portion of the problem might be the framerate tracker I used as well. I'll look into this further but I ran it on the crappiest laptop in the house and logged resource usage;

AMD 4 Core A8 with R5 series integrated graphics; Really crappy laptop unplugged;


I'm really not sure what i can do to remedy fill-rate issue; The scene above is once again using the same settings as unity, just batched a little more compactly, and in terms of raw resources used, unity is using double what my system would on both the main thread and the gpu;

I'm going to finish the editor and get more optimization work done regardless; Definitely interested to verify for sure what's going on.

Also posting two demos through today and tomorrow; 40 km 4k terrain and animated heights maps to show off editing speeds;

I'm thinking if i put out the 4k 40km, the extra over head on 58 000 batches; lol will push unity terrain behind regardless of fill rates anyways; Be interesting to see results in that scenario;

 

I believe fill rate is an issue just for this demo purposes, I would not bother too much....

In a real world usage you will have vegetation, game logic and such which will benefit from the extra processor time available with terrafirma.

As I see it when your game has too much to do and you start loosing fps and fluidity due to the processor being overloaded with stuff to do, you will se the benefit of terrafirma, it is hard to show this in an easy way, but I believe this is where its value lies in.

 

Thanks for putting that into better words then I could have; lol

Especially the last point; I'm going to releases a 40 km terrain in a few hours that I think should show how extra processing over head on unity's end can cripple performance;

Been hard at work finishing up things; Did alooot of optimization work;

First of everything is now much cheaper; Here how performance compares of now :

Terrafirma Top Unity Bottom


-I've managed to pull back 700 frames on my end, now running at almost 4x the frame rate;

-Batcher has been optimized; Still needs one last modification which will really chip down processing times again;

- Add Quad Tree frustum support;

Quad Tree Frustum

-Sub-divisions are restricted going left to right;

-This essentially reduces works on the frustum culler; As of now it's not much a of a problem where it stood before, but being able to limit it's operation can reduce processing time on platforms with less resources.

-I also reduced work being done by the shader; The splat map portion is simple as it's going to get on the surface portion, the vertex function has been majorly simplified.

-Stitch map now supports patch numbers beyond what i think current domestic technology could even process; So map size will not be limited;

Also managed to get the new stitch mapper in; Most of the editor is done; Just cleaning up code and will most likely incorporate my grass system last minute; Unity trees aren't bad; Grass on the other hand seems to be taxing, so as a start for vegetation handling, ill use unity trees with my grass, and enable unity vegetation to run independently as before on my terrain as well.

Anyways get animated terrain and 40 km landscape demos ready; I'll be posting them soon.

 

You know that other than a cool demonstration, there is no practical usage of a one piece 40k terrain. You'll experience floating number precision-loss around 15 to 20k unity units (15km to 20km) so all the physics calculations will be 'funky'.

If you want to create a real-world usable piece, I suggest to concentrate either on the streaming or at least zero out the entire terrain (move the player and the neighborhood to the zero point, but it would be very expensive to move the entire terrain...) from time to time. On this size it would be tricky.

Also it is a crucial point if you want to support the big open worlds (like Skyrim for example), to have a lodX version, which is always loaded and shown in the background (when you have a big mountain and you can look down to the terrain and you have another mountain on the other side of the map and there is a gigantic valley between them...) if you can create this secondary, lowres version automatically and remaining at least on the level of the unity terrain in terms of performance, I think your tool is an instant buy without question.

 

I've actually already compensated for this, haha. And it actually doesn't cost mush, aside from some calculations related to bounds handling for frustum culling; I could tweak this so bounds are transformed before the frustum test rather then having the table rebuilt every frame the terrain moves, other then that though the system wouldn't sense a change in the frustum culling or patch distances so the mesh doesn't need to be adjusted, redrawn, or processed in any differently then it would have the frame before the reset; I'll have a few options for large terrains anyways to compensate for resetting positions;

Streaming support is in the works as well. Most of it's there for the most part anyways, in terms of loading and off loading terrain partitions with stitching between enabled terrain objects; Can also load and off load terrain built from saved data with or with out a background pool; Still more to do, but the foundations there. I'll set up a large terrain object now though; I've never heard the term lodX but, i posted a shot prior to optimizing of the same 40 km i was planning to post running between unity and my system; I'm managing 500 fps vs 4 fps with 40000 unit draw distances; In favor of terrafirma as of now, with out much of a visual compensation, lods transition distances could have been set higher; I'll look into that however though;

I also don't want to tell people how to use my tool. If your design calls for streaming or position resets, i should have that covered in a way that's non invasive anyways.

There's a few ways i could reduce overhead further using multiple resolutions for the level map. I'll look into that as well; In that regard i can also lock patches to the lowest level and avoid processing unless with it's a certain radius pretty simply to. I'll try and add in some options in that regard as well.

I'll focus on the animated terrain demo for now anyways in that regard. I'll push the LOD radius up higher as well and post more shots comparing both performance and the visual between massive single terrain objects. Showing where things stand now.

Reducing the level map resolutions is a pretty neat idea idea regardless though as well as locking distance patches to specified levels; I'll look into this lodX as well, lol. Thanks for the input.

 

Because I've just made up. I just wanted to express the "lod way higher than 0" state.
So you know, you have a lod0 terrain upclose, lod1 for in the middle distance, maybe lod2 in the distance and lodX (really low-res) when you are watching the other side of the terrain from the top of the mountain.
I guess you're familiar with the infamous Skyrim pictures: http://media.moddb.com/images/members/1/545/544217/Skyrim4.jpg You know, in the background the villages and the mountain on the other side of the map are lodX.

It's good to hear you're already working on the things I just mentioned. I'll keep an eye on your project, it is very promising.

 

Tested the demo on Win10, i5-4590, 8gb ram and Nvidia 1070.

At various points in the fly-around of the map I was seeing a 600-1000% increase over the standard terrain FPS.

 

It's more likely that that distant geometry is some sort of billboard imposter. A kind of screengrab of the geometry.

 

Ahhh okay, my bad, lol!

Well first off based on our correspondence, I just added in support for user specified levels of detail, so fine tuning the resolution between the highest and lowest level is now configurable and you can have as many levels as you need between them. Transition distance are configurable as well;

I also have a light weight transition method that just checks against the distance, so I'm going to make this layer-able with the gradient method, and allow for patch sizes to sub-divide as the mesh get's courser. Also going add in quad tree based batching/subdividing; Essentially the same method unity uses, i'm going to attempt to make that layer-able as well, so each leaf node would be treated as sub-terrain object, that would be sub-divided itself and proccessed like full, independent terrain objects.

Thanks for the suggestions though; I typically go rouge on this type of project; Essentially it's easy for one person to overlook things. Kinda figured the mesher was done, but I can see there's more potential work in the future;

I think there are impostors, but the mesh quality does decrease dramatically the further a patch of the terrain is from the camera.

Update :

First off i'll mention what's left in regards to first submission to the asset store:

-Optimize Batcher
-Optimize Culling
-Optimize Level Post Processing
-Rest of the interface

-Grass (Maybe depending on time)

I figure i can have the first two done in a day or two. Just been putting it off as they don't work bad, they could just work better. I figure a day or a few hours with the level post processing, again it's not bad either; These things are the only items really registering on the profiler as of now though; With that work complete the system should be occupy much of your cpus time;

Future Meshing Features(Post Release)
-Various Methods To further compact processing; Quad Trees, different patch dimensions across levels, ect;

Demo

Going to post a small demo later showing off mesh topology adjustments in realtime; I have three test options that are all updating every vertex across the terrain every mesh update.

First test is using a three image sprite sheet the system switches through and injects into the terrain.



This one runs without adding any real noticeable overhead in terms of processing.

The next is using mathf.perlinnoise to update the entire map every frame; I can manage 60 fps as I'm brute forcing my texture writes;

You can control the scale of the noise, as well as axis movement speed; With more optimal methods for updating textures, real time erosion should be more then achievable; Real time desert dune formations, ect;



Ultimately the time it takes to do this completely dependent on the number of pixels your attempting to update in a texture2d; Passing these changes and having the mesh reflect them costs essentially nothing however, just call setpixels, then apply and point the height map you generate to the height map.

With that said I'm building one last scenario, where i'll move a mound around the terrain plane in real-time via the cpu. (How you modify the the height-map texture2d is up to you.)

I'll finish this up anyways in the morning and post it; Just needs the last method and an interface.

 

Would it be possible to do a Mac demo build also?

 

I should be able to less that's not possible on a PC, I'm honestly ignorant to the entire Mac OS / iOS scene. But when I release more demos, i'll make build's for as many platforms as possible for now. Sorry Apple people for my complacency.

Update

I honestly needed a good 2 days to catch up on sleep. I'm back to work though and have some what of major update in terms of the systems performance. My quad tree was subdividing under the worse circumstance due to not be able to test for bound frustum intersections easily, using unity api. Now though, it's finished and it's running great and as it was intended to.. Soo happy.. :')

I managed to get 4096 bounds test down to 149, 3.0% the test count, vs the my older brute force attempt; Which is great because I don't have the luxury of using unity's culling systems for a number of reasons. (Batching, displacement, mesh proportions.. ect.. Batching being the big one.).

Frustum Culler Test Reductions :


Testing patch boundaries against the frustum was really the only over head present when idling. On this 4k map, it was taking a total of 4096 (64 x 64 patches) bounds tests, it's now taking 149 bounds tests to achieve the same result, as previously mentioned;

This was honestly the most pain in the... brain.

The quad tree was trivial. I needed to test bound intersections, which isn't possible with testplanesAABB as i mentioned above and had to do this optimally as well.

Took my a couple days of banging my head till i realized I got the plane normal direction backwards. Anyways frustum culler is done;

Only thing left with the Mesh handling system now is :
-Batching system optimizing;
-Patch Level post-possessing optimizations;

The editing interface for the most part is done as well.

I have a lot of options to build into the interface. I figure that shouldn't be to much work though.

For example, i figure there will be at least two options for culling, with the quad tree based with options for max subdivisions counts for example. Anyone using the system will have full control over the batcher as well.

Also planning to get it running on "newer" hardware on the GPU. So you'll be able pick and choose ultimately where the you want your processing burden to go, across majority of the system, even the batcher at some point, geomorphing is future feature as well on hardware that supports tessellation.

I'll post the height map animation stuff soon. It's honestly been overwhelming to say the least finalizing everything and getting demos ready at the same time. Next post though, i'll come bearing gifts.

 

Seems like it's almost ready!
Would you consider putting it on the Asset Store as it is now, and then do optimizations in an update? I'm sure everyone following this thread is eager to try it

 

I actually feel it's ready now.I just need to give you guys an interface from where things stand at this point.

Update-O-Rino..

Finished the batcher; and ohh boy does it fly.. As of now optimization layers are complete until after release.

I'll keep this one brief. ;p

I'll be focusing all my time from here on out till release on finishing the docs, interface, getting demos and videos/tutorials ready.

Essentially were at the final stretch. I can honestly say with confidence, i won't have any regrets releasing this version. Things got a lot more technical than i had initially anticipated due to some concerns. The "hard part" is officially over now though;

Me and my girl friend are going to spend the weekend getting presentation material and the assets homepage finished off as well. I'll be detailing/highlighting release features and differentiating planned features vs first release features there. Things i've mentioned like geomorphing, will be a future features for example, though it would probably take a few hours to integrate, at this point im not going to take the chance that hours turn into days, lol.

I'l also be getting demo builds out to people as well to ensure the mesher is stable. Just need to really crack down on the interface.

Anyways I need sleep, i'll show off the batcher's performance tomorrow.

Should also mention i've added in a coarseness/fineness control system to the lod transitions. Essentially testing will "subdivide" the closer patches being tested are to the camera.

Though it's optional.

 

Updated the main post a bit; I've went over everything and fleshed out the features the system supports as of now and that will be included in the alpha release;

This is the same List I posted on the main post;

Full List Of Release Features

Mesh Generation parameters :

-Source height Map (Optional)

-Unity Sourced (Optional)
-Max Height Map Dimension; Up to 8k, Though Not warranted. Up to the max size hardware/Unity will Support;
-Height Map Dimensions Limited to Square images (Any Format), that are evenly divisible by a power of 2;
-Option to double, triple, ect; the vertices count densities, through height map interpolation; (Use a 1024 x 1024 vertex mesh to represent a 512 x 512 height map); ​

-Options for Patch Dimension / Height Map Divisions (How Many Chunks the Terrain Consists of
-Options for terrain Scale / Size (Identical to Unity)

-Option To Pre-Generate Normal Maps: (Bake Normals; )

-Adjacent Neighbor Check
-Diagonal Neighbor Check
-All Neighbor Check

-Good For Static Terrain, Low Powered/ Older hardware, No noticible differnce on modern hardware. Tested using a cheapy, 950 gtx. Just an option if you really need to push things, in scenarios I can't fathon, lol;
​

-Option to handle normal on the GPU :

-Adjacent Neighbor Check
-Diagonal Neighbor Check
-All Neighbor Check

-Must for real time modifiable terrain (Not a must but should be!! ;P ), No noticeable difference between baked and non on modern hardware. ​

-Options For Triangle Winding Options:

-Unity Winding (Same Formation as Unity)
-Opposite​

-Three Material Options With Optional Components (Possibly More Before Release)

-Standard
-Specular
-Diffuse

-Normal Maps Optional;
-Triplaner Optional
-Planned** Texture Array Methods For Displacement and Traditional Texture map handling;​

-Editor

-Height and splat map editor designed to mimic the unity terrain editor interface;
-Custom Brush Support
​

-Instancing

-Support Instanced and non-instanced based meshing. Though the batcher​

Custom Mesh Drawing/Updating Method : (In Order Of Operation)

1.) Initial Tree Handling :

1A) -Fill Frustum Tree (Optional Should Use When Using the Batcher)

-Option for Minimum Node Size​

1B) -Fill Distance Tree (Optional)

-Option for Minimum Node Size​

2 ) Step 1

2A.) -March Along The Distance Tree, Building Partitions; (1B Enabled)

-Partition Size is based on distance;
-Options for patch/sub-partition sizes in partitions;
-Check if sub partition is frustum. (1A Enabled)
​

2B.) -March Along Each Terrain Patch; No Compression/Partitioning (1B Disabled)

-Check if sub partition is frustum. (1A Enabled)​

2 ) Step 2

-Update Patch Levels based on Partition Settings;

-Option for Max Levels of Details

-LOD Options

-Gradient

-Option for Accuracy Distance (How far the effect blends out from the camera)
-Option for square distance check (Fast Inaccurate, no sqrt)
-Option for circular distance check (Slower Most Accurate (Mathf.Sqrt()**))​

-Radial Circular

-Options for defining max level Render distance;​

-Radial Square

-Options for defining max level Render distance;​

3.) -If Level Map Has been Modified;

-Batch Level map into instances;

-Option For Max Batch Sizes for Each Level;​

4.) -Draw Batches / Update Filter Stack

-Via Native Rendering Plugin
-Via Shared Mesh (Fall Back)​

5.) -Update stitching

The distance tree is something I recently added; I'll flush that one out more later. Essentially it partitions the map based on distance from the camera and the system updates the mesh per partition rather then per patch. With this we can effectively control the accuracy of testing as mesh progress further from the camera rather then brute force testing against each patch. Overall it reduces bother transition and idle processing so it add no over head; I was able to reduce patch level testing in a quick test down from 480 to 114 distance test/patch adjustments. I'll have more on that later anyways.

Draw Calls Have Again been further reduced with this setting and terrain transition time has been halved with the current setting i'm using;

Terrafirma Top Unity Bottom

This terrain as far as patch transition sizes go, are identical on both ends once again.

Never really showed memory consumption either; Below is the stats when taking the screen shot above.


So with that said. I'm going to get to work on interfacing all of this and getting dev builds out. From there I plan to work on the documents and make tutorials and presentation materials while working with testers to iron out any over seen issues. Finally comes release after this.

 

This is looking excellent Murpenstien, about time someone tackled Unitys 'Elephant in the room' of terrain performance.

 

Well, everything is in place as of now. A good chunk of the interfacing is done. I'll be sending out dev builds in the next few days.

Should also mention that I wasn't fair to unity. I realized my 4k test was wrong on Unity's end. I set the pixel error to zero I guess by mistake which was rending the entire mesh full detail; Below is a proper comparison :

Terrafirma Left Unity 3D Right

Here's a 4k height map scaled to 40000 x 40000 ms' (40 km x 40 km). I was able to double my speed from previously with the distance tree I came up with. So it now runs roughly 4 times as fast as unity. I haven't personally seen a solution documented that works exactly like this method so I'm excited personally to get dev builds out there and see what short comings if any exist in contrast to unity. Personally though for big static maps (non-infinite) like this I'm finding it's more optimal in-terms of the long range visual to give a single TerraFirma instance as much data as possible and let the system break it down for efficient handling vs using multiple terrain instances;

Anyways optimization work is definitely done for now. Figure it'll be two weeks, three tops and the system will be submitted for publishing on the asset store. Interfacing and documenting is already well under way. If I've contacted you in regards to testing as well, i'll be getting pre-release builds to everyone as soon as possible. From there until the first stable build with vegetation handling at the very least I'll be making regularly updates available as well ahead of what will be available through the asset store, to those people as well. Closer to release also I'll most likely be issuing more dev copies once the system is sent out for publishing review as well.

EDIT

Was messing around and managed to take another major load off the cpu. Ignorance on my part making assumptions about unity; Since the renders were never being turned off, as I was switching unused renderer mesh filters meshes to null; Unity was doing processing on essentially empty objects (I know I'm stupid). In the circumstance above I'm actually getting much better results, my concerns before was these calls (renderer disable/enable) would stack but the difference is something I can honestly live with as the batcher and the filter stack already ensures that there shouldn't be that many of these calls to make a difference, and with the frustum culler set up right, I'm honestly in tears. This has been hands down one of the most fun projects I've worked on. With that said I'm glad the end is near for the primarily stuff. I'll leave an option to have the system not touch the renders as before, it doesn't seem to make much of a difference on smaller maps and you'll get speed ups on level transitions. For much larger maps though, id say its a must as I've come to learn 260 meshes out (65536) 256 * 256 is a little much to loop through even if the instances "empty".

 

Here's something to go with above mentioned edit. Here's the system running the same scenario now :

4k (4096 x 4096 pixel) height map scaled to 40 x 40 km's;

Terrafirma Left Unity Right

Terrafirma Top Unity Bottom


At this point I have no idea what more can be done to make this go any faster... (I actually do.. but i'll hold off for a bit, lol)

I need to finish of the distance function still, so i know when that's done I can get tri counts down to what unity produces. Nothing big in terms of work anyways, just have a habit of leaving lesser tasks till later, in this case it's moving code and a lil math. (Multiply two numbers lol...)

I have however pushed the system down to produce the same amount of tris just distributed differently and I was managing 2200 fps. Can't wait to show that off.

Anyways looks like i'll be re-benching on/off while i finish up. Just got excited and thought I would share.

 

Everything sounds fantastic. I'd say this asset along with Mega Splat are among the best assets to come along for Unity. And much needed and long overdue. Thanks for the awesome work. I look forward to trying this out soon.

 

Honestly I love working on things like this almost obsessively. Thanks for the comment though, kind words such as yours make the whole venture all the more worth while.

As always, thank you.

Update

Okay, performance is much better, and scales pretty good.

All logic and scripts have been cleaned and commented for the most part; Majority of the interfacing is finished as well. I added in more optimizations to. Things couldn't be better. Finalizing stuff is going much quicker then I expected.

Terrafirma Top Unity Bottom

1028 x 1028 height map

Terrafirma Top Unity Bottom

Height Map Sourced Working with Lennart;

Everything is coming out great. All the scripts are cleaned up. A few functions need to be added but majority of the interfacing is done. I figure a couple more days and I'll be getting dev builds out and starting the docs.

There's also more options than I listed previously. Completely over looked a few of them. I'll update everything soon.

Also throwing in traditional quad tree CLOD, GPU tessellation based LOD transitions and option to offload all the mesh handling aside from the actual updating to a separate thread. I'll leave each one of those till im working on the documents. They are may not make it in before first release. But at the very least will be complete soon after.

I figured I'd show a visual of the distance tree method I wrote up;



There's a lot going on here. So first off, i set the max patch size twice as high as the default to make things are easier to see. Essentially the tree breaks the map up into nodes / jobs with different accuracy tolerances based on distance. Nodes closer to the camera will process raw/uncompressed. Nodes further from the camera will start to degrade accuracy. Essentially the point of this is to ensure the system is wasting time on portions of the mesh where a single pixel could contain a few to tens of triangles. This map is 40 km by 40 km, so 4 kms out the mesh complexity doesn't require as much priority as there isn't enough pixels to draw the detail at that distance.

The whole thing is configurable as well.

Should also mention majority of the settings can be changed run time now across the whole system, so you can visually see how different settings will effect the visual / performance in real time.

I can't help but feel im forgetting something but, this good enough for now. The end is soo close now....

 

So...I'm curious now, given the excellent performance you are getting...
Let's say someone wants a huge "world". I'm just going to throw out 8x8 world made up of 4k maps.

How I would handle this traditionally is break them into say...512x512 res patches.
I'd load up the ones around the player, the ones very close to the player would be shown in high detail, the ones further out would be shown in low detail. However, one challenge is the edges between the low detail patches and high detail ones. But this is stuff you already know I'm sure. I'll get to the point:

How would you handle this using Terrafirma? If you use the "break into tiles" approach, how well will Terrafirma automatically handle transitions between separate terrains, shown at separate levels of detail?

I'd even be curious as to how well Terrafirma would be able to handle loading up the entire world. NOT THAT that is a valid use case, but given the excellent performance you've shown...it would be sort of a stress test.

Thanks for you enthusiasm on this, it's catching!

 

Hey great question;

Like unity, you can declare the neighboring terrains around your terrain; I'll add in an interface for designating neighboring terrain objects, visually. Though the api is there so this can be set up via script as well. Stitching works as expected between terrain objects along with the culler. The distance tree wouldn't do much when running 512 x 512 height maps so it would be best to leave that off. (Map is already "partitioned" in that case. ) The batcher will run over each terrain instance, so that's about the only negative, unity's quad tree system actually acts the same however. A "leaf" will never be created along the seam of two terrain instances with unity terrains as it breaks the mesh up independent of the height maps around it.

My system can be/will be extended to circumvent for this, just need to get around to it as it's not trivial, each instance will need to sample neighboring terrain splats, height maps, ect so it can sub-divide across terrain data sets, I should have heights partially done for normal calculations, but to the degree it would need to be to support the batching bits, it'll have to wait till I have a bit more free time, lol. I'll fool around tonight running multiple terrain instances and show performance comparisons.

Loading speeds and generation times are more then fast enough to handle 512 x 512 segments with ease. The only real processing revolves around calculating patch deltas for each level. These are usually created as the terrain is modified, in the case of generating a new terrain instances however from a height map, or when a height map is injected, this process needs to build the entire minimum render distance field at once. (When using the patch gradient LOD transitions settings. Radial methods don't rely on this field.) It's not a slow process by any means, but you'll notice skips loading in 4k height maps from scratch.

I'll try and post some stuff running multiple terrain instances either later today or tomorrow. I have a lot planned for handling more then one terrain object, for loading/offloading data. Just want to ensure I take the right approach and think things through rather then just whipping something together that "works".

Sorry for the "depth" as well. It's hard to outline what's possible or what approaches you could take in regards to this as there's a lot of layers to the system in regards to configuration. I'm finalizing everything as we speak now so I'll make the documents available as soon as possible.

Minor Update
Majority of the interface is interfaced. (Materials and communicating changes aren't finalized) I'll post more on this when I wake up, showing off the interface as it stands now. Everything came together much faster then I thought though. Really just want to scrutinize the heightmap and splat editor and go over everything again. I also need to modify the system visually. But for the most part I'm wrapping this "dev cycle" up very soon, lol. Big things left to do now are documents along with finalizing api features and labeling, minor bug fixes that came with interfacing and that's pretty much it as well as few minor additions I left that shouldn't take any longer then an afternoon to implement.

I figure once I get part ways into documenting the system and I'm sure nothing needs final tweaking/modifying, i'll begin focusing on posting videos and demos as well. Figure give 3-4 days and I'll be finished every above aside from docs and api.