Jove 2.0 - Replacing the rendering pipeline

Few questions:

1. I take it adapting shaders to the new lighting system is a matter of changing inputs and there is no additional code that uses up limited resources (like texture sampling)? I have some heavily packed shaders (specifically, triplanar ones, which is always a problem to fit into DX limits with decent enough feature set) that use up almost all resources allowed, so I would like to know if I would be able to port them to Jove without cutting features.

2. How would Jove handle use cases with a very large environments? For example, FPS games requiring depth range bottom threshold set low but using 3D backgrounds in 10-50km range (for parallax effects during fast travel or animated elements). Simple example:



Traditionally environments like that are rendered with two cameras slaved to one transform to solve the issue of limited depth buffer precision (with one camera rendering e.g. 0.1m-500m and another rendering 500m-20000m range; or by decoupling background geometry and scaling it down to render it with a second camera using the same depth range, but that's very inconvenient if you aren't making UT-style backgrounds completely separated from the walkable geometry). But as far as I understand from your posts, Jove is mostly tailored for use with one camera only. Would I be able to set up multi-camera setup described above with Jove and if I will be, are there any potential issues with using that idea I should know about?

 

As always, your words are appreciated and encouraging

What I mean to say is, tech demos that look like they are the next big thing usually are not. For the simple reason that it is way easier to make a technique work in an isolated case, or with very specific constraints or whatever. I do not think one could make a game with that technique in this day and age, the hardware is not there yet. We will get there, but not this generation.

1. If you want to do custom deferred shaders (say animated UVs or whatever you want to do) and feed the data into the main Jove shading function, that's pretty easy. I currently only have a single deferred shader, but adding your own would be trivial. Here's the fragment shader of the said deferred shader

Code (CSharp):

1. GBufferProperties frag(v2fStandard i)
2. {      
3.     float4 diffuseTex = tex2D(_MainTex, i.tex.xy);
4.     float4 attributeTex = tex2D(_AttributeTex, i.tex.xy);
5.     float4 normalTex = tex2D(_BumpMap, i.tex.xy);
6.    
7.     float3 textureNormal;
8.     textureNormal.x = normalTex.w * 2.0f - 1.0f;
9.     textureNormal.y = normalTex.y * -2.0f + 1.0f;
10.     textureNormal.z = sqrt(1.0f - textureNormal.x*textureNormal.x - textureNormal.y*textureNormal.y);
11.    
12.     GBufferProperties gBufferProp;
13.  
14.     gBufferProp.GBuffTex1.xyz      = diffuseTex.xyz * _Color.xyz; //Color
15.     gBufferProp.GBuffTex1.w        = 0.0f; //Currently unused, fancy things planned!
16.     gBufferProp.GBuffTex2.x        = attributeTex.x; //Metallic
17.     gBufferProp.GBuffTex2.y        = attributeTex.y; //Occlusion
18.     gBufferProp.GBuffTex2.z        = attributeTex.z * _TranslucencyMult; //Translucency          
19.     gBufferProp.GBuffTex2.w        = attributeTex.w * _Color.w; //Smoothness
20.     gBufferProp.GBuffTex3          = EncodeTangentNormals(textureNormal, i.tangentDir, i.binormalDir, i.normalDir); //Normal
21.     gBufferProp.GBuffTex4.x        = EncodeDepth(i.clipPos.w); //Depth
22.    
23.     return gBufferProp;    
24. }

Basically, feed it whatever you want just make sure things end up in the correct channels Note to self, make a more user friendly struct...
Also, as for porting, you can use forward mode in Jove. The downside to forward objects is that if you want them to contribute to AO they need to be rendered twice (on the plus side though is that if they are rendered twice that pixel won't be shaded at all in the deferred pass, saving performance).

2. As far as rendering of meshes go, I'm still using everything Unity does on the CPU side. I mean, I hit Camera.Render() and then there's a completely custom shader pipeline, but the rendering calls and the managing of the vertex buffers and everything else is still Unity. So everything relating to the actual rendering (not shading) has the same limits and possibilities as Unity.

And as for multiple cameras, that is totally possible in Jove. I just added support for reusing shadow maps between multiple cameras as well. This is not two images composed to one in Photoshop, this is done using two cameras rendering to their half of the screen. The bottom camera is reusing the top cameras shadow maps, meaning shadows only has to be rendered once. Please note that this obviously means the second camera cant move too far from the first or look in a too different direction since, well, shadows are fit to the frustum to avoid wasted performance.



Also, I do not know how clear I have been with this or if the information has gotten lost in the thread, but what I am to release soon is not a finished version. It is an alpha version of Jove 2.0, so I just want to warn any potential customers that if you do jump on the Jove train this early there are bound to be bugs and it is likely that components will change before beta. That said, I'm stoked about getting this out to the public and receiving feedback

 

Do regular old Unity image effects still work? For example, could I slap on some Amplify Motion Blur? Also, is it possible to have overlay cameras set to deferred as well? (that is, in Unity's deferred pipeline I can't seem to have my weapon camera set to deferred without having things go completely broken, is it different in Jove 2?).

 

Yes and no. Those that rely on the depth buffer will not work, as Jove uses its own depth buffer and not what is called "_CameraDepthTexture". If a post process does not use the depth buffer, then theres no problem. If it does use it, the sampling of depth needs to be converted to match Jove instead of Unity. I have no idea if Amplify Motion Blur would work out of the box. I guess it could, but I foresee there might be an issue with how they cache the projection matrix, since Jove does its own thing and does not use Unitys projection/view matrices.

I'm not sure what you mean by overlay? If you mean a camera that render on top of the main camera (like a minimap might do) then yes, both can be deferred.

Jove does not differentiate between forward and deferred cameras. Everything is a camera, and you get two layer masks instead of one. They are called "deferred mask" and "forward mask", and how they work is pretty self explanatory
Basically you put all your forward shaded objects in any layer thats included in the forward mask and then the same goes for deferred shaders in the deferred mask.

 

So when can we see it being uploaded to asset store waiting for approval?
Better tell me at once

 

Sorry to bother you with all this Temporal effects talk but all the techniques are so cool to me. Do you plan on implementing something like TXAA

It's the only AA solution that I've seen get rid of the crawling effect in games.

 

You were very clear about this, but while I agree that the coming version is unlikely to be fit for a public release of a game (having incomplete feature set, potential issues and all), I would guess it will still be very useful as it will provide a foundation you can start porting your content to.

Speaking of which, can you detail the texture setup you have chosen to use? I mean, channel order, amount of separate maps per material, etc.

 

A standard Jove shader takes 3 textures:
MainTex.RGB = main color
MainTex.A = unused
AttributeTex.R = metallic
AttributeTex.G = ambient occlusion
AttributeTex.B = translucency
AttributeTex.A = smoothness
And a standard normal map.

The transparent Jove shader has a transparency map in MainTex.A

There are two emissive shaders as well. One uses a fourth texture that stores the emissive colors in the RGB channels and the other one uses the alpha channel of MainTex multiplied with the RGB channels of MainTex

The Jove glass shader works like this:
MainTex.RGB = main color
MainTex.A = transparency
AttributeTex.R = distortion
AttributeTex.G = ambient occlusion
AttributeTex.B = energy absorbation (how much light that passes through is absorbed. White means all energy passes through to your eye, black absorbs all light behind the glass)
AttributeTex.A = smoothness
And a standard normal map.

 

TXAA is a whole science in itself I don't think you can even integrate it into an engine/game without direct help from Nvidia. I'm gonna go temporal, but TXAA uses MSAA as well which I am not going to do due to technical constraints.

I'm very close to being able to upload. It is being uploaded when it's done

 

Well that's a relief to hear, to be honest! I've had experiences with some PBS setups that used bloated texture sets (separate RGBA occlusion maps, etc.) and I'm glad to see your setup is using just one neatly packed map.

 

Glad you like it


I ported all default Unity particle shaders to Jove and added 3 new versions. "Alpha Blended Premultiply", "Alpha Blended" and "Additive" also come in lit versions. It's not much but it should last until I can get to work on my own particle system

The purple light is a spotlight pointing down and the green light is a point light. It is also lit by the sun and the sky. It also probably goes without saying, but the top left half of the image is the standard additive shader

 

I'd like to say that while I've been avoiding DX11-only Unity products, I do appreciate that you're taking a wide-ranging, comprehensively DX11 approach with this tool. It looks like it will pay off with a top notch product!

I think your timing is excellent as well.

With Jove 2.0, I'm looking at DX11 as finally being a sensible option for Unity game graphics.

I'll be keeping an eye on this, and I wish you much success!

 

Sorry to go slightly off topic here, but anybody know of tools (or just a decent workflow) for putting various texture maps into different color and alpha channels? What I've done so far is just manually copy and paste into the appropriate channels using Photoshop. The mention of the Jove attribute texture above got me thinking that there's probably a better way, or there could/should be.

 

Yeah there's not much point to limiting yourself to DX11 just for one feature Thanks for you support!

I've had that issue myself and I'll likely make a helper script at some point. But you're right, somebody must have done that before


Added shadow mapping support for particles, both casting and receiving. With the limitation that transparent objects can not receive shadows from other transparent objects (but they can from opaque objects).

 

Beautiful

 

Mmm, transparent shadows....

Was originally wanting to support Mac in my game, but that is just so cool. May not want to anymore. PC only is starting to sound really good.

 

He said he would implement a stripped down version for DX9 which will be compatible with Mac

 

Actually, I've got a question as a follow-up to your response about Amplify Motion.

You said you don't use Unity's projection/view matrices, right? Have you considered adding built-in support for vector based motion blur?

 

Yep. Its not high up on the priority list though, but we will get there.

I have indeed and it is not far away The three larger features I'm going to do next are
1. Volumetric scattering (fog and stuff)
2. SSR
3. Virtual Camera

I might change SSR and Virtual Camera. Virtual Camera basically means mimicking a real camera with effects such as DoF, motion blur, vignetting, chromatic aberration etc.

 

Since you do localized cubemaps as well it would be cool to see SSR like killzone does it where they fallback to that if the reflected point is behind geometry of off screen.

 

It just occured to me, and this might be down the line a bit and maybe user contribution can do a decent job easily, but automatic support for vr HMDs, OR being only one of those coming, would be lovely, pretty much all of my actual 'real game' based plans have vr in mind be they mobile or desktop so OR support would definitely steer me towards committing to Jove if it turns out rather lovely. If you question the importance or chances of VR's success, go and have a go then come back and hear me rant! This only occured to me cause of your mentioning the virtual camera, and i wondered how that would work in vr, would it be relevant or would it actually be quite cool to have classic camera phenomena with depth perception

 

nice!

 

You know, I've been thinking.

I don't think I've ever seen just general transparent shadows in any AAA game yet. If this isn't some next-gen stuff, I don't know what is.

 

Those were implemented both in Frostbite 2 and CryENGINE3, I think. Not that it makes the implementation in Jove any less impressive.

 

According to this

http://docs.cryengine.com/display/SDKDOC4/Shading

CryEngine shadows are not affected by transparency. Can't seem to find any info on Frostbite.

 

Ah, yep, CryENGINE seems to only support transparent particle shadows as a special case, not shadow casting for generic transparent materials.

 

Ah yeah, I'm willing to bet they're using traditional projector-style shadows for those...

 

That particle shadow casting is going to be so helpful for me.
My smoke particles I made look great in a well lit debug area, although in a game environment they don't blend in that well (shadows not reacting with it, smoke not casting shadows itself etc)

Very very happy

 

Thats the general idea We'll see about the specifics when I get to it though.

Jove 2.0 isn't gonna support VR to begin with, mostly since I have no way to test it and developing without being able to test is a nightmare. Eventually though...

 

Is there any reason to believe it wouldn't just work though? Does Jove 2.0 use some kind of special cameras or something? My experience with Oculus Rift is that it'll work with most "normal things" (geometry that can be seen by perspective cameras).

 

To be honest with you I am not sure. Jove 2.0 does not use Unitys camera class, so if Unity does something special for VR (which I assume it does, I'm not very familiar with how it works) chances are it isn't gonna work. That said, I'm sure it is not that much work to add, and I will add it as soon as I know what I actually have to add

 

VR in Unity is at its most simple, 2 cameras each rendering to their half of the screen, but then it gets to altering the cameras rotation for covergence, their spacing for interpupilary distance, an adjustable barrel distortion post process and so on. Effects that are dependant on the view of the camera can break with the OR, with only one camera rendering properly, (reflections being a typical one) so there are some considerations, but i could understand all that coming later

 

To optimise performances, i think we can do :
- Render the split screen setup of the two cameras, without post process,
- Do post process on the screen, on the same way as if it was 1 screen,
- Do barrel distorsion
So it would be compatible with vr, without the cost of rendering pp effects twice.
Or maybe something like depth buffer based 3d reconstruction can have good performances.

@lazygunn Is it worth to buy oculus rift dk2 or it's better to wait for consumer version ?

 

The distortion is controlled on the Oculus plugin side and must be performed in a very specific and precise manner, so I don't think what you're proposing (imitating standard OR setup output) will work.

 

Aieth,just wanted to let you know that Bart wronski(gfx programmer on Assassins creed 4 and far cry 4) has released his SIGGRAPH paper on the volumetric fog he used in assassins creed that's based on compute shaders. The paper is here: http://bartwronski.com/publications/

 

Wow that looks really nice, makes me want to go out and get AC4 on my PS4

 

Thanks, had not seen the siggraph version I was already planning on the improvements he did to it, good to know they work in practice

 

If you intend to learn the technology and develop for it (heavily advised, imo), then the second devkit seems a good buy (havent got mine yet, labouring with the 1st devkit). If you're familiar with computers on a technical level then you'l make great use of it. If you just intend to play the games, you could wait, but either way i advise anyone to grab a google cardboard kit as its very cheap and test it with a friends phone if you dont own a suitable phone so you know what the effect is and what it means (Generaly the effect on you is profound and you'll want in)

Regarding the barrel distortion effect, i have a DIY shader somewhere for it, in fact a DIY VR kit in general that i should be able to find, the barrel distortion isnt really all that complex although needs to be controllable i think as a post process for each eye, like most settings to account for differences between eyes (being able to compensate for astigmatism and wotnot)

 

Any glinting new updates that are so shiny and polished that it blinds your eyes like a well scrubbed toilet with bleach?

 

Lol so descriptive!

 

I'm working on finalizing stuff, refactoring, tooltips, documentation etc. I have a few screenshots I can share though.

Here is a picture of 11 translucent cubes, ranging from something like 0.05 translucency to 1 translucency

Translucenct materials​

And here are two more screenshots of particles. The first screenshot has the sun turned off, you can clearly see how the smoke receives a blue tint from the sky light. Teh second screenshot has the sun on, and as you can see is lit by it. Lit particles have an additional parameter called translucency, which works like the translucency in the first screenshot.

Only skylight, no sun


Skylight + sun
​

 

[RELEASED] trueSKY Alpha for Unity Pro - Volumetric Skies, Clouds, and Time-of-Day

Well this has just been released to the public for only $50... man I'd love to get it but it's probably a no-go with Jove 2 right?

 

@Licarell To get it to work with Jove 2 I think it would need some modifications. (Mainly shader side). As the depth buffer is not unity's. Also the lights are replaced in Jove 2.
But it shouldn't be to much of a job to do so I think.
We'll have to wait for Aieth and see what he say's as he's the developer.
For the most part it's not a problem modifying shaders to fit with Jove's way of lighting/checking depth.

 

I think you should wait as Aieth has already implemented half of that(the atmospheric scattering sky) and volumetric clouds are on his to-do list I believe.

 

Oh I agree but... When you have a plugin that is specialized and a award winning plugin that is used in AAA games already... couldn't there be a little wiggle room, just a bit of collaboration between developers...

 

I do not know how it works, but based on the short description it probably needs a little massaging. Since I do not use Unitys light class it would need to be converted to use a JoveLight instead. I guess it is done as a post process effect using a full screen quad, which would work fine rendered in Joves forward mode, you just need to be able to set the layer for it. If those two conditions are fulfilled, it should work no problem.

 

Small heads up for those of you following this thread. I am by now pretty sure that Jove 2.0 is going to be ready for an alpha release before this weekend is over Barring any major last minute issues.

 

Can't wait to check it out! : )

 

Sweet, I've been dying to give this a try...

 

Throughout the development of Jove 2.0 I have been asked many times what the point is. With Unity 5 coming, there already being free physically based shading etc. The thing is, there is so much more to rendering than just a shading model. I mean, if just using physically based shading meant your game was next gen there would not even be a point to Jove 2.0, as the first version already does that! In order to help illuminate what I am talking about, I dug up some old marketing material for the first version of Jove. Below are two pictures of the same model, one rendered in Jove 1.0 and the other in Jove 2.0. As I'm sure you can tell, there's quite a difference between the two

Jove 1.0


Jove 2.0​

This first release of Jove 2.0 is only the beginning. I have so much more planned and so many things I want to do with it.

(what's even funnier with that image comparison is that the first one uses baked lightmaps while the second is completely dynamic and it still blows the first one out of the water)