DOF on IOS, How we did it (mini post mortem)

Depth of field and Bloom on Unity IOS ( Run Fatty Run mini post Mort-em)

Recently I've had the chance to venture into IOS. In many ways its like the year 2000 all over again in terms of some hardware specs, but in many others its defiantly 2012 with HD resolutions and massive amounts of memory. With a few wins and losses, I thought I'd take the time to share some of the things we have uncovered, firstly as an overview, then in later posts with detailed descriptions of each stage.

​

A friend of mine at Disparity Games, Jason Stark asked me if I could get DOF and Bloom happening on their latest project Run Fatty Run. After looking at the project and its frame rate at the time, it looked like a real possibility and to cut a long story short, we did get it working on ARM 7 devices Iphone4, IPod 4, IPad2 and later. Before I describe how, let's cover some basics first.

What are the general steps needed to do a standard DOF?

1) Create a depth buffer.
2) Write the scene to that depth buffer frame buffer.
3) Read from that depth buffer frame buffer writing to X number of down sampled textures
4) Blit the final result to the frame buffer based on depth and DOF parameters.

Sounds pretty straight forward right, so why haven’t we seen more games with DOF implemented on IOS?

Step three is not possible because we have no read access to the depth buffer using Unity IOS 3 ( It will be available in Unity 4, Fantastic!) . Step three seems broken and doubly so when you discover a single grab of the frame buffer on an Ipod4 in HD, dropped the frame rate to near 20fps with only a single cube on screen.

Not interested in the cant dos? Well neither am I and this is how we addressed the main bottle necks as an overview.

1) We created a camera render texture at full screen resolution and used that instead of the frame buffer.
2) We wrote the depth as a grey scale value to the alpha channel of the render texture and used the RGB channels as per normal.
3) By reading from the render texture we avoided reading from the frame buffer and were able to write to various smaller textures, down sampling each time.
4) We then combine the down sampled textures based on depth and blitted the result to the frame buffer. In our case we integrated it into the GUI camera pass, using it to clear the camera to save on fill.

​

This covers the main bottle necks and their work arounds and for those familiar with the shader side of Unity its probably all you will need to start your own implementation. However over the next week I'll be posting detailed information on how we achieved each of the steps and the decisions behind them.

To see the results you can view this video or even better purchase the game here.
If you have any specific questions you can contact me via the details on my website.

You can also find a temporarily available unity package with a very simple demo scene here

Kind regards,
Brn

 

This effect looks awesome! Nicely done. I will definitely be watching this thread when you post more details

 

Really nice , please put this technique on the asset store

 

Purchased the game just because you proved it could be done and then posted this. I don't have iOS Pro so I can't test it out yet, but if I get some time I'll try doing it outside of Unity.

 

Teriffic! Thanks for sharing!

 

@ Stramit, Thanks Stramit. I'm really looking forward to Unity 4 with its new functionality. I've edited the first post to reflect the the Info about the depth buffer.

@ opsive , I am da bawss coolpowers , thanks for checking out the thread and the game. I'll be able to do the first detailed posting in the morning.

@ Lars Steenhoff, At this stage I'm not sure if I will put it on the asset store. The system doesn't package well, because it requires custom shaders for every material you want to be effected by DOF and some tampering to be done to the GUI, cameras ect. The best I could perhaps offer is a free sample scene package with the option to donate. That way if its not suitable for your project, no one is put out. Its because of the messy nature of it that I decided to do the mini post mortem. Besides sharing this sort of info is a nice way to give a little back to the Unity community.

Cheers,
Brn

 

This is different considerably to the DOF I did for an old project for polycount. I think that the "custom shader" thing is intriguing, is there a reason it has to be like that?

 

Hi Hippo,

It was the best way I could think of to write the depth data into the camera render texture alpha channel. It produces a little extra overhead on the vertex shader but next to nothing on the fragment shader. I guess the other method I could have used was a replacement shader technique, but that would require extra passes. Also by cramming the depth data into the alpha channel, I could read/write depth and color values with a single a texture fetch or fixed4.

The other bonus was that I could write the desired blur factor into the alpha channel instead of the depth, which is the method I went with. This made the final composite much cheaper because I didn't have to convert the depth value into a blur factor for every screen pixel. It also gave me the option to do a near blur almost for free because it was calculated at the vertex stage of the objects shader. In fact by writing the blur factor it allowed us to do quite a few other tricks such as motion blur at a per vertex fidelity ( we ended up using another type of trail blur instead but it did work ).

I'm sorry its reading like a brain dump, I'll cover this all a bit better in the morning with some screen buffer captures.

Cheers,
Bruno

 

Although the Asset Store suggestion is an obvious one, I can't stress enough the added advantage of personally working with Bruno to develop a custom solution for my particular game. Instead of relying on an "out of the box" package we could work together to produce effects that took advantage of the particular camera angles and visual style of Run Fatty Run.

And he could look at my particle effects and make me cut down on the horrible fill rate abuse.

If you're looking for a magic, "Make my game look awesome button," talk to this man. He'll make your game look great and probably make it run faster while doing it.

 

Glowing praise from a respected developer indeed. I can echo that sentiment, its worth every penny to find a decent dev that knows his or her stuff.

 

How is the depth buffer access in Unity4 going to be different than in Unity3? It is possible right now to write to the depth buffer on mobile devices. In fact I am using this ability in my game. What is going to change?

 

Hi macdude2,

I could have described that much better. Its reading from the depth buffer that's the issue in unity 3. I believe in unity 4 it is no longer a problem. I'll edit the post to make that clearer, thanks macdude2.

 

...many thx for sharing your knowledge !!!
a sample scene would be fantastic !
anyway great post !

 

Looks awesome! Nice work Bruno and Jason, checking out the game now.

 

@ Jason, Hippo Muckel, Thanks for the kind words.

I apologise for not getting the second post done. As I was writing it I found a few places where I could improve things, and tidy up the system generally. By the time I'd changed a bit here and there I was half way to creating a usable package. So I'll do the free package up first then follow it up with some documentation.

Kind regards,
Bruno

 

Thanks Dan,

Fantastic work on Ski Safari by the way. I regularly use it to bribe my kids into good behaviour, Its particularly good because even my youngest can play it. There's allot to be said about a single input mechanic done well.

 

Awesome job, and thanks for posting the how-to. This is how intermediate game coders like me learn how things are done!

 

@actuallystarky: I can second that working with Bruno is pleasure!

 

This is exactly what I'm am trying to find a solution for now. I don't have any experience writing shaders, and understand very little about them. I understood what you were saying when you broke down the process, but converting that into practice is unfortunately beyond me. I've been trying to find a good way to learn and completely understand shaders and shaderlab specifically, but so far not much luck. Is there anywhere you might be making this available. We'll quite happily pay for it.

Took me a little while to tie together that you were the same man that wrote the xray shader on youtube. That's a fantastic one too, thanks for that.

 

any chance to got a DOF friendly solution on the asset store out of the box! i m willing to pay to get this!

 

Hi,

I've been a little overrun with things of late so I haven't had a chance to do any more on the package to get it to the Asset Store. However I've put the package together on my site:

Link.

Feel free to test it out on your project and please credit my work if you use it.
Package contains a sample scene with an example shader or two.

Kind regards,
Brn

 

I don't know what to say. That's incredibly generous of you!

I know you're really busy, but I'm looking for someone to teach me shaders from the ground up via email correspondence if you're interested. You'd set the pace so it'd just be whenever you got back to me. Mainly I'd like someone to bounce questions off. I've started going through video tutes, and that's great, but I like to verify that I'm understanding things as I learn. If you're interested let me know, I'd be massively appreciative.

 

Cool thanks, ive been following this thread in the hope you would release it.

Have you played around with unity 4 and the access to the depth buffer?

I would pay good money for a set of iOS post effects that run faster than a slide show.

 

@ ddeaco , I think you will find these effects slightly faster than a slide show . The demo scene runs in the high 50's on an IPod 4 in HD. Also the way the system is set up its very easy to extend to create extra effects. For example in RFR it also is used to do a simple real-time reflection. Personally I've had motion blur and screen warps working with almost no additional cost. I've been advising an Indie team that is doing very well use the system to achieve some really awesome stuff. I think they will release some captures in this thread soon.

I'm going to have a look at using the depth buffer in unity 4. Depending on how quick it is to read from it might be a great step forward. However if porting to all mobile devices is ( Android variants) important avoiding the depth buffer might still be a good alternative.

@ AdsySingle , Feel free to send me an email now and then, I cant promise anything like full tuition, but Id be happy to give you a pointer or two and guide you to the correct documentation ( which is half the battle IMO).

Kind regards
Bruno

 

Thanks for releasing this - it looks awesome.

I'm working on getting it integrated into my project, and I have a quick question - if I don't want a objectto be affected by DOF, is there a way to exclude it? It looks like you are using the alpha layer to determine how blurry something is (or I could be completely wrong, I'm still learning). If I have a shader that uses the alpha color to for transparency then setting the alpha layer doesn't work if I don't want it to be blurry. Is there a good solution for this? You could render the object to a different camera with a different culling mask, but then you run into a sorting problem.

Again, thanks for posting this, it is really appreciated.

-----

Edit: Solved! Reversing the alpha value worked for my case. 1 means not blurry, 0 means blurry. Although it would be nice to know if there is a way to get access to a fifth fixed variable, but that is a general shader question.

 

Wow, thanks for this!

 

Hi opsive,

Have you tried adding colormask RGB to your particle shader. Although the unity docs say that its not a good option for mobile hardware, I cant find anything in the PowerVR docs that suggest it would be. Perhaps Its an armv6 issue only (fingers crossed). Also if you want to blur your particles based on depth, tex2Dlod() might be a nice way to go about it. I believe its supported by GLES 2.0. I'm unsure on how it will affect pre fetching of the particle texture.

anyway nice adaptation on inverting the the Alpha channel. Did this result in an extra calculation though? I vaguely remember setting it up the way to save on a cycle or two. As for the fifth fixed variable, if you mean a fifth channel for the rendertexture? no this is not possible without an additional rendertexture and a whole heap of overhead.

Cheers
Brn

 

Hello,

Is there a way to turn off blur and just use bloom?

Thanks,

 

Great Post .

how to use with my own shader instead of NeverBlur Shader ?

 

Hey opsive,

Can you please elaborate how you implemented the problem with the transparent objects not being affected by the blur? I am trying to make a particle system that should not receive any blur.

and great job brn !

 

I took some inspiration from the approach listed in this post. I developed it further into a "progressive blur blending" algorithm that provides much better visual results without decreasing the performance.

Here are the two major improvements I made to the original algorithm:
1. I made an optimization by downsampling the RenderTexture (which looks as good when blurred, but requires x4 times less pixels to process).
2. I improved the Gaussian blur algorithm, by using self-modification based on the required blur amount:

blur = dX * blurKernel[dX] * blurBias;
​

The first two terms of the equation is the usual blur equation, and the blurBias term depends on Circle of Confusion (amount of blur). By using this modified equation I was able to "scale" the blurring algorithm, and achieve a smooth realistic transition without any drop in performance.​

Another limitation I've encountered is that iOS doesn't by default render to alpha channel. The only two options are HDR rendering and enabling depth texture. I find the depth texture option to be faster and more precise.

This improved version of DOF is available on the asset store in my FxPro asset. I hope people will find it useful: https://www.assetstore.unity3d.com/en/#!/content/22189

 

I just got a popup when I booted unity this morning.

ABSOLUTELY NOT COOL.

I've paid for dozens of plugins. Can you imagine what would happen if they all started using the update window to serve advertisements? Having to close 40 popups when I boot unity? I reported this to support and I suggest you remove it.

 

Amazing asset and works with android too.

Thanks @brn

 

Hello
It seems it doesnt work in unity 2017