How to integrate 360 video with Unity

Hi @touloused I tried a 2k 360 video for Android device with Skybox Panoramic shader, but it didn't run, but the same video with 720p and 1080p was working. can you suggest what could be the issue?

 

Hi @touloused sorry for not including all the information,

I was using Motorola g4 plus android device, OS- Android 7.0 and Unity2017.1

One more details, I used Google VR SDK to split the screen. With Google VR SDK 720p and 1080p video was running properly, but 2k video didn't run.

I even tried the 2k video without Google VR SDK in the same device but it was not working.

I have uploaded the video source information, please have a look.

I hope I have provided all the information, please tell me if you need the project file or the video to test in your system.

The video is running fine in unity editor.

........

And all videos show's this warning in editor
"Tiled GPU perf. warning: RenderTexture color surface (2560x1440) was not cleared/discarded, doing <run with profiler for info>"

Thanks.

 

Snapdragon 617 supports video playback up to 1080p:
https://www.qualcomm.com/products/snapdragon/processors/617

 

Hi @Selzier thank you for pointing out the issue.

 

Is anybody else getting pretty bad tearing and compression blocks when playing back their video in the Unity Editor? Perhaps some codec issues?

 

I got the player built out, but it was fixed with the camera facing in one position and wouldn't let me look around the scene. I thought this was built in with native integration for cardboard...did I miss a step?

 

You can also support stereoscopic image if you flip the x and z radial coordinates for the right eye. (assuming the video is split vertically down the center)

 

Hey, this works great for me in the editor, but I get compiler errors when trying to build.
Is there something I'm missing?

The type or namespace name 'AnimatedValues' does not exist in the namespace 'UnityEditor'
Also 'ShaderGUI', 'AnimBool' and 'MaterialEditor' could not be found

 

Hello!
I followed this tutorial to play 360 videos. I edited and built my project on a PC and it works well on PC and Android. However when I copy my project to a Mac, console shows "Metal: Fragment shader missing texture binding at index 0 (_Tex / Skybox/PanoramicBeta)" . When I run the game, the sky is green with some distortion. It would be great if anyone could tell me how to fix this >.< Thank you!

 

Dear touloused, thanks for sharing the awesome work! Publishing a 360 video with Unity has become super simple indeed!
One issue that I am having is that the top and bottom of the 360 video is "hollow" as you can see from the attached images. Is there a way to "tighten" the hollow gap on the top and bottom of the videos to make it perfectly 360?

Thank!

 

very nice ! have to test this! it should be much more simple but is it possible to use stereo panorama images not videos? images don´t suffer from video compression and limited resolution. i did not find a way to do even this out of the box right? just simple stereo backdrop images for vr.

 

i know how to use images as skybox and maybe i can setup cameras to render differnt skyboxes but what would unity consider for lighting (GI Ambient Reflection etc!?) i am kind of lost here

 

Hi, what's the recommended aspect ratio for stereoscopic 360 video? Ive been working with 2160×2160 mp4 files and I've had to transcode them every time for Unity to recognize. Is that a file type thing or a size thing? Also what's the largest aspect ratio that will work?

 

nice hopefully i get some time to test this! i really wonder what the shader is doing is it just overwriting the clear flags per camera and what is the skybox for lighthing and reflection doing if i merge the panorama with actually 3d content in front of it. good work anyway and a really must ahve for vr!

 

Great addition, but for some reason all of the 180 sbs videos I try tear and jitter pretty badly unless I res them down to really low resolution or reduce the fps, even if I make them a power of 2 like 2048x1024. If I take the same files and try them in something like whirligig or vive software's video decoder project in Unity, they work fine. I'm wondering if there is some setting on my system that is messed up that I need to change? I've got a pretty souped up dev PC with a 1080 video card for my Vive so I would think hardware wouldn't be an issue.

 

Same here, I haven't been able to figure out a solution (see my other post)

 

Code (CSharp):

1.     void Update ()
2.  
3.     {
4.  
5.         if (Input.GetKeyDown(KeyCode.P) && movie.isPlaying)
6.  
7.         {
8.  
9.             movie.Pause();
10.  
11.         }
12.  
13.         else if (Input.GetKeyDown(KeyCode.P) && !movie.isPlaying)
14.  
15.         {
16.  
17.             movie.Play();
18.  
19.         }
20.  
21.     }
22.  

Hello Tolouse.

Sorry if I am posting this twice. I found the forum AFTER I posted the comment on your blog.

I need to control the playback of the video with the keyboard similar to the attached code.

I have tried for several days and cannot make it work.

Your documentation is awesome and I could build a working project that plays in my Oculus with no problem, but I cannot make it work with keyboard control

I want to play, pause and rewind.

I would really appreciate your kind help

THANK YOU

 

Great news that this shader is available now!
I just wanted to share my experience with the shader in combination with a MP4 movie made with the Gear 360 (and exported with the Actiondirector) and the Google VR for unity with Cardboard.
In the editor all movie file sizes worked fine but on my phone (Samsung S5) the gear 360 size (3840 x 1920) did not play. When I used a smaller size (1920 x 960 px) it did work. I guess this has to do with my phone not being the newest?
Thanks for this new feature!

 

Snapdragon 800 can play 4k Ultra HD video (3840 × 2160) at 30 FPS and H264 only. Make sure the video isn't 60fps or h265.
https://www.qualcomm.com/products/snapdragon/processors/800

 

can you tell me how to use it? I want to use this but i dont know how to...

 

Here is a description of how to use it.

@ Selzier: Thanks, maybe the fps was too high indeed.

 

This may not be the forum for this and apologies if that is the case, but my video works fine in the editor but the animated skybox defaults to a different static skybox when I actually bake out into a .exe. I'm working on a VR project but it sounds like that shouldn't affect anything according to the IP -- I haven't been able to find anyone else with the same problem which leads me to believe I'm special. Any thoughts are much appreciated, even thoughts that suggest I'm a big tool

 

Hi, I'm trying create stereoscopic panorama viewer for Gear VR, but there is one thing, that I can't resolve...
I use SkyboxPanoramicShader (from this article), It works good, except all things looks too big when device dressed. I'm really feels myself small inside scene. Then I've found that is stereoscopic effect (or not?)- when we are seeing two identical images by both eyes, our brain correct sizes for us. Here man with nicname "Stereoscopacetic" explain it. If close one eye and then dress on GearVR, then I feel myself normal.

After that, I've tried to create 3D stereoscopic image (Vray stereoscopic render, see my.jpg), and use "3D-layout" parameter in shader, but nothing changes for me - all thing look too big whatever ...
Now I'm realy disapointed and don't understand how to realise correct viewer. Maybe GearVr can't show 3D-stereoscopic content correctly at all? (some people talks that Google Cardbox show same panorams correctly).

Please, help me to understand, is it possible to create correct VR-experience with GearVr and SkyboxPanoramicShader

related topics: one, two

 

I figured for sure this issue was just either me doing something wrong (still might be) or my computer being messed up. So I popped in a spare 256GB SSD and did a fresh install of Win10 and Unity 2017.2.b9 and SteamVR and wound up with the same issue. Several 4096x2048 180 SBS mp4 Videos just stall when trying to play them, even on the near/far planes instead of the skybox texture

If I use the AVPro Asset or the Vive Software Video Playback Asset they play fine in Unity. I can also play them fine using the built in Windows OS Player

I have a Quad core 3.5 GHZ cpu, a GTX 1080 card, and 8GB of ram so I should be OK spec wise. I also noted that the CPU is pinned at 100% when using this Video Player but with others the CPU sits around 60%.

So I will file a bug report, but it's just so weird no one else is having this issue.

 

Not sure if yours is the same issue I was running into, but I found that on GearVR if I had Single Pass set as the Stereo Rendering Method (Player Settings) when I was testing it, the same half of the image was rendered to both eyes, thus making everything feel really far away and flat. I think it's a bug with how the skybox is rendered when doing a single pass. You can test this by drawing something in a specific spot in the stereoscopic image that's different for each eye and then closing one eye at a time while looking there.

Try setting it to Multi Pass and see if that helps

 

Hi, I'm having issues with my project build for Gear VR using this shader. Does anyone know how to fix this?

 

@touloused - do you find better performance feeding equirectangular videos or cubemap?

 

Hey so I have been using this in a project and works great. I then imported my project package to a new project (so I could keep a lot of the settings) but now when I run I get this

Assertion failed: Assertion failed on expression: 'srcAttach < m_CurrentFramebuffer.colorCount && "We should always resolve only current RT"'

Tried remaking it from scratch and still getting the same error, got any advice?

_----------------------------

Ok seems like this only happens when I use the 4096x2048 which is the size of my video, if I change the texture to be 4096x2049 or anything else it works, super weird!

 

Hi
Thanks a lot
I'm a very very new user of Unity and succeed in adding a 360 video.
As I'm a very new , I'm now trying to add Virtual element in this video.
I saw a Unity conference where they talked about adding a second sphere (if I well understood to mix the virtual element and the 360 video)
I've tried some things, but didn't really worked out. I've succeeded in added a 3d character on my video but I'm guessing as it is not in a second sphere this character appears in front of the video ...
If you could help I would be grateful
Thanks

 

@touloused

Can you comment on the exact requirements for the codec/format supported in the player?

yuv420? bitrate? max resolution? etc etc...

 

Hello, I tried to apply texture decoded by AVPro. But the the decoded texture seems like always upside down, thay said it is the original format. If so, how can I apply this kind of texture to Skybox-PanoramicBeta.shader? Can it provide some kind of vertical flipping function? Thanks.

 

Hi..did you get any solution to this issue?

 

Hi, thanks for this "easy" way. But I am still having trouble getting this to run on my Smasung S5 mini.
On my computer under Win 10 x64 it works, since I put the .cs file in a folder named "Editor" (I think this information should be added to the video tutorial and the script tutorial as well.)

I am using VUZE 3D (top, bottom) footage in the newest Unity-version. At 3840*2160 the screen on the phone stays black (except for the cardboard player icons), at 1280*720 I get really weird pictures:



Okay, apparently it works fine on a Samsung S7, my S5 mini seems to be too weak.

 

Hi there,

I'm attempting to implement Offset / Offcenter Cubemaps in a stereo video player.

As seen here at Oculus Connect 3 - they allow increasing resolution in the forward direction of 360 videos, taking pixels from behind the user.

I've gotten as far as updating the skybox shader you've written to accept stereo cubemaps, and to remap the cube layout to the somewhat unique 2 x 3 grid format that Facebooks Transform360 FFMPEG plugin produces, but am falling short being able to rotate specific faces to match Facebooks Transform360 Offset Cubemap layout (2 x 3 grid)

Using:

float3 coords = normalize(i.texcoord) + float4(0,0,-0.5,0);

And passing result to:

float2 tc = ToCubeCoords(coords, i.layout, i.edgeSize, i.faceXCoordLayouts, i.faceYCoordLayouts, i.faceZCoordLayouts);​

As seen in images below, the 2 x 3 layout (in LR stereo layout) rotates the left, front and right so that their edges align, and does the same for top, back and bottom (for more efficient video encoding).

The result however is seen in the second image.

I'm trying to follow along with the "ToCubeCoords" inline function in the shader and really only sort of get what is going on (using dot product and z axis to determine which face is being targeted in the layout)

It's unclear to me how to pass the necessary logic to rotate only the left front and right faces anti clockwise before passing back the coords.

Any shader writing advice from the experts appreciated.

 

Hi @touloused ! I'm using 2017.3.0b2, and when I use the Skybox Panoramic shader in the editor, everything seems ti work fine. I'm building for Google Cardboard, so I have the GvrEditorEmulator in my scene, the video plays, and I can look around. When I build & run on my iPhone 6, I get a black screen and an error that says
"Execution of the command buffer was aborted due to an error during execution. Ignored (for causing prior/excessive GPU errors) (IOAF code 4)".

I'm using a 960x960 3D 360 video, so resolution shouldn't be an issue. Do you know what the issue might be here?

Here's a link to a Drive folder containing my project: https://drive.google.com/file/d/0B2VH_MEBOklTQ1A1WnhBT0R4WG8/view?usp=sharing
Open the stereo360testSkyboxPanoShader scene- it contains what I mentioned above.

Thank you!

Best,

Jacob

 

In case anyone else comes across this problem - I had an issue with a seam at 180 degrees (i.e. "behind") with equirectangular mode. The attached screenshot shows what I'm talking about. I fixed it - mostly - by disabling mipmapping on the (still) texture. There's still a slight seam by default, regardless of any other texture settings (e.g. wrap mode) - this can be decreased by adding the following lines to the shader in the frag function.

Code (CSharp):

1. float2 tc = ToRadialCoords(i.texcoord);   // Look for this line and add the following:
2. float stitch = 0.0005;
3. tc.x = min(1.0-stitch,tc.x);
4. tc.x = max(stitch,tc.x);

The value of stitch can be increased a bit to remove more of the seam, but at some point the resulting texture overlap will become too obvious.

 

Hi, I have no coding experience. Could you tell me how to create the Skybox/Panoramic material from the files you have posted?

 

Hello! We're using this to display stereo (upper/lower) panoramic images and are encountering a very visible seam. This doesn't happen when playing stereo video files.

Using Unity 2017.3.0b6

Created a material
Used Skybox/PanoramicBeta for the shader type
Dragged the image (7680x7680) into the Spherical (HDR) area
Then dragged the new material into the Skybox Material area

When playing, it looks great - the stereo effect is working perfectly, but there is a very visible seam.

Here are my import settings.

 

I just figured out why the seam was happening. I had to uncheck "Generate Mip Maps" and that made the seam go away.

 

Just got this up and running like a charm, however the audio isn't playing back on my GearVR. Any thoughts?