Using new Mozilla JS video decoder in Unity

Mozilla just released "ORBX.js, a video codec roughly comparable to the industry-standard H.264 that can be decoded entirely in JavaScript."

Check it out on Ars Technica:
http://arstechnica.com/information-...rstechnica/index+(Ars+Technica+-+All+content)

Seems like you could do many cool things with this technology, for instance running a Unity game on a server and piping the output to a browser or mobile device as a video. All you'd need on the device itself would be a relatively thin client that decodes the video and handles the GUI.

However, what I'm interested in right off the bat is trying to use this technology to decode videos within Unity, at runtime. It handles live, streaming feeds, which has been on the wish list of many Unity developers for a long time.

Assuming I can find the time, I'm going to start investigating the feasibility of porting / using the ORBX.js library within Unity. Who's interested in collaborating on this?

 

Wow nice,

i got pretty excited reading this until i hit the Challenges remain section.

 

Defero,
Yeah, I go back and forth on this the more I read about it.

On the one hand, it's nice that it's low latency and the decoder is open-source. On the other hand, the fact that the algorithm is not open source and it's not decoded at the hardware layer are real buzz killers. Reaction from developers has been fairly negative: http://dummdida.blogspot.com/2013/05/orbxjs.html

Then there's the fact that after the initial media announcement (in May), there has been almost no chatter about ORBX. It made a splash and then disappeared, which tells me that once people checked under the hood they were not enthusiastic enough to keep working on it.

 

Quick update:

It looks like Amazon is planning to use ORBX for a new game streaming service they're setting up.

https://aws.amazon.com/marketplace/pp/B00FGB60MK

The idea is to run games on an instance on Amazon, and then stream video output to a thin client - they do it all pluginless in the browser with the ORBX.js library. On the Amazon side, you have an instance comparable to a current or maybe next-gen console.

I'm wondering if this is the way consoles are going to go: rather than purchasing an XBox or PS to keep in your home, you purchase time on a cloud instance and stream the games from there. Functionally, is what Amazon is offering any different from a console platform?

 

So this technology can liberate game developers from all piracy, performance, platform and compatibility issues and nobody is excited? The narrow interface of the system allows almost perfect safety from hackers, as it needs to read only user input where buffer overflow attacks and the like seem to be preventable.

Ok. We all know there will be a lag problem. The question is: What will be the minimum lag be? For some games like adventures it may not be big deal.

As the Amazon system seems to have twice the performance as an Xbox One, it may be able to stream multiple games with mobile resolution. I hope Unity will offer a plugin for such a machine one day!

How about emulating multiple Android devices on such a machine, for example 20 or even 50? Then a special Unity plugin for Android, iOS etc. could do the client stuff and execute the Orbx.js. The cloud service would render the game for dozens of players. Nobody could every steal your app because it runs on a server.

The other problem may be that cloud companies like Google and Amazon will be trying to get control over it. Eric Schmidt joined the advisory board of Otoy recently. As games are an important strength of existing platforms (iOS, Android, Windows, etc.), some big companies are likely trying to prevent this development. Because if you could stream every game to every machine or device, new competitors to the established platforms can take over. All they need is an Html5 plugin to run many games and applications. Google for example needs Android to spy on mobile phone users. A new competitor platform would put their Android investment into question. And lets not forget the US government's secret services want to monitor every activity of the world's citizens and stream that information into their big data algorithms. That's why this technology will probably run only on servers from US cloud companies. I hope there are some forces in tech world I don't know who can prevent this dark scenario. I don't want games to be part of a new surveillance system, like the Internet has become one.

Hope this technology is real and the intentions of the companies and governments are better than they look like today. And that it's not just a temporary marketing show from Amazon and perhaps Google to say cloud computing has a future despite all privacy concerns.

 

FYI, we got our application successfully running from AWS using ORBX.js. Here's a screenshot:



All in all, not too bad. The lag between user input and action in the application was not bad, ranging from not noticeable to maybe 1/4 second. Of course, we are on a nice network here at work.

It correctly captured all mouse buttons and scroll wheel, and keyboard input, though the keyboard input was definitely a bit slow.

The application performed pretty well, without noticeable hiccups. It seems to downsample the video stream at random intervals, which was kind of annoying - that's probably its response to low bandwidth or high latency or high CPU/GPU use on the instance. It did have noticeable artifacts sometimes, for up to a second, when quickly changing the content on screen. However, most of the time it was actually pretty impressive.

It runs entirely in the browser, no plugin, and loads pretty quickly. One strange side effect is that two users can load the same stream at the same time, and it will take input from both people. So my coworker and I spend a bit of time fighting over the application I could actually see this being a cool mechanic for games, if the backend could be made to distinguish between the users. Something where you're trying to flip blocks to one color and your opponent is trying to flip them to another color.

It would probably have issues with shooters, especially with FPCs, because so much of the screen can change frame to frame. But for a Civ-type game, or an adventure game, or a 2D scroller, it would probably work well.

Our next steps are to get it running full-screen in the window and limit to one end user. I'll try to capture some video for those who are curious.

 

Here's another screenshot showing our application running in a window on a remote desktop, which is running on an Amazon instance and being streamed to ORBX.js

Here we're spinning the globe while I take the screenshot. You can see the typical frame-to-frame blur of movie encoding. You can also see a few artifacts, like the stripe on the left side. There are definitely occasional artifacts in the stream, and yeah, it can get a little blurry while moving things quickly, though you don't notice it nearly as much while interacting as you do with the static screenshot. Think watching a movie vs watching a still frame of the movie - you don't notice the blur while it's playing.