Turn your phone into a First Person Shooter Controller

Yo guys! Made this last week. Curious what your thoughts are in regards to usage and adoption of something like this? Have you seen something like it before? Do you think people will use it? Let me know!

 

I had a similar idea, but instead use the screen as a "window" into the game and just play on the phone.

It looks really smooth btw.

Are you using the gyro or the compass? Or both?

 

Gyro at the moment! It got the job done pretty well

 

Cool idea. I like the idea of a reactive controller, where it can change the available buttons and functions depending on what's going on.

With a touch-screen based device there are downsides, like a lack of tactile response, but there are also upsides, like being able to do stuff that's not possible with normal buttons.

 

Thanks! Appreciate it. And yeah, I'm wondering if there are VR applications for this.

 

hmm
at some point in the past i was sorta griping over the fact that ... like a PC keyboard/mouse input has many buttons, but, a xbox 360 controller doesnt have enough buttons

with a phone as input controller.. could have any layout imaginable for input i suppose .. that seems cool
its like
there has been many peripherals, custom controllers in gaming,
a problem is youd have to manufacture them (lotta money)
but with this .. everyone has a smartphone

lol oh yeah i thought of this long time ago.. dont know how to do it and dont have a use for it presently but i have like .. my dream game that would use it

hows that work? .. networked i guess durr
hmm .. is there such a thing as like .. plug phone in to PC with usb .. or usb wireless stick, and it takes inputs from the phone/tablet ??
cause if there isnt .. there should be, and if there isnt thats really stupid.. why isnt there? ppl be too stupid? lol

ugh i swear i give out million dollar ideas everyday, and here iam broke a.f.
ah yeah there is .. good lol

crossbow is pretty sweet nice!

 

Would be interesting to use it as a gun positional tracking for realistic aiming. My experience with accelerometer isn't great, but gyroscope seems very accurate. So accelerometer could simply be used to detect shaking of the hand, but with gun always returning to position smoothly.
This way you'd actually use gun's sight to aim.

 

Having spent over a year on control systems like these there are definite pros/cons. 2 out of the 3 assets in my signature have dealt with this. Easy Wifi Controller (EWC) is general purpose and I built a multitude of example scenes (for marble type game, fps, etc.). The Apple TV one I built a similar control scheme as EWC using the siri remote accel/gyro as well.

I will say when not in a VR environment, I don't really see control schemes like these as preferable. There is a disconnect between your aiming and the screen you are viewing it on which tends to break immersion quite severely. In VR I do see a future in control schemes like these but the cons of accel/gyro/magnometer need to be addressed. I think both valve vr controllers and oculus touch will do so, but having played with phones/tablets I don't think they are quite good enough to be the default. They are fine as an optional choice but have the following drawbacks.

1. Lack of precision (accelerometer/magnometer noise and gyro drift)- Even if you use sensor fusion there are a bunch of edge cases where the precision of something like aiming a gun is a no go. Simpler cases like racing games using tilt is fine, but accelerometers and magnometers are noisy (jittery) so in a sensor fusion environment you generally smooth this out at the sacrifice of precision. Gyro's are not noisy but they tend to drift over time which needs to then be recalibrated with the noisier sensors. You can almost always notice this if you do several quick orientation changes (like aim left then right repeatedly and then stop). You will generally notice you are off by about 20 degrees when you stop and then about 2 seconds later poof there is a sharp adjustment even though you didn't move. Internally this is the gyro racalibrating with the magnometer but it highlights the point and happens all the time to a lesser degree, which is these 3 sensors together are not super precise in the hardware that most phones/tablets use. Something like aiming a gun especially over the course of a level needs to be precise if you hope the user to actually choose this control style.

2. latency- In general you want the latency to be under 20ms (ideally under 15ms) for any control type. On good wifi I can achieve this in EWC (the apple tv one is well under that) but not all people have good wifi. You mentioned 30ms in your tests and most users can definitely percieve that especially when it doesn't line up with the frames of the game. For a 60FPS game (16ms frames) even if things line up perfectly the input is still 2-3 frames late which is not ideal. Everyone is different but unlike films in which people were content with 24fps, controllers and there actions can be sensed as delayed at much higher frequencies. It is best to get the reaction times better for a good "feeling".

I have no doubt that oculus touch and valve vr controllers will address all the needs but keep in mind at a hardware level there doing much more than a phone/tablet. They will likely have better sensors in them and in addition they also have the cameras assisting them. To get something that feels right I think that for some time this will be necessary until the sensors are so cheap and so good that you can start putting many multiples in devices.

Very neat demo though, it's always nice to see someone else playing in this area.

 

I have a use for this for one of my extra projects. Am interested. Asset?

 

Lack of precision, lag, and I guarantee that holding your hands in air for prolonged periods of time will be tiring.

Probably won't be very useful, although Wii U controller could probably be used in this fashion.

Also it is definitely not what I'd use in a first person shooter game.

 

@neginfinity is right about the cons, in an fps you wont be a match for any player who is using the a mouse.

I've did this too, way back in the days. It super easy to achieve and I've seen people do it before. You're basically doing what Unity Remote 4 does, but over wifi.

If so many people have made this before how come you don't see any fps games coming out that use this?

• There are more specialized devices than smartphones that handle the only guroscope input or joystick input
• It's not effective in general

If you are interested in using this in a game you would be better off looking into other genres of games, specifically ones that don't require quick reactions.

 

As an avid FPS player my honest opinion (and I say this with the best intentions) is that this project is a complete waste of time. I would never ever even try this and I doubt that any one of the FPS players among my friends would.

 

To be honest, I'd be interested in precise lagless gun/rifle controller and a virtual shooting range (just a screen, without oculus).

That kind of thing won't be an "FPS" though. Something akin to House of the dead and older lightgun games.

Unfortunately with the death of CRTs making lagless gun became significantly harder and lightgun games were always a niche genre. Available motion controllers tend to use image recognition for determining gun position, which is a bad joke in my opinion (because you'll have unavoidable processing lag).

 

@pdwitte ping me if you intend to pursue this further

 

While I agree with your sentiments around current FPS's from a competitive perspective, I find it a bit of a jump to state it's a complete waste of time, especially in a VR context. Given the choice of the two, most people wanting to win will currently choose mouse/keyboard because it's the most precise (next followed by a controller). Any first person style controls I've played in VR though is not ideal in either control scheme and it's clear that something needs to change. While I don't think a phone's motion controls are good enough, I'm pretty sure some form of actual hand aiming via oculus touch or whatever vr input style wins out will be preferred in a VR context for first person style games.

The one thing I'm not sure about is how the movement will be handled. For GearVR it's clear that a swivel chair seems to have won out. Valve, Oculus, and PSVR will have greater range but not that much range so for these style games I'm curious as to what actually will move you, though I suspect it will still just be a joystick and perhaps only rotation will be driven by real world coordinates.

 

So far I haven't had the oppurtunity to experience any kind of VR systems . I've played motion controlled games though, including the first Red Steel on the Wii and some PS Move minigames. Personally I'm all for everything that enhances immersion in a 3D experience, but I find current gen Motion controls to be actually detrimental for that. When I use an input method that has a high disconnect between what I do and what happens on screen (press button -> swing sword), the suspension of disbelief works fine for me. Be it mouse and keyboard or a console controller, the input device doesn't feel like it's getting in the way of my immersion. I would never think "but I only pressed a button, why do I see myself swining a sword?". With the Wiimote on the other hand I have a smaller difference between what I do and what I see (swing wiimote -> swing katana in roughly the same direction but with latency and different speed), but there always is a difference left. None of the methods I've experienced so far had actual 1:1 transmission of movement, it was all triggering canned animations based on the rough direction of your swing. Very limited, and to me that feels like the "uncanny valley of motion controls". It's always present for me, that the game didn't accurately replicate what I just did and I think about that fact instead of the game. In the end I often ended up doing some weird motion that is even further disconnected from the onscreen action, because it is less tiring, more efficient in the game context, and the "proper" way didn't immerse me anyway. I don't like when I do the e.g. sort-of-sword-swing with motion controls and then get a canned animation because that makes me think everytime "that's not what I did". When these things were close to enter the market we all thought that we'd be having awesome lightsaber fights etc.. But imho we'll never get 1:1 motion controls because that would raise the skillceiling by orders of magnitude. Imagine people would suddenly need actual swordfighting skills to compete online in Jedi Knight 13 some day. But that's only part of the problem. You'll never have the proper feedback in such a system when the ingame object collides with the ingame world. I've played a gladiator duelling game against a friend with PS move controlls and I tried my best to do some sensible attack/block/counter moves and he on the other hand tried jumping up and down doing overhead swings as fast as he could. Competitively he crushed me that way. But what I found worse is that the disconnect between IRL movement and onscreen movement became sooo jarringly different for both of us, that I'd have preferred to just be pressing buttons again. Motion controls don't account for being stunned or staggered or even landing a hit that stops your swing. And how could they? Maybe in some future highend device with robotic arms that give tactile feedback. Something like the phantom omnidevice, but big enough to stand in and get the accurate tactile feedback of holding a two handed sword, would be an interesting experience. I'd see players knocking out their teeth with it, but interesting nonetheless.
Now lets talk about guns, because that's what the original idea was about. It's just easier to point out all the flaws with sword gameplay examples. To make it feel "real" a rifle would need to weigh several kilograms and have really heavy kickback. Both are impractical for actual use. Hell, even without holding any weight my shoulder would start hurting when I hold my arms up in front of me for a few minutes.
How would aiming with the gun work? Does the gun replace the mouse movement? Isn't any disconnect between your head movement and ingame camera movement the primary cause of sickness with VR experiences? I never tried it myself, maybe you can confirm this. My impression from what I read is that for a VR FPS to not be nauseating the head movement needs to replace he mouse for the most part, correct? It might work if the motion controlled gun really only controls the ingame model of the gun, but the position had to be accurately transferred to not break immersion. That wouldn't work with games like Counterstrike where you have crosshair based aiming. For games like Arma, Battlefield, CoD or Insurgency it might work, but they'd have to redo the entire gun animation system, which I doubt will happen for niche markets. The shape would need to closely match that of the onscreen gun, you'd need to have the stock area be shaped in a way that doesn't collide with the VR headset when you aim down the sights and recoil mechanics would need to be reworked.
I'm not dismissing the entire field of motion control research as a waste of time, I'm saying doing this with a phone as your controller, that has 30ms of lag and no physical buttons, is a waste of time. Imho the phone approach is a dead end. I can't imagine anything worthwile coming from this. This is the minimum set of buttons that I need for the FPS that I play almost daily: w, a, s, d, q, e, r, f, y, x, c, shift, space, lmb, mmb, rmb, scroll up, scroll down. You just can't fit those on a touch screen. It would be impossible to accurately access those without tactile feedback. And that is the MINIMUM set that I'd need, I actually also use tab, ctrl, esc, m, b, enter, alt.
I see plenty of problems with dedicated gun shaped motion controllers as well, even if they had close to zero latency and the games would go out of their way to accomodate them. But I'm not saying that that shouldn't be explored. Fitting enough buttons on them is a challange in itself, but at least getting the same number of buttons that a controller has seems possible.
I have yet to actually try VR myself and I'm really hoping there will be a market for it and I hope FPS games will be a part of it. I have a suspicion though, that the most practical solution for VR fps will be quite the compromise. I could imagine it would work best to have a very wide virtual monitor with blurred edges in your field of vision that roughly sits where my 3 monitors would be, fixed in the virtual space. I'd keep using mouse/keyboard/gamepad exactly like the input schemes work today, but have a window into the world infront of me that gives 3D depth perception and all the distractions of my desk are hidden away in blackness. When I move my head sideways, the virtual camera of the game world would also move, so peeking around corners would be possible. I could very well imagine such a compromise to be a welcome addition to the established way of playing FPS games and it would require relatively little effort on the part of the devs to implement into existing games. But I could also very well be totally wrong and overlook jarring flaws in that concept because of my lack of experience with VR.

 

I agree that hardcore gamers won't adopt this as an improvement over mouse + keyboard for classic fast paced FPS.

But for other applications I think it would be a great idea,. Basically for wiimote type of games on PC... without the barrier of having to market a new controller.

 

I agree completely that if there is a simple action (button press) that represents a real world action that we are more likely to accept it than a not entirely correct more complicated action. Your sword fighting example is a good one that is likely to be too complex to be tackled in the near future. The biggest disconnect would occur when two swords hit eachother and there is no way to simulate this and why it will break down. The sword swinging would be just fine in a fruit ninja type experience or the simplier lightsaber games like the one on the apple tv where you only deflect disrupter blasts and not actually swordfight. Anything beyond that would suffer from what you describe. The gun analogy is not this complex though and in the VR space can be simulated.

I don't fully agree on the skill based argument. There will be an audience that will like it less than a simple arcade style game, but there will also be those who like it more (provided you can accurately simulate it which for swordfighting you cannot). In any genre there is usually more than one style of game. In shooters there are the arcade like unrealistic ones like Halo or UT, and then there are pretty realistic ones like the ghost recon or splinter cell type games where one bullet usually killed you. These usually had drastically different audiences but there was a large following for both. It's important to note that there are usually steps to this realism it isn't always everything or nothing. Racing games are a good example, there are arcade style games where your car is essentially bulletproof, simulations that if you crash your front left the car drifts and goes a lot slower, and shades in between. As long as the system is predictable the middle ground is fine. Most people tend to accept that the game does want some damage if you smash into the wall but yet doesn't want to fully crumple entirely realistically and sets up a few damage states. It doesn't have to be all or nothing as far as simulation goes and the same would hold true for a gun in an FPS. I think your statement on when the action becomes unpredictable or lacks proper feedback (like your swordfighting example) is when it is worse than a simple button press.

In VR the head movement has to be just like reality looking around otherwise nausea will ensue. This is close to your mouse look but not entirely the same thing. The up and down is as expected but the left/right will turn your head only which is a slight difference from classic FPS's where it rotated the entire body. Not saying you can't make a game differently but in the end nausea free will win out. No one wants to get queasy. You also have the ability to lean like reality which was only in a few FPS's as a keyboard key. This leaves movement and left/right rotation that will still have to be tackled by other means (most likely still joysticks on the valve vr controller/oculus touch). I don't agree that the weight has to be correct there are enough demos that I've played with and at least to me my brain doesn't need to need to have things be heavy even if I've used them in real life and know how much they weigh. Out of what I've played with so far making things pretty realistic seems most comfortable. In a FPS that would mean that you would actually pick up the gun and look down the virtual "sight line" if you want to or hold it at your hip if you don't need precise aim.

Totally agreed on the mobile phone not cutting it, but until everyone can get there hands on later in the year either of the two more robust systems, its as close as most can get. It's not off by that much, just a bit annoying when the gyro re-calibrates it's drift. As an end-user it isn't acceptable, unless you are making a simple racing or marble type game, but as a developer it does help you get a jump on what's coming and being aware of the things that need to be tackled.

Gear VR is the only system that is currently consumer ready and it obviously doesn't have any premade motion controls yet. In the meantime on gearvr there are certainly a number of games that have used a controller. To me, they all feel flat compared to the simpler experiences that rely only on the helmet. I'm not saying they're bad games some were pretty well done, but using a controller in VR usually makes me prefer to just have played it on a flat screen, whereas the other titles make me feel like VR will probably make it this time around. Until motion controls comes out those full VR experiences tend to be isolated to turret type, racing, or spaceship games which can easily do without motion controls, where others like FPS's are forced to use a controller until the Oculus touch or Valve VR controllers come out.

 

I'd like to address few points here. (weapons/props in VR).

Swords. Some of the first games had free sword movement. Examples are Thief (original one) and Die By The Sword. I'm surprised nobody tried to port those to VR (or make their clones) yet. Lack of physical feedback will be a big problem, of course, but aside from that I think it should be doable.

Guns: You'll need separate movement of the head and the gun. So you'd need to aim for real. The shape (and possibly weight) of the gun shouldn't matter much, as long as it sorta resembles in-game weapon. It might be possible to make control sorta flexible and adjustable to accomodate different weapon types. Heck you could even make it transform itself, but that'll be definitely expensive.

Also see Trespasser game which had freely moveable moveable arm you could use to interact with environment.

In both of those cases, canned annimations will have to go. Also you'll need some sort of input device that will read movement/position of your arms (shoulder->elbow->wrist->hand, preferably down to the fingers and for both hands) AND correctly calculate position of your weapons.As far as I know, there's no cheap tech of this kind available, but I might be missing something. Without replicating hand movement you'll need to remove in-game hands completely and just display floating weapon instead, plus maybe a marker denoting position of your hand - in case you drop the weapon. Failure to do so will create disconnect with the game world. Another problem is that for some reason a lot of motion control seems to use cameras and image recognition to calculate controller position. I think it is a fairly dumb idea (lag), and someone should definitely look into alternatives.

Keep in mind that it is still possible to use VR games with simple gamepad. That works well and doesn't create discomfort, surprisingly.

Heck, I'd LOVE that.

 

It's a wii for the computer.

It looks cool though, just dont make it a horror game... so many dropped and broken phones.

 

I've played through all 4 Thief games and unless you mean another Thief game I'm 99% sure that Thief: The Dark Project only had swing left, swing right, overhead swing and block. I never played Die by the Sword but I remember it had free sword movement. Dead Island (possibly also Dying Light) also has a sort of free mode, but I didn't like it.

That was a fun game (with many bugs and flaws), way ahead of its time. I think your idea of porting some old games like Thief, System Shock 2 or Trespasser to VR is a very good one. Their low end graphics should allow them to be rendered at sufficiently high framerates for VR even on older computers. Also some of them are popular franchises that people surely would love to dive in again in full VR. That would probably be the actual 3D VR experience, that has been promised to us in the 90's, even graphics-wise ^^.

I'd have to try it first, but I'd probably like it too.

Do you know a method that has less lag than camera based approaches?

 

I've seen few apps using vibration as tactile feedback. Works quite nicely IMO.

In any case, I had a similar idea, but instead it would be general-use gamepad. Never had the time to work on it tho.

 

No, I think it WOULD be a first person shooter. You view world using first person camera and you shoot.

 

@greggtwep16: very good arguments! Somehow I didn't see your post when I wrote my reply to neginfinity 0_o. I didn't mean to ignore you.

 

It'll be virtual shooting range, similar to house of the dead. OR lightgun arcade game. FPS implies free movement.

Nope. You need 3d positional sensor. That the problem. 3D orientation sensors exist, but not positional ones.

There's some mentions of magnetic sensors on the web, but nothing specific, and frankly, you'd need electronics/robotics engineer for that question.

Another option would be an "arm" attached to floor/ceiling with gun controller mounted on it. That would work well for some sort of turret simulator, but not for a something like sword. That kind of system wouldn't have any significant lag. However, robotic arm with a gun controller is going to be very expensive, and you'll need servos in it, to make sure moving the gun doesn't require significant force.

In short, it is very easy (but not cheap) to make lagless controller similar to what is used in robotic arms sometimes (you have small mechanical arm, you move that, and bigger robot arm mirrors the movement - called master/slave control), but not track free-floating object in space.

 

Lag is not the problem with the phone (the threads original implementation states 30ms but you can easily have wireless systems with less than 10ms response time). Even using standard wifi I can achieve with EWC less than 15ms response time with both ends wireless and half that if the PC is wired and only the phone is wireless and thats using regular wifi. Bluetooth systems like the Apple TV can easily achieve the same latency which is virtually identical to a wireless xbox or playstation controller that everyone has been using for years. If the response time is less than 15ms it's going faster than your 60FPS game (16ms) and is not distinguishable by anyone. GearVR is currently 60FPS, future VR solutions will be 90FPS or 120FPS but even at the highest if you have less than an 8ms response time it can't possibly be the bottleneck and wireless (either wifi, bluetooth, or custom wireless) can definitely achieve this. It's best to separate out this specific's threads solution (which unfortunately uses a browser) and what is easily achieved with the wireless medium in general.

If you use any of the other common technologies where latency isn't the issue with the phone, you come to realize that precision is the true culprit. The cameras that are in oculus touch or the lighthouse sensors in valve's solution are to have another set of sensors to ensure that precision of position and orientation are correct. Even the more robust motion sensors (like the Invensense 9xxx series) aren't good enough if you put them in the phones (most phones still use the older Invensense 6xxx series). The cameras are there to give you more accurate results not to combat lag.

 

Check this out: http://produckhunt.co/

I'm working on a game that uses phones for controllers and websockets are fast enough over wifi that I have no lag(It's much much faster than my old version that used UNET)

This is some kind of a strange joke right? Try any new VR headset and you'll see that you don't need a mechanical arm, in fact even the DK2 tracking is fine. The lighthouse system is the best I have used (I haven't used the CV1 Oculus) and it uses a cool system.

 

I wish UNET were faster, I've been looking to port over my EWC asset for awhile now so that I could add the console platforms, UWP, Tizen, etc. but it does seem to not performance test close to the speed of .NET sockets. Hopefully 5.4 brings more performance improvements.

 

I think it's just a bunch of overhead, when making things simpler it gets slower, a shame though. With time I think they will get to a point where it's good but not for a while yet.

 

Looks pretty awesome to me! Nice work

I've never seen a controller like that before but it makes a lot of sense.

 

I tried DKII, the system lags badly on occasion, about 2 or 3 frames behind actual head position on occasion. It is not good enough. Since all headsets seem to be using the same tech, I'm skeptical about all of them.

It isn't noticeable if you stare forward most of the time, but start turning head around, and it is noticeable.

There's already enough delays processing input, and you don't need to add more. With 15 ms you have 50% of being one frame late processing user input.

I want input lag to be zero milliseconds. With robot arm manipulator you can get fairly close to that. With crt lightgun cabinet you can get fairly close to that. However, I've yet to see any image-based system that could do the same.

 

This is going to be highly dependent on the computer that is used. VR does need quite the beast of a machine to be a smooth experience. I will say that this is not in line with my experience with DK2.

If 15ms is too much and you actually perceive this then you have to stick with a wires based solution. This would also mean that you don't like wireless xbox or playstation controllers and can perceive the input lag there as well. You'd definitely be in a very low percentage of people that can, but it certainly stinks if that's the case with your senses.

Obviously CRT type technology is a thing of the past due to other drawbacks, and robot arm manipulators, gloves, or any wired solution can be achieved but most people prefer wireless (probably due to most can't perceive the latency). With the display being so close to your face the threshold of perception will certainly increase for everyone in VR, but given that the average non VR game having 133ms of latency (the majority of that in the display not input) there is a lot that can be done to minimize the full spectrum of latency. It's no accident that Oculus and Valve are riding so close to the metal on the displays, it's the biggest challenge for them to tackle. Timewarp and other approaches to this problem should be able to significantly reduce display latency. Even currently with non VR, input to PC represents well under 10% of the overall latency. Often, even with a wireless controller the input latency is faster than the framerate of your game.

 

I like the shooting range idea

 

Love this analysis. Thanks so much.

 

You can email me at the email on the youtube video if you'd like Happy to chat.

 

Yeah! This uses websockest too. I put it together at GameJam, so it's pretty hacky, but it's node.js, websockets, and it runs in the phone's browser, getting ~30 ping on a decent connection. I think with native on mobile, GO on the backend, and better distributed servers we can probably get it down to <20ms.

 

Yep! Definitely true. I wonder if there's an application where precision is not 'that' important, but so that the tech is still fun enough to play it for a little while.

 

Yeah, I'd have to get a pretty good lawyer on board for that one! lol