Resolved ARKit support for iOS via Unity-ARKit-Plugin

Awesome, thanks for the quick reply!
- For Crash on iPhone 1 - under "General/Software Update" it says "iOS 11 Your software is up to date"...is there someway to force a updated build of iOS11?
- For Crash 2, i'm updating the ARKit plugin now. Fingers crossed

Thanks again!
J

 

Yeah, pretty sure I had it set to something very large, like 5000. I'll triple check though and get back to you.

 

Hi! The iOS version is 10.3.2 on my iPhoen 5 (on my iPhone 6 as well). Do I have to update the software on my phone? Thanks.

 

Oy, yep. plugin update worked. Thanks for that. Also an iOS11 build update hit the other phone this evening, all good there as well.

Thanks again all.

 

Just released on bitbucket repo due to popular demand: Unity ARKit Remote! Also released on Unity Asset Store.

Here's a demo:

And here's usage instructions:

Unity ARKit Remote

This is a two part solution for game developers who want to iterate on ARKit apps (made with Unity ARKit Plugin) from within the Unity Editor. It consists of an iOS app (UnityARKitRemote) that provides the ARKit data from a compatible device that it is installed on, working in conjunction with a GameObject called ARKitRemoteConnection in your Unity scene for your ARKit project. This GameObject simulates ARKit working in the editor by passing data via UnityARSessionNativeInterface to the other GameObjects in the editor. The UnityARKitRemote app on the device will forward the following information to Unity scene in editor:

-The video feed coming from the device camera (separated into Y and UV textures) as provided by ARKit.
-The camera translation and rotation based on device movement
-The ARPlaneAnchor addition, removal and update events along with data about the plane affected
-Point cloud data

Requirements:
It has all the same requirements as the Unity ARKit Plugin, with an additional recommendation of using Unity 2017.1 or later, as the PlayerConnection works way better and has better UI support in that version.

Future work on this:
-HitTest API
-AR Session Configuration

Steps to using remote:

1. First download the latest Unity ARKit Plugin code that includes the Unity ARKit Remote from either bitbucket or Unity Asset Store.
2. Build the scene called UnityARKitRemote out to your compatible iOS device. You should use “Development Build” in your Build Settings. This is the only build time to iOS you will need to endure - the rest of the iterations on your project can happen in the editor. When you build out to iOS, you should change the product name and bundle identifier in the PlayerSettings to signify that it is the Unity ARKit Remote.
3. Open the scene which contains your app that uses the Unity ARKit Plugin. From the ARKitRemote folder, add the ARKitRemoteConnection prefab into the root of your scene. If you want to test it out, it has already been added to EditorTestScene in the same folder.
4. Run the UnityARKitRemote app from step 2 on your device. It should display a black screen with “Waiting for editor connection..”
5. Press play in the editor: your game window should have a green screen with “Please connect to player in the console menu” near the bottom.
6. In this step we need to connect the editor to the Unity ARKit Remote app on the device. This is where Unity 2017.1 comes in handy: it has a menu item in the Console window to “Connected player” with a dropdown of all the available players to connect to. Select the one that corresponds to your device. In Unity 5.6 variants, you have to create a Profiler window via Window/Profiler menu. And then at the top of the Profiler window, there is a dropdown “Active Profiler” from which you select your device.
7. If you were successful in step 6, you should have a button on the top part of your Game window labelled “Start Remote ARKit Session” in editor that when pressed will start the ARKit session on the device and start transmitting data to the editor. The editor should be displaying the same video as the device, as well as navigating through your scene, and it will momentarily start showing the point cloud data as well as the planes found.

Let us know how you like it, and what improvements you'd like.
Thanks.

@ina @SweatyChair @Alex_F @DerekLerner @TheTaj

[edit: it has landed on Asset Store]

 

Have a look at this announcement: https://forum.unity3d.com/threads/a...nity-arkit-plugin.474385/page-13#post-3153415

Then look at the latest source on bitbucket repo, as the Unity ARKit Remote does exactly what you need (sending Y and UV textures over the network). See UnityRemoteVideo.cs and helpers on the ARKit side, and ARKitRemoteConnection.cs and UnityARVideo.cs (uses same shader as ARKit apps do to render RGB color) on the destination.

 

Please check the requirements: you need at least an iPhone 6S (or some other iOS device with A9 or better processor), and you need to install ios 11 beta on that device.

 

@jimmya : Great idea! Any plans to integrate this with Unity Remote?

 

None at this time - Unity Remote data is received in editor native code and not in script, so there are some complications. The communication here is handled strictly by scripts (PlayerConnection and EditorConnection APIs). Is there something that Unity Remote provides that you would like to have?

 

I have implemented the OpenGL part of code that retrieve Y and UV texture from pixelBuffer, I tested it with a slightly modified shader and works great.

I created a pull request here: https://bitbucket.org/Unity-Technol...support-opengl-es-2-as-the-rendering-api/diff

 

No, it just seems like it would be a nice enhancement of Unity remote. We would have 1 remote app instead of 2.

 

When would it be possible to test AR in Editor Mode?

For example some debug version of this events in Editor Mode:

UnityARSessionNativeInterface.ARAnchorAddedEvent
UnityARSessionNativeInterface.ARAnchorUpdatedEvent
UnityARSessionNativeInterface.ARAnchorRemovedEvent

plus some AR camera feedback would be nice to have too.

 

Really interesting to see you working on this. I have been thinking about localisation and how this could "easily" be achieved from a users perspective. My thoughts are to align something in the real world with a known marker, so your users can always reference that. Looks like your doing something similar but with 3 markers instead of 1.

 

Hi there! I am currently working on a demo for scanning your environment and whenever a plane is detected (that's not the floor) a cube is instantiated that extends to the floor. I did this by replacing the debugPlaneprefab to a cube that has a script attached that changes its position and scale. I got the offset height of the cube correct, but I don't know how to scale the x and z coordinates (length and width of the cube). I want the cube to match the length and width of the planeAnchor that is found so that the cube is the same size as whatever obstacle in the way. Is this possible? If so, how can I get the plane extent and have the cube rescale?

 

Hi @christophergoy this is an entire new code in which we are trying to do what you mentioned before. How would you recommend to copy the render texture into a texture2d in a good performance way?

Thanks

 

Did you not read https://forum.unity3d.com/threads/a...ity-arkit-plugin.474385/page-13#post-3153415? Doesn't that do what you're looking for?

 

When a plane is reported by the ARAnchorAddedEvent, it contains all the information in the C# equivalent of https://developer.apple.com/documentation/arkit/arplaneanchor . That has all the info you need?

 

Hi @jimmya, I updated the script package from the asset store (updated yesterday) and now I'm receiving these error from Xcode building the scene UnityARKitScene without touching anything.

"Use of undeclared identifier 'ARTrackingStateReasonInitializing'; did you mean 'UnityARTrackingState"
"Duplicate case value: 'UnityARTrackingStateReasonInitializing' and 'ARTrackingStateReasonExcessiveMotion' both equal '1'"

I build a lot of demos with the previous version and I didn't have any issue.

Thanks,

 

Upgrade to XCode Beta 3

 

Hey @jimmya — I have placed the ARKitRemote in my app. When I attempt to connect via the console I see the following error. Any pointers for where the problem may lie?

 

If you're using Unity 5.6.x, I would recommend upgrading to 2017.1 - the player connection works way better there. Make sure you've built the remote app using Development Build. After all these steps, if you still have problems, you should try force-closing the remote app, reload your Unity project (which effectively shuts down and restarts Unity), open the remote app on the device and try the connection again.

Unity ARKit Remote uses the same connection as the Profiler, so if you can make Profiler work with your iOS device, then it should be able to work for this.

 

Hello, Will the ARKit plugin work with:
Version Released
2017.2.0 Beta 4?

Cheers!

 

Yes. All versions of 2017 past 2017.1.0b5 should work.

 

Been playing around with Unity and ARKit the past couple days. Its easy to get up and running.
here is a video grab of the experiment:

 

Hey @jimmya — I believe it was the "Development Build" checkbox that fixed it for me. So now I am having another issue. Instead of seeing video within Unity and on the actual iOS build on my iPhone I am seeing pink painted everywhere. I can see video behind the pink for half a second and then it gets painted over with pink. I am on 2017.1 and I have been updating the ARKit Unity Plugin about once per day when y'all release new stuff.

See screenshot:


Thanks,
Tanner

 

Looks like the shader compile step may have failed - that color signifies a shader error. Are you using OpenGLES or Metal? You should be using Metal.

 

Hey @jimmya under Player Settings > Other Settings > Graphic APIs it shows "Metal" by itself. Is that where I should be looking? See image...

 

The graphics api looks correct, but you should probably have the experimental graphics jobs turned off. BTW, are you trying with the test scene that we've included? That might be a good first test to make sure there are no other conflicts.

 

Hey @jimmya — this started occurring a few days ago before adding in the AR Remote app to the workflow. Also I tried turning Experimental Graphics off and that did not solve the issue. Do you have any additional suggestions?

 

Do you see any errors in the console related to this?

 

While it would of course be great if ARkit had something like this (vision looks way to complicated for me honestly) that could be extended to unity I was just thinking for now it would be nice if ARkit first of all didn't overrule Vuforia (might be a mistake on their part) and second if their was support for them working together (like Vuforia have with the Hololens).

So a "Vuforia world" could live inside the ARkit world without messing with the ARkit's overall tracking.
If you skim a few lines in this tutorial on integrating Vuforia with Hololens you'll get what I mean. Their implementation is very dev friendly: (read from Configuring a Vuforia App for HoloLens)
https://library.vuforia.com/articles/Training/Developing-Vuforia-Apps-for-HoloLens

 

Hi @christophergoy — I figured out what was causing the issue. I was trying to use a Trail Renderer on an object.

 

Ok so I have a iPhone 7 updated to iOS 11, Xcode 9 beta 3 and Unity 5.6.2p2 - After trying to load my Arkit Scene it failed so I tried out the example scene and as i opened the folder on Xcode - Xcode crashed. Whats up with this?

P.S These are the problems i was experiencing with my own ArKit Scene

 

I have also run into those errors in the past
1. The only way i got around this was to switch to my master profile in IOS
2. You need to change the name of the project to something other than com.unity - change it to com.myproject.something
3.You need to specify ios 11

You can also configure these under player settings before building in unity

 

Does the Camera in ARkit have world location ?

Like If I have two apps running the same app , and I get the position and rotation of the camreas using the camera.Transform.LocalPosition , Unity co-ordinates, or are they relative to the Latitude and longtude in the real world or is it or are all camera at zero ?

 

Still running into these two errors even in the latest unity arkit plugin when building in xcode

The namespace is incorrect and the other seems to be duplicate. So I just changed the namespace to UnityART... and comment out the other one.

switch (trackingReason)

{

case ARTrackingStateReasonNone:

return UnityARTrackingStateReasonNone;

case ARTrackingStateReasonInitializing:

return UnityARTrackingStateReasonInitializing;

case ARTrackingStateReasonExcessiveMotion:

return UnityARTrackingStateReasonExcessiveMotion;

case ARTrackingStateReasonInsufficientFeatures:

return UnityARTrackingStateReasonInsufficientFeatures;

default:

[NSException raise"UnrecognizedARTrackingStateReason" format"Unrecognized ARTrackingStateReason: %ld", (long)trackingReason];

break;

 

May i ask, (Sorry if I sound like a complete noob in advance, came from Android) how do I switch to my master profile in iOS? Also my project is already set up like that with "com.SkaZonic.arkitscene" as the bundle identifier. I guess I specify iOS 11 with my build settings set at a minimum of iOS 11 what else can I do to specify iOS 11??
So these are the two problems I get - (they're signing problems).

 

So for some reason personal team is not working i discovered. You need to actually have a regular apple account setup. When I tried to use personal team it didnt work. I could only do it under my company profile. I also got that error with maximum never registered unitll i switched the the profile to my company. This could be a bug in xcode beta.

 

Get latest of the plugin code, and install the latest XCode Beta 3

 

Whenever you start an ARKit session on a device, the camera on that device will be at (0,0,0)

 

This is really an XCode issue, so Apple Developer Support might know better, but I'll try and answer from what I think I know. The error message seems to indicate that you're trying to build to a device that you have not built to before, but your free personal license only allows you to build to 3 devices. There is no automated way to remove devices from your personal team, so your solutions seem to be: 1) build it to a device you have built to before 2) ask Apple support to remove old devices from your account 3) create a new apple developer account and use that one

 

Thanks solved the issue! Just made a new account. Again thank you Unity team for this awesome plugin!

 

Does anyone know how I would implement a sort of waiting screen for the plane to be detected and then tell the user it is ok to place an object? Also Has Someone said this already?

 

Hi all,
After updating to the latest ARKit plugin, I've begun getting an error on play in the editor whenever my mouse is anywhere inside the game window:

Screen position out of view frustum (screen pos 715.000000, 511.000000) (Camera rect 0 0 1325 520)
UnityEngine.SendMouseEvents: DoSendMouseEvents(Int32)

Can anyone help with this? I'm getting this error anytime the Unity AR Camera Manager component is enabled. Which is pretty vital for things to work.

Thanks!

 

Hey I'm having the same issues. I'm using Unity 2017.1, and I have updated the ARKit.

Assertion failed: Screen position out of view frustum (screen pos 676.374695, 141.260361) (Camera rect 0 0 701 162)
UnityEngine.SendMouseEventsoSendMouseEvents(Int32)

 

This usually happens if your camera does not get initialized to some good value. I've got some lines at the top of ARKitRemoteConnection that is supposed to remedy this:

Code (CSharp):

1.             //put some defaults so that it doesnt complain
2.             UnityARCamera scamera = new UnityARCamera ();
3.             scamera.worldTransform = new UnityARMatrix4x4 (new Vector4 (1, 0, 0, 0), new Vector4 (0, 1, 0, 0), new Vector4 (0, 0, 1, 0), new Vector4 (0, 0, 0, 1));
4.             Matrix4x4 projMat = Matrix4x4.Perspective (60.0f, 1.33f, 0.1f, 30.0f);
5.             scamera.projectionMatrix = new UnityARMatrix4x4 (projMat.GetColumn(0),projMat.GetColumn(1),projMat.GetColumn(2),projMat.GetColumn(3));
6.  

But I am not really sure how you got your project in such a state - the scenes in my project do not have that problem.

 

I'm having a small problem with ARKit. I'm working on an app that will let me place objects down on a surface and then change those objects with UI buttons. But the only way I can do that is to make the HitCubeParent a child of an object controller.
Right now, I think it's breaking ARKit's functionality. I can view the objects, change them, and resize them, but I can't place them nor can I walk around them.
So I guess my question is does parenting the HitCubeParent kill the functionality?

 

Doh! I realized why - I didn't account for scenes that do not have ARKitRemoteConnection gameobject. I'll fix it, but in the meantime, you could add that gameobject to your scene to make it work.

 

I've confirmed that my far clipping plane is set to 1000. I haven't measured exactly where the object clips on device, but I'm pretty sure its 30 meters still. Have you had a chance to test this, does it work for you?