NatDevice - Media Device API

Should natcam face be used with opencv for unity face tracking or Dlib face tracking by the same author?

 

I also requested support for vuforia

 

RAMPAGE!

You are monster, keep up the good work

 

NatCam can track faces by itself. You don't need OpenCV or Dlib. If you want to detect landmarks however, you will have to use OpenCV (at least on iOS, because Android can detect landmarks. NatCam just doesn't support it).

 

All Vuforia needs to do is allow developers provide custom frames to their tracking engine. It has already been requested by numerous developers before, but they don't seem to have an interest in doing so.

 

I have fixed the orientation bug that prevented face detection and tracking on Android. Now I just have to consolidate other things in the face tracing pipeline and then it'll be done. Once completed, I will go ahead and release 1.3f3 with the fix and release two more NatCam Snippets episodes covering face tracking and OpenCV.

NatCam 1.3f3 will also contain the rendering pipeline optimization. I simply have to fix a hang when NatCam.Release() is called with multithreaded rendering enabled. Once this is done, the optimization will be complete.

 

I think that might be because if they did that you could be able to save 'User Defined Targets' and they dont want you to do that because then you would not need to use 'Cloud Recos' and thats where they make their money...

 

Actually, they might still have their preset targets. All we need is the ability to pass the camera frames to Vuforia ourselves. Right now, Vuforia completely handles camera frame acquisition, and it is completely isolated. If the SDK allowed it, all I'd have to do is pass NatCam.Preview (or the underlying preview data) to the Vuforia engine, like how NatCam handles OpenCV.

 

Hi, I'm really interested in NatCam. I've been using WebCamTexture for photo capture functionality. I am wondering how NatCam does with camera and orientation changes, especially with displaying the preview on a rawImage. I've found WebCamTexture to be problematic with flipping X and Y axes in the preview as you change orientations and camera, with varied results across different platforms, devices and device generations (especially Amazon Kindle Fire). I've had to write a lot of complex code to handle all these variables and was wondering if NatCam's preview handles this better?

 

Hi @detournment_19 , yes NatCam correctly handles orientation changes at runtime. This is because it orients the camera preview based on the current orientation reported by the OS (and not Unity, which is what it did pre-1.2).

I would have to say, camera apps should not have variable orientations. This goes against the unwritten rules of UX. You should instead lock the app to portrait and handle orientation changes manually by adjusting your UI.

 

Hi Lanre, Does "PhotoSaveMode.SaveToPhotos" save the image to the DCIM folder or does it just save the image to an app specific folder and just update the gallery with its location?

Thanks!

 

Hi @FanSpeedControl , PhotoSaveMode.SaveToPhotos saves to external storage (SD card) and notifies the gallery of the image. The directory is not app-specific so deleting the app doesn't also delete the image).

 

I'm trying to get editor testing to work - natcam works on device but not on Windows 10 unity 5.4.0?

 

Hi @ina . I presume the preview stays white. If so, check the facing of the camera you initialize, and try changing it (if you are using UnitygramBase or inheriting from it).

 

State of the NatCam API address:

NatCam 1.4b0 is on track for release early next week. So far, here's the feature list:




+ Greatly improved speed of NatCam and Unity on Android.
+ Added NatCamPreview component for preview scaling, focusing, zooming and panning.
+ Added ScaleMode.FillView for NatCamPreviewScaler.
+ Added timeStamp integer field to Barcode and Face.
+ Added SaveMode enum, deprecated PhotoSaveMode enum.
+ Added SaveMode.SaveToAppDocuments to save photos and videos privately.
+ Added limited support for SaveMode on Fallback Interface.
+ Deprecated NatCamPreviewScaler component.
+ Deprecated NatCamPreviewZoomer component.
+ Deprecated NatCamPreviewGestures component.
+ Deprecated NatCamPreviewBehaviour base class.
+ Deprecated ZoomMode.ZoomSpeedOverrideOnly.
+ Deprecated NatCam.RequestFace. Use NatCam.OnFaceDetect event instead.
+ Fixed Android N compatibility.
+ Fixed EXC_BAD_ACCESS crash when switching cameras on iOS.
+ Fixed preview being oriented wrongly when only one supported app orientation is selected in Xcode.
+ Fixed compiler warnings in NatCamNativeInterface.UpdatePhoto().
+ Fixed build error on Android because of targetSDKVersion.
+ Fixed OnPreviewStart/ExecuteOnPreviewStart not being invoked when camera was played after being paused.
+ Fixed OnPreviewStart/ExecuteOnPreviewStart being invoked too soon after camera was played after being paused.
+ Fixed faces not being detected on Android.
+ Fixed preview flipping momentarily when switching cameras on Android.
+ Reduced the amount of text that NatCam logs.
+ Changed Android API minimum requirement to API level 15.

We plan to include the big feature in b1: Video recording. Stay tuned, and let me know if you'd like to test the beta.

 

Hi, I bought Natcam and I started to experiment with it few days ago. I’m really satisfied about performance, it saved my framerate on Android and I decided to adopt it for my product, a mobile game that features augmented reality. But there are some issues with the stability of Natcam that I want to point out, that currently prevent the integration with my product.

This is my current setup for Natcam and Unity:

Interface: Native
Preview Type: Readable
Detect Metadata: Off
Facing: Rear
Resolution: HD
Framerate: Lowest Framerate
Focus Mode: Off
#define ALLOCATE_NEW_FRAME_TEXTURES is commented

In the player settings multithreaded rendering is disabled, minimum api is set to Android 5.0 and I’m using a Motorola Moto G 3rd generation with Android 6.0 to test builds.

The first issue, that I have solved is with OnNativePreviewUpdate event. My callback looks like this:

Code (CSharp):

1.  
2. public void calculateAmbient(ComponentBuffer bufferType, UIntPtr buffer, int width, int height, int size)
3. {
4.        unsafe
5.        {
6.               byte* ptr = (byte*)buffer.ToPointer();
7.  
8.               //only read pixel values
9.               for (int i = 0; i < size; i++)
10.               {
11.                      //do stuff
12.                        
13.                }
14.         }
15. }
16.  

When I tried to read the pixels I saw only white and blue stripes on the screen. In the first moment I thought that was my fault with pointer conversions or something, I also tried to write a native plugin instead of using unmanaged code, but the result was the same. After having fought for hours I discovered that there was something strange in the way this event was dispatched in NatCamNativeInterface, so I tried to modify these lines:

Code (CSharp):

1.  
2.  
3. if (OnNativePreviewUpdate != null) {
4.  
5.      OnNativePreviewUpdate(ComponentBuffer.RGBA32GPU, unchecked((UIntPtr)(ulong)          (long)RGBA32GPUPtr), (int)width, (int)height, (int)size);
6.  
7.       if (mReadablePreview)
8.            OnNativePreviewUpdate(ComponentBuffer.RGBA32, RGBA32Ptr, (int)width, (int)height, (int)size);
9.  
10.   }
11.  
12.  

And I replaced them with these:

Code (CSharp):

1.  
2.  
3. if (OnNativePreviewUpdate != null) {
4.  
5.      if (mReadablePreview)
6.           OnNativePreviewUpdate(ComponentBuffer.RGBA32, RGBA32Ptr, (int)width, (int)height, (int)size);
7.      else
8.          OnNativePreviewUpdate(ComponentBuffer.RGBA32GPU, unchecked((UIntPtr)(ulong) (long)RGBA32GPUPtr), (int)width, (int)height, (int)size);
9.  
10.  }
11.  
12.  

And it finally worked for me, I was capable to correctly read the values of pixels. Maybe my change is not correct, so please tell me if the previous code had a specific purpose.

The second issue is about mipmap, it was just a test, I currently don’t need mipmap and they are also pretty much expensive in real time, but I want to point out the problem anyway. If I modify on NatCamNativeInterface, the part where you create a new Texture, setting mipmap to true, I have the same result described before, White and blue strips on the screen.

The third issue is with flashlight, on my phone it doesn’t work, just running the basic example scene.

Ok, now the last and the most serious problem which make impossible for me to include Natcam in my project if this issue is not fixed as soon as possible. Basically, showing just the image from camera, as the basic example does (no callbacks OnNativePreviewUpdate), with the setup described above, when I push the home button on android and the app goes in background, it crashes almost 100% of the times. If I change the interface from native to fallback it works, I think it’s a problem in releasing the resources, I tried to fix it but I didn’t succeed.

 

That's great to hear! Thank you!

Your change is incorrect. Please revert it. The very first parameter of the NativePreviewCallback delegate is of type ComponentBuffer. NatCam will provide you with all the pointers it has. You must manually check for the appropriate pointer you want before using it. You get stripes because the first pointer you get is the texture name (GLuint) of the NatCam preview texture in OpenGL. You're lucky you didn't get a SIGSEGV. You want to do this:

Code (CSharp):

1. if (bufferType == ComponentBuffer.RGBA32) {
2. //do stuff...
3. }

The NatCam preview texture isn't meant to have mipmaps. This eats up 30% more memory, and has an effect on performance. If you need the preview texture to have mipmaps, you must create a new Texture2D with mipmaps, fill it with the preview data you get from OnNativePreviewUpdate, and call Apply() with updateMipmaps set to true.

How do you mean? Are you using NatCam.ActiveCamera.TorchMode or NatCam.ActiveCamera.FlashMode? The former sounds like what you want, while the latter only has an effect when capturing photos.

This is the first report I've gotten of this behaviour. Please get the logs from logcat (unfiltered) and email them to me. Thank you!

 

Ok I tried to revert the code and check for the buffer type and it works. Thank you! I completely ignored the importance of buffer type. From what I understood the event is fired two times with different parameters, you basically have to "catch" only the event with the type of buffer you are interesting in, to read the data.

I mean, just running the basic scene, when pressing the torch button nothing happens. I haven't done further investigation on the problem. Maybe is just the demo that is broken in some way?

I sent you an email.
Thank you for your time.

 

Yes that is correct. You have to select which one you need.

The torch button you're referring to in the demo scene is actually for setting the flash mode, not the torch mode. If you set the flash mode on and capture a picture, then flash will be used. It has no effect when running the preview because it controls the camera's flash mode (like a normal camera app).

At the very least, the icon should change. It the flash mode is auto, then an 'A' will appear next to the icon. If the flash mode is on, then the 'A' will disappear. If the flash mode is off, the flash icon will grey out. If the camera doesn't support flash modes, then the icon will not change.

Seen. Will get working on it soon.

 

Just bought NatCam and added to my project. I added the Unigram behavior to my existing camera object and made code changes. Works fine in the editor, but when building for Android API 16+ (Unity 5.3.6, OS X 10.11) it fails - couldn't covert java classes to dex format. I removed NatCam and reverted my code changes, builds fine. A separate project with just NatCam also builds fine. I re-addeed NatCam to the project but did nothing with it, and it still fails to build (no code changes, no components added in scenes or prefabs). Trying to figure out where the conflict is, and the logs are hard to figure out on this one. I'm happy to send it if that would help.

 

Paste the logs from Unity here. By the way, do you have any plugin that has google_play_services.jar anywhere in your project? This could be the reason why it fails (NatCam already comes with it, so the build will fail because of duplicate classes).

 

Yes, I have a Flurry plugin that includes it. I just removed that and now building is successful. Thank you!

 

You're welcome!

 

I'm seeing some issues with the preview on Samsung Galaxy Tab 2 and 4. I created a little demo app using the Monogram demo. On the Tab 4 the preview doesn't show up, it just becomes a blue screen behind the UI. On Tab 2 I get a preview but it looks like it sampled a 1 pixel wide column from the center and stretched it out over the whole rawImage. One thing on the Tab 4 is I can take a picture and then I get the captured picture displayed correctly. Are there known issues with these devices? Or settings I need to adjust? The Tab 2 is Android 4.2.2, the Tab 4 is 4.4.2. On other tablets such as insignia Flex Elite and a fairly new Kindle Fire the display works fine.

 

Does anyone have success or experience using this asset with ARToolkit to integrate AR into your app?

Thanks.

 

Can you send me the logs from running NatCam on these devices? I am looking into starting a whitelist and blacklist for NatCam so that developers know which devices can be problematic. Email me at olokobayusuf@gmail.com

 

I've looked at the sources of ARToolkit. It doesn't seem to support camera preview injection. This means that it won't be compatible with NatCam.

 

NatCam 1.4b0 will be undergoing a full rebuild of the Android side of things. We have received a few bug reports for different things on Android so we will be better optimizing the Android layer to work correctly and handle errors much better.

If you are currently experiencing any bugs with NatCam, please email me about it and include information about the device(s).

If you are not currently experiencing any bugs with NatCam, please email me about it and include information about the device(s).

In short, please email me about what version of NatCam you are using on what device and if there is any issue you are facing. We want to begin working on whitelisted and blacklisted devices. Now is also the perfect time to let us know about features you would like to see. Email me at olokobayusuf@gmail.com.

 

Hi @Lanre - what's the current state of face tracking feature?
Also, ARToolkit is now open source, hosted by Daqri. I'm sure we could get some support for the preview injection if we hit those forums.

 

The face tracking bug on Android has been fixed, but it won't be available to developers until 1.4b0 is available. We are skipping releasing 1.3f3 on the Asset Store to instead make 1.4b0 available to developers. I've looked at the sources of ARToolKit on GitHub. I will look into implementing a support wrapper once 1.4b0 is completed.

 

Hi guys.

I thought I'd update you all on the state of things. My work on NatCam slowed down significantly over the past couple of days because I moved cities. I am still settling down, but I'm resuming work on NatCam. NatCam 1.4b0 is on track; it has undergone significant changes and optimizations especially on Android and should be available to test by the end of this week. I will keep this thread updated as I go along.

Lanre.

 

Is it possible to test NatCam with Unity Remote 5? It is possible with WebCamTexture so I'm wondering if I can continue testing my app this way after integrating NatCam into my project.

 

Do you have a timeline estimate for ARToolkit integration? My project relies heavily on multiplatform support for QR code scanning and AR markers. So far your plugin looks like a no brainer if it can do AR as well.

 

Thanks for the update, excited to try the beta! You've been doing great work. Don't worry though, take your time.

 

I love working with NatCam and make my own AR framework with it, but i am constantly getting issues on iOS with how the camera orients itself. If a user has the device orientation locked, the camera feed looks normal as intended and works like i want it to work, but a very different scenario presents itself if the user disables the orientation lock.

My apps are locked to Landscape or Portrait and Unity will never change the orientation, but NatCam wants to do just that. It seems that NatCam changes the orientation of the camera view itself using something like Screen.orientation rather than Input.deviceOrientation(just to use two orientation reports Unity provides).

While i can implement rules that adjust the camera feed to be shown correctly, the feed will still pause briefly while it changes, delivering a not so desired user experience. My guess is that NatCam queries the device orientation natively and this will give an effect like Screen.orientation where it will be reported as changed even though the device orientation is locked.

It would be desirable if NatCam only switches the camera feed orientation if the actual device orientation changes, not if the screen orientation switches.

While i haven't tested too much on Android, i do not believe it's an issue there.


Edit:
Just went over some of the source code for iOS and always reporting the rotation to be landscape left will fix my issue in an application that is locked to landscape left. This hack will do for now, but it is an issue worth looking into.

Code (CSharp):

1. void GetOrientation (AVCaptureVideoOrientation& rotation) {
2.     rotation = AVCaptureVideoOrientationLandscapeLeft;
3. }

 

Hey guys! I'm sorry that I have been away for the past few days; I have been in the hospital. I am now resuming work on 1.4b0.

 

Hi @Simon_says NatCam doesn't work with Unity Remote. Unity Remote seems to use it's own specialized protocol, something that we are not looking into. You will have to deploy; you get full access to the device camera controls by doing this.

 

Not yet, the pressing task right now is NatCam 1.4. Once that is completed, I will look into ARToolkit integration.

 

Hi @GMM. This is a known bug in 1.3f1 and 1.3f2. It has been fixed. Download the attached NatCamExtensions.mm file and place it in Assets>NatCam>Plugins>iOS. You are correct that it doesn't affect Android.

 

Sorry to hear that. I hope you are doing better and that it was nothing too serious.

 

I'm much better now. Thank you!

 

Hi @Lanre, I wrote you an email past Friday requesting some aid with some issues we are having with NatCam. Any chance you've seen it?

I wrote extensive details about the issues, code, logs and versions I am using, but in synthesis:

• The pictures have a very long lag to capture and save the photos (total about 6 seconds);
• Every time a picture is taken, a new texture is created, and memory spikes until the app crashes (about 100MB per photo). Our application requires taking more than 30 photos in sequence;
• I am not able to use the "setExposure" function, it says it doesn't exist in "DeviceCamera" (it does find other functions such as "setFocus");
• We would like to set the camera to focus on infinity, whereas the setFocus function only allows focusing on a preset point in the vector space (or so I understand).

Any help would be very much appreciated. Thanks!

 

I was actually about to reply to your email; I have a ton of unreads from my hospitalization. I'll reply here so that other developers who might have the same questions can see the answers:

When NatCam captures a photo, it receives landscape-left-oriented JPEG bytes. From that image it must:
- Convert the JPEG byte[] to an android.graphics.Bitmap (slow for very high resolution images).
- Correctly orient this bitmap by applying the appropriate transformation matrices (which entails moving individual pixel-bytes around, also slow for high resolution images).
- Send the Bitmap to the native layer where its pixel-bytes are copied into a temporary buffer (blazing fast because memcpy).
- Send the pointer to the temporary buffer to Unity, where Unity will load the pixel-bytes into a Texture2D (also blazing fast because memcpy).
- Unity must then upload the photo data to the GPU, which means calling the notoriously slow Texture2D.Apply method (slow because it uses glTexImage2D on GLES which is slow, but the slowness is also multiplied by the fact that the photo could have a pixel density as high as 12MP).

We have tried a number of ways to speed up points 1 and 2, and the current system is the fastest. On iOS, the exact same operation is done, but iOS has ta framework specifically geared to using ARM Assembly intrinsics (optimized specifically for the iPhone CPU's) to greatly speed up the operation. Android however has no such framework, but now that you have brought this issue to my attention, I will look into implementing it manually. The workaround to this issue is requesting a lower photo resolution using DeviceCamera.SetPhotoResolution().

You will need to implement extreme memory optimizations. Do you need all photos in memory at the same time? Keeping that much uncompressed image data is sure to crash a mobile phone, it doesn't matter that it is NatCam taking the photo. I recently created a native solution to this using NatCam's iOS layer. Email me to talk about it further. A workaround will be taking photos with lower resolution as suggested above.

I presume you are using 1.3f2. Where exactly is this issue being generated from? Exposure was changed from a function call to a property, so use DeviceCamera.ExposureBias.

It sounds like you will benefit from having the Android Java sources. I will email them to you. You will have to implement the Camera.Parameters.FOCUS_MODE_INFINITY mode in the Android layer.

Sorry for the late response, I have been hospitalized since last week.

 

Hi Lanre,

Thanks for your fast reply. I hope you get better soon

About time optimisation:
Can we just store the generated JPEG bytes to a file w/o having to convert to Texture2D or do any transformation?
My application doesn't need to show the image on the screen nor does the orientation matter.

About the photo memory:
It seems that every picture is occupying memory, and its not being freed after use. We would like to know how to free up the memory used by each picture after saving it as a file. We need just one picture loaded at a time, only to store it in a file. then take the next picture.

Exposure: Solved;

We will check the java sources and get back to you with any questions.

 

You would need to modify the Android sources in order to do this. Simply send the byte array from the PictureCallback to a FileOutputStream.

You must explicitly release a Texture2D by calling MonoBehaviour.Destroy() on it. Texture2D's do not get automatically released when they are dereferenced. Since it sounds like you only need one texture in memory at a time, you can comment the ALLOCATE_NEW_PHOTO_TEXTURES macro at the top of NatCamNativeInterface.cs. This will allow NatCam reuse an already-allocated Texture2D instead of allocating a new one each time. Doing this means that you don't have to call Destroy().

Great!

 

Hi Lanre,

I love to use the Natcam to create applications. However, I am facing a problem of using the Natcam.
I used the example project to test the app, it work well when I capture the image by using the front camera. However, when I take the photo by using the back camera, it crashed.

Can you give me a helping hand to solve the problem, please?

P.S. I am using Samsung S7, the Andriod version is 5.1.1

 

Hi @Wongyan it sounds like there isn't enough memory. Send me the full logs from logcat. My email address is olokobayusuf@gmail.com. You can try reducing the photo resolution of the camera by calling:

Code (CSharp):

1. DeviceCamera.SetPhotoResolution(...)

 

Hi Larne,

Finally convinced my boss to invest in NatCam... So happy to be getting decent images now!

... so here I am again!

Just one small question you might have an answer to:
When I do NatCam.Release(); my Android app "closes/restarts"...
Any Idea what could be happening here?

(or... if I want to stop taking pictures, should I release NatCam or is Pausing good enough?)

Cheers,

Wim

 

I'll need to see the logs from the device. We've addressed the app crashing when Release was called for several reasons over several versions of NatCam. Please email me the logs from logcat (unfiltered) at olokobayusuf@gmail.com.

Ideally, you shouldn't call Release until you close the app, and even then, NatCam automatically calls Relelase. Release is not like WebCamTexture.Stop, it tears down NatCam's native resources and the next time NatCam needs to be used, Initialize must be called. Call Pause if you only need to suspend NatCam and will use NatCam again during runtime.

Irrespective of the fact above, there's still a crash on our hands that needs to be fixed, so let me have a look at the logs. Thank you!

 

Hi guys.

I'd like to announce a significant change in NatCam 1.4: photos will not be corrected for app orientation. This is because of how difficult it is to attain uniform results across devices *ahem Android* and because of how computationally expensive the process is. It becomes extremely inefficient if you need to capture multiple photos and combine them because you could orient the final result instead of wasting time on each photo. This change also means something important: significantly faster photo captures especially on Android.

Of course, the issue with doing this is that the photos would not be correctly oriented depending on the app's orientation. Captured photos will always be in the landscape left orientation (the 'natural' orientation of mobile cameras). Here are some scenarios where this could be an issue , and how we will advise you to address them:

- I need to display the photo to the user
Instead of correcting the image by manipulating pixel-bytes (which is very slow), consider using a GPU shader to manipulate the display element's UV's. This way, you get free re-orientation. Here's a very helpful thread on doing this.

- I need to save the image to the disk
NatCam will automatically save the photo with an appropriate EXIF orientation tag if you use SaveMode.SaveToPhotoGallery or SaveMode.SaveToPhotoAlbum (these enums are new in 1.4, so don't worry if they don't look familiar). Using SaveMode.SaveToAppDocuments however will NOT orient the image or save an EXIF tag (because Unity doesn't honor EXIF data). You will have to do what is suggested below if you need to save an orientation-correct photo.

- For some other reason, I need to correctly orient the image itself
You have one of two options depending on the synchrony of your operation.

Async:
Use a RenderTexture and a correction shader then blit the photo and perform Texture2D.ReadPixels on another texture that will be correctly oriented. The reason why I say this is async is because on most platforms, you will have to wait to perform this operation in OnPostRender. So use this in conjunction with a callback so that you can propagate the resulting texture.

Sync:
Use Array.Copy, Array.Reverse, and index manipulation to transpose and flip the texture's GetPixels32() into a resulting pixel buffer that is correctly oriented, then load the buffer into another texture.
​

Let me know if you have questions about any of this. Thank you.