NatDevice - Media Device API

Hi!
How does "NatCamScaler" work?
I attach it as component to the RawImage, but it does not affect it in any way...

 

Hi,
Implemented this plugin, what a time saver! Only bumped into a major bug today.
We need the user to be able to tap on the screen to set the light exposure bias (so tapping a light area will make the camera image darker, and vice versa).

Tried to set this, with the tought it would magically enable a "tap on screen to set exposure automatically according to tapped area), but no luck:

Code (CSharp):

1. NatCam.Camera.ExposureMode = ExposureMode.Locked;

So I tried to implement my own implementation for this:

Code (CSharp):

1. void Update()
2. {
3. if (Input.GetMouseButtonDown(0))
4.             {
5.                 _lastMousePos = Input.mousePosition;
6.  
7.                 NatCam.Camera.ExposureBias = _initExposureBias;
8.                 NatCam.CapturePhoto(OnCapturedPhoto);
9.             }
10. }
11.  
12. void OnCapturedPhoto(Texture2D photo)
13.         {
14.             Color c = photo.GetPixel((int)(_lastMousePos.x / Screen.width * photo.width), (int)(_lastMousePos.y / Screen.height * photo.height));
15.             NatCam.Camera.ExposureBias = (1 - ((c.r + c.g + c.b) / 3.0f)).Remap(0, 1, NatCam.Camera.MinExposureBias, NatCam.Camera.MaxExposureBias);
16.         }

This causes a crash of the app after tapping a few times the screen.
What causes this crash?
Or does NatCam already have a "tap on area to adjust light exposure based on tapped pixels" feature, and how to use it?

 

Hello. I am starting to work with NatCam. I already have a WebcamTexture that is being manipulated by other classes. I only want to control the camera zoom. I cannot find examples for this. I am tweaking NatCamU.Core.DeviceCamera.FrontCamera.ZoomRatio to do this and nothing happens. Can I do it? How do I do it? Thanks.

 

Yes, this is what MiniCam does. The NatCamView component in conjunction with the NatCamView2D shader do this.

No, you would have to rotate it using Utilities.RotateImage which takes time (moving around millions of pixels), but can be done concurrently using IDispatches so as not to hang your application.

We have just completed a unified orientation pipeline for the 1.5b4 beta. Now the orientation is always correct, and can be fed into a new NatCamPreview component (this component is the fusion of NatCamView's orientation and NatCamScaler's scaling functionality) and the Utilities.RotateImage API. Email me if you'd like to have the beta.

 

We have fixed this in the new beta, 1.5b4. Please email me your invoice number and I will send it to you.

 

It works on a number of my test devices including the S3, S5, S7, and others. Can you send me full logs?

 

OpenCV takes a lot of CPU time because it is very expensive. This is not because of NatCam. You should consider dispatching your OpenCV operations to a worker thread to maintain a solid framerate.

 

NatCam itself does not leak memory. These 'leaks' are small, and will be cleaned up by the garbage collector. Using OpenCV however, a leak will be because allocations are not tracked. Can you send me your code?

 

All these issues have been fixed in the 1.5b4 beta. Please email me and I will send you the beta.

 

This has been fixed in 1.5b4.

This has not been reported before. Can you share a screenshot? What version of Unity are you using? Also, please email me so I can work with you closely to investigate this issue.

 

We have addressed this in 1.5b4. Now, NatCam.OnPreviewStart is called on camera and orientation changes, so you can call NatCamScaler.Scale().

This issue has been fixed. Please email me with your invoice number and I will send you the latest beta.

 

This has been fixed. Shoot me an email with your invoice number and I will send you the current beta.

 

We changed how NatCam handles the Google Play Services dependency in 1.5b3. I believe we have discussed this. To recap: NatCam will now use the Google Play Services AAR's included in the Android SDK, with the option of selecting which version you want and which dependencies to fulfill (currently using the base, basement, and vision libraries).

 

No. This happens because in portrait, the preview texture/buffer is actually 480x640.

 

I believe you have resolved this after our conversation.

 

It makes the RawImage's aspect ratio match that of the preview so as to avoid 'stretching'. In OnPreviewStart, you have to call NatCamScaler.Scale on the NatCamScaler (make a reference to it in the editor). See the MiniCam example for how it is done.

 

This is actually extremely easy. In most cases, the camera API will automatically do this for you when you focus at that point, so simply call DeviceCamera.SetFocus.

This is because of a memory leak. You need to free the photo after you use it:

Code (CSharp):

1. Texture2D.Destroy(photo);

 

Hi. You cannot use WebCamTexture and NatCam at the same time. You should use NatCam only (it will automatically use WebCamTexture on platforms where it does not have native support). We added proper zoom support in the new beta, 1.5b4, so email me your invoice number and I will send you the beta.

 

State of NatCam Address:
Hi guys. I deeply apologize for being out of touch for the past two weeks; I have been spread across a few projects that took a lot of time. Now, I have resumed full work on NatCam. The API is getting much more stable; we completed the 1.5b3 beta for the Core and Pro specs, and recently just completed the 1.5b4 beta for both specifications too. Below is the changelog:

NatCam Core 1.5b4
+ NatCam.OnPreviewStart is now called when cameras are changed and when the orientation is changed.
+ Added a unified orientation pipeline for captured photos.
+ Added NatCamPreview component for properly displaying textures on UI panels with scaling and orientation.
+ Added ability to mirror image in Utilities.RotateImage.
+ Added bitcode support on iOS.
+ Added support for zooming on a large amount of iOS and Android devices.
+ Added DeviceCamera.MaxZoomRatio.
+ Fixed crash on Android when NatCam is unable to open the camera.
+ Fixed exception on Android when NatCam.Camera getter is called before NatCam.Camera has been set.
+ Fixed bug on Android where switching cameras caused preview to freeze on older devices.
+ Fixed bug on Android where DeviceCamera.TorchMode getter always returned Off.
+ Fixed bug where preview was incorrectly oriented after NatCam.Release() was called.
+ Fixed bug on Android where preview would flip briefly when camera was switched.
+ Fixed bug on NatCamLegacy where preview did not update after Pause() was called.
+ Fixed bug on Android where setting a camera that did not exist caused a hard crash.
+ Fixed 'Camera is being used after Camera.release() was called' exception on Android.
+ Fixed build error when building Android project with NatCam and Android Ultimate Plugin.
+ Deprecated NatCamScaler component.
+ Deprecated NatCamView component.
+ Deprecated NatCamZoomer component.
+ Deprecated NatCam.HasPermissions. Use NatCam.Implementation.HasPermissions instead.
+ Deprecated DeviceCamera.IsZoomSupported. Instead, check that DeviceCamera.MaxZoomRatio > 1.
+ NatCamScaler.Scale() must now be called manually. Call it in NatCam.OnPreviewStart.

NatCam Professional 1.5b4
+ Added ReadablePreview flag on NatCamAndroid. This is a workaround for the GPU driver bug that caused lag on the S7 Edge.
+ Added bitcode support on iOS.
+ Added NatCamToMatHelper script for OpenCV/DLibFaceLandmarkDetector.
+ Fixed null reference exception when NatCam.PreviewBuffer is called on NatCamLegacy.

_________________________

NatCam Test Suite
We are also introducing the NatCam Test Suite, an open-source project that we run every pre-release build of NatCam through to test API's and catch bugs. You can find it here. Feel free to fork the project, add your own test cases, and create merge requests.

__________________________

NatCam Extended 1.5b4 is a few days away from being completed, but will feature all the functionality that did not make 1.5b3.

 

HI, My app has been using Natcam before.
I want fixed orientation at NatCam.PreviewBuffer.
return landscape buffer when portrait orientation. Is there a way?

And, I have some issue too.
1. How can I get preview resolution at OnPreviewStart?
I tried NatCam.Camera.PreviewResolution. but that return 1280*720 when portrait orientation.
2. When run app portrait, PreviewBuffer(at OnPreviewUpdate) returned value(width, height) was not right at first callback.
ex) first callback 1280*720 and next 720*1280

 

Code (CSharp):

1. var dimensions = new Vector2(NatCam.Preview.width, NatCam.Preview.height);

This only happens on iOS because of an orientation change that occurs during the first frame. You can either skip the first frame or make your pipeline handle dimension changed

This is not possible. The preview texture is where the buffer is gotten from, and the texture will be portrait in portrait mode.

 

Lanre,

Can you please provide the updated version of the documentation for PreviewMatrix at https://olokobayusuf.github.io/Members/NatCam_PreviewMatrix.html

NatCam.PreviewMatrix seems to create massive memory leaks.

 

The documentation shows the new API coming in 1.5f3. The current API which is a getter will allocate a new Mat each time it is called, so you have to make sure you cache it then release it once you are done with it.

Code (CSharp):

1. var matrix = NatCam.PreviewMatrix;
2. // Do stuff...
3. matrix.release(); // Failing to do this will result in a memory leak

You have to be careful to not lose track of the allocations you make:

Code (CSharp):

1. int width = NatCam.PreviewMatrix.cols(); // Allocation leak here because the matrix isn't retained and released
2. var matrix = NatCam.PreviewMatrix;
3. // ...
4. matrix.release();

The new API helps prevent this by allocating the matrix for you and simply updating it in subsequent calls (instead of allocating a new one):

Code (CSharp):

1. Mat matrix;
2. NatCam.PreviewMatrix(ref matrix); // `matrix` is allocated here
3. // Do stuff...
4. NatCam.PreviewMatrix(ref matrix); // `matrix` is updated here, not reallocated

 

Thank you. Maybe this is also related to 1.5f3, but the code provided only works properly when my phone is in landscape. When in portrait, the pixels are offset.

Do you have a solution to this?

 

How do you mean that they are offset? Can you share a screenshot?

 

At the moment. How can I write to the texture in Preview and see the results on the screen? Thanks.

 

This is not possible; in a technical sense, you can't write to a texture, but you can modify its contents and upload the modified texture data to the GPU. You would have to blit the NatCam.Preview texture and any other modifications to a RenderTexture and use that. Make sure you do not modify NatCam.Preview (do not call NatCam.Preview.UpdateExternalTexture or SetPixels...). Bring your modifications into your own texture and use that instead.

 

All right. Thank you very much!

 

How can I get texture or pixels from preview without wasting resources displaying the image?

 

It depends on what you want to do with it. If you want to use it on the GPU (blitting, displaying, and so on) then use the NatCam.Preview texture. If you need the preview data in system memory for a CPU operation, then you must use the NatCam.PreviewBuffer API (requires NatCam Pro). The preview buffer is kept natively, so depending on what you are looking to do, you might want to copy it into the managed layer (byte[]) using Marshal.Copy.

 

Hello
I'm taking over a project from another developer that used NatCam Professional, the project used to work fine in an earlier Android release but after upgrading to the lattest android release for our test phone the app started crashing with this call stack from logcat:

03-23 19:44:47.877 28003 28041 E GraphicBufferAllocator: FATAL: can't find the gralloc module
--------- beginning of crash
03-23 19:44:47.877 28003 28041 F libc : Fatal signal 11 (SIGSEGV), code 1, fault addr 0x40 in tid 28041 (Thread-29)
03-23 19:44:47.878 611 611 W : debuggerd: handling request: pid=28003 uid=10190 gid=10190 tid=28041
03-23 19:44:47.960 617 4076 E ANDR-PERF-MPCTL: poll() has timed out for /sys/module/msm_performance/events/cpu_hotplug
03-23 19:44:47.960 617 4076 E ANDR-PERF-MPCTL: Block on poll()
03-23 19:44:47.999 28315 28315 F DEBUG : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
03-23 19:44:48.000 28315 28315 F DEBUG : Build fingerprint: 'samsung/heroqltetmo/heroqltetmo:7.0/NRD90M/G930TUVS4BQC1:user/release-keys'
03-23 19:44:48.000 28315 28315 F DEBUG : Revision: '15'
03-23 19:44:48.000 28315 28315 F DEBUG : ABI: 'arm'
03-23 19:44:48.001 28315 28315 F DEBUG : pid: 28003, tid: 28041, name: Thread-29 >>> com.ADVR.ADVRPreviewGVR <<<
03-23 19:44:48.001 28315 28315 F DEBUG : signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x40
03-23 19:44:48.001 28315 28315 F DEBUG : r0 00000000 r1 c577e024 r2 00000780 r3 00000001
03-23 19:44:48.001 28315 28315 F DEBUG : r4 00000303 r5 00000780 r6 00000438 r7 00000438
03-23 19:44:48.002 28315 28315 F DEBUG : r8 00000303 r9 9dd5d244 sl 00000001 fp 00000000
03-23 19:44:48.002 28315 28315 F DEBUG : ip 97984e50 sp c577e010 lr 9797d067 pc 9797dec2 cpsr 200f0030
03-23 19:44:48.023 28315 28315 F DEBUG :
03-23 19:44:48.023 28315 28315 F DEBUG : backtrace:
03-23 19:44:48.024 28315 28315 F DEBUG : #00 pc 00006ec2 /system/lib/libui.so (_ZN7android22GraphicBufferAllocator5allocEjjijPPK13native_handlePj+85)
03-23 19:44:48.024 28315 28315 F DEBUG : #01 pc 00006063 /system/lib/libui.so (_ZN7android13GraphicBuffer8initSizeEjjij+54)
03-23 19:44:48.024 28315 28315 F DEBUG : #02 pc 0000600b /system/lib/libui.so (_ZN7android13GraphicBufferC2Ejjij+158)
03-23 19:44:48.024 28315 28315 F DEBUG : #03 pc 00011ef0 /data/app/com.ADVR.ADVRPreviewGVR-1/lib/arm/libNatCam.so (_Z23GenerateFastReadContextjjj+1788)
03-23 19:44:48.024 28315 28315 F DEBUG : #04 pc 006740ed /data/app/com.ADVR.ADVRPreviewGVR-1/oat/arm/base.odex (offset 0x5f1000)

The device is a Samsung Galaxy S7. model number SM-G930T. Android version 7.0
Any advice?

 

You seem to be using OpenGL ES2 on the S7, whereas it supports OpenGL ES3. In Android N, Google placed restrictions on accessing system libraries, which affects NatCam when running on OpenGL ES2. Allow OpenGL ES3 in Player Settings and you should be all set.

 

Hi guys.

Here's a minor update on the current state of NatCam. NatCam Core 1.5b4 is complete and tested, so is NatCam Professional 1.5b4. As for NatCam Extended, I have completed a FaceCam example application that demonstrates NatCam's face tracking pipeline.
We have decided to drop face landmarks in the pending release. We will consider adding this feature in a later release. I expect to finalize the changelog and submit version 1.5f3 for all specifications this week.

The next thing on the list is an all new NatCam VR demo.

 

Hi
We just bought NatCam, it is good. But we found that if no google play services installed on Android, NatCam will through a class not found exception.


03-27 18:02:05.331 13732-13745/? I/Unity: AndroidJavaException: java.lang.NoClassDefFoundError: Failed resolution of: Lcom/google/android/gms/R$string;
java.lang.NoClassDefFoundError: Failed resolution of: Lcom/google/android/gms/R$string;
at com.google.android.gms.common.internal.zzg.zzi(Unknown Source)
at com.google.android.gms.common.GooglePlayServicesUtil.zza(Unknown Source)
at com.google.android.gms.common.GooglePlayServicesUtil.zza(Unknown Source)
at com.google.android.gms.common.GooglePlayServicesUtil.showErrorNotification(Unknown Source)
at com.google.android.gms.common.GoogleApiAvailability.showErrorNotification(Unknown Source)
at com.yusufolokoba.natcamextended.NatCamExtended.SupportsMetadata(NatCamExtended.java:205)
at com.unity3d.player.UnityPlayer.nativeRender(Native Method)
at com.unity3d.player.UnityPlayer.a(Unknown Source)
at com.unity3d.player.UnityPlayer$b$1.handleMessage(Unknown Source)
at android.os.Handler.dispatchMessage(Handler.java:98)
at android.os.Looper.loop(Looper.java:158)
at com.unity3d.player.UnityPlayer$b.


Some Android mobile in China doesn't re-install google play service with them. This there anything we can do about this?

Thanks.

 

Unfortunately, there isn't. NatCam Extended requires Google Play Services on Android for any metadata detection (barcodes and faces). If you do not use any metadata detection, you should be able to still use NatCam. Find and comment out the call to NatCam.SupportsMetadata in your project.

 

Too bad, we are using NatCam to scan QR code, I think this also need metadata like barcodes and faces, right?

 

Yes, QR codes are a form of barcodes.

 

Hi Lanre,

My app is restricted to devices which have a flashlight on the Android PlayStore.
According to the Android docs ( https://developer.android.com/google/play/filters.html ), some filters are automatically added by the PlayStore depending of what the app may use.

Do you know any possibility to remove the flashlight permission from a Natcam-using app ?

 

We made the flashlight capability optional in 1.5f1. What version of NatCam are you using?

 

I think we still use the 1.3 version...

Thanks for the info ! We will update as soon as we got some time to do so

 

Looks promising but still needs work, like if the person isn't standing straight or has eyes closed it won't find the face.

Still I'll bookmark your asset and check it regularly.

 

Several factors could improve this, like bumping up the resolution. We will keep improving the API.

 

I am trying to build an archive for testing using testflight and get this error:

bitcode bundle could not be generated because '[REMOVED]Libraries/NatCam/Core/Plugins/iOS/libNatCamCore.a(INatCam.o)' was built without full bitcode. All object files and libraries for bitcode must be generated from Xcode Archive or Install build for architecture arm64

The project compiles and runs if I deploy direct to my iPhone.

I am using the Unity 5.6f1 beta build.

I have hit a dead end with this and google / forum searches have not been fruitful.

Any pointers would be appreciated.

-Thomas

 

I am using NatCam.PreviewFrame(ref frame); at initialization and each frame to get a Texture2d from the camera that I use GetPixels32 from, modify them (OpenCV stuff) and then I put those pixels in another texture that will be displayed. This is working fine in Android, but in iOS frame always return null. Am I missing something? Thanks.

 

Bitcode support is coming in 1.5f3. For now, you must build with bitcode off.

 

Can you try running the GreyCam example and let me know if it doesn't show? Also, how do you mean at initialization? You should only call NatCam.PreviewFrame in OnPreviewStart and after (OnPreviewUpdate). When running GreyCam, make sure the resolution is set to LowestResolution.

 

Hello Lanre, thank you for your outstanding work. I have a question about an application development on IOS. Is it possible to display video on mobile without being in full screen (with fixed preview dimensions)? It works on PC and Mac but not on mobile ... Sorry for my english.
Thank you

 

Yes, I do it on PreviewStart so I can use the texture information to initialize some things. Ok, it is all right if I call NatCam.PreviewFrame at the end of the OnPreviewUpdate method? I will test GreyCam and write back. Thanks.

 

Yes. Simply apply the NatCam.Preview texture to whatever panel or material you choose. Make sure you do this in the OnPreviewStart event.

 

GreyCam scene shows all right.