NatDevice - Media Device API

Then NatCam.PreviewFrame should not make the frame texture remain null. Check that the function returns true.

 

Is it posible with NatCam professional to extract all the frames from a face-video taken from a mobile device?

 

What exactly do you mean by face-video? NatCam Pro gives you access to a preview buffer that is updated with the pixel data of the camera preview on every frame.

 

Hi Lanre,
I don't know if I'm the only one facing this issue :
I'm using natCam 1.5b3 and on iPhone 5c/Unity 5.6.0f3 with your miniCam exemple.

1/ When I'm using the front camera and taking a photo, the preview appears flipped Horizontally.
Everything is fine on Android.

2/ Concerning the rear camera on Iphone in portrait mode
After doing :

Code (CSharp):

1.  
2. NatCam.Pause(); //<= causes the flip bug
3. //Back to my photo gallerie
4. NatCam.Play(); //Run NatCam again
5.  

Then the preview appears directly flipped like that:

(everything is ok with android)

I think this is related with this bug :
https://issuetracker.unity3d.com/is...sometimes-results-in-corrupted-view-on-ios-10

Is there anything to do ?

Best regards,
Seb

 

I got a couple of questions:
When recording video from the camera with Android why does the background flashes?
Why is it necessary to run Natcam.play() in order to record?

Thanks in advance

 

I'm looking into this. Thanks for bringing it to my attention.

This was fixed in 1.5b4. I sent you the beta.

 

This is a bug that has been fixed. The workaround is turning on multithreaded rendering in Player Settings.

The camera preview has to start running before a video can be recorded from it.

 

how to NatCam Initialize? i found Natcam.release() but i can't found Natcam.Initialize()
plz help me;

 

Can not adjust shutter speed in Natcam?

 

Hi, is there a way to test your plugin before buying it? We would like to implement it in our Android app, currently the WebcamTexture from Unity have poor performance and sometimes make our app crash.

 

show wrong image when set orientation to auto rotation

 

Hi Lanre, thanks for the latest beta, I can work around the rotated photo issue now using Orientation.Mirror. However the videos I record in landscape still show up as portrait. Any suggestions?

 

Hi Guys,
On the lastest beta there is an issue with auto-rotation when using NatCam.Pause();
Lanre is working on it. I hope he could fix it soon.
Seb

 

Hi. NatCam.Initialize has been deprecated. Simply call NatCam.Play when you want to start the preview. NatCam will initialize itself however it needs to.

 

No. NatCam only provides a way of setting exposure bias (EV).

 

Because of the nature of NatCam (all code as it is an API), there is no trial version. There are APK's though, like the MiniCam example app.

 

We only added autorotation support in the 1.5b3 beta (after 1.5f2, the current release version). 1.5f3 will be out in a few days.

 

Hey, you're welcome. I'm somewhat confused as to how videos you record in landscape show up as portrait. To record landscape videos, your app must be in landscape orientation. Physically rotating the device to landscape will not do.

 

I am working on it now. It will be done today or tomorrow.

 

My app is in landscape mode, but in any mode the recorded video looks like I rotated the device to portrait when recording.

 

Please email me the full logs from logcat. Make sure NatCam's verbose mode is on.

 

this is in iOS

 

I see. Send me the logs from Xcode then.

 

Hi @Lanre I`m currently using the lastest beta (1.5b4) and found some problems when working with openCV:
- Just building and playing the "VisionCam" demo in an Android device in Portrait Mode makes the preview really slow (almost never updating). Is there a workaround for this? (I logged in the device using an Console.Log utility similar to unity one and didn`t receive any error message). In the Landscape device orientation the Preview frame rate is perfect.
- I`m going to use OpenCV to detect the face landmarks for an AR interaction. To do this I will use the NatCam.PreviewMatrix(ref matrix) function. To keep the CPU usage ok, the input resolution (preview Resolution) will need to be low resolution, so here is my question: Is it possible to have the preview at higher resolution to show in the screen (without modifying anything in it with OpenCV) and internally when getting the OpenCVMat to do calculations resizing it to a lower resolution?

Thanks
Ulisses

 

This sounds very interesting, especially because it could relate to the lag that is present on the S7 Edge. What device are you using? Please email me and let us continue to look into this issue.

Although the NatCam.Preview texture is handled separately from the preview data pipeline, they share the same dimensions (for the most part). You would have to resize the preview matrix after calling NatCam.PreviewMatrix using Imgproc::resize.

 

Hi @Lanre I'm found version 1.5f2 crashed on huawei android mobile, can you send me the last beta version, my email is: losetear@gmail.com

 

Can you email me the crash logs from the device? I need to know if the crash has been reported before/fixed.

 

Hi Lanre,

I've finally managed to get the photo save to work perfectly on an iOS device. What I mean by work perfectly, is for me, when each saved photo is oriented like a normal "upright" photo; regardless of the orientation of the device when taken. Basically like the Photo App on an iPhone. Anyway I used the accelerometer as you suggested, and everything is great. But, I now just have 3 questions:


1. It seems that saving a mirrored captured photo takes much longer than one that is not. Do you have any idea why this would be; just off hand?

2. What is the best way (speed being top priority) for me to flip the mirrored photos taken when using the rear device camera on iOS. I believe Rattlesnake listed some flip code on page 18. Is that something I should look at?

3. Doing more research on this and it seems that the iOS Photo app save's a "mirrored" photo (to the Photo Gallery) for any pic taken with the selfie camera (DeviceCamera.Front in my case), and non-mirrored save with the rear-camera. This is the reverse of NatCam's save (or at least my implementation of it) where all of the photo's saved with the Rear camera (not the selfie cam) are "mirrored" when I look in the Photo Gallery and the front camera (selfie) photos taken from my app are saved non-mirrored, which is incorrect for my case because I'm trying to follow what Apple does with Photo App. So...

What should I do here? Flip all photos rear and front camera, since they seem to be behaving the reverse of what I want?

Thanks!

BTW, code below works for Portrait App (non-autorotation) to save like Apple Photo App (minus the mirrored stuff above which I'm still working out):

Code (CSharp):

1.  
2.     public void UsePhoto(NatCamU.Extended.SaveCallback callback)
3.     {
4.  
5.         // Validatate photo to be saved
6.  
7.         if (_photo == null)
8.             return;
9.  
10.  
11.         // Save to device - APP is a Portrait only (no auto-rotation to other orientations)
12.  
13.         if (_capturedScreenOrientation == ScreenOrientation.LandscapeLeft) {
14.        
15.             if (_capturedDeviceCamera == DeviceCamera.FrontCamera) {
16.  
17.                 // normal
18.  
19.                 NatCam.SavePhoto (_photo, NatCamU.Extended.SaveMode.SaveToPhotoAlbum, Orientation.Rotation_0, callback);
20.  
21.             } else {
22.  
23.                 // mirrored
24.  
25.                 NatCam.SavePhoto (_photo, NatCamU.Extended.SaveMode.SaveToPhotoAlbum, Orientation.Rotation_180, callback);
26.             }
27.  
28.         } else if (_capturedScreenOrientation == ScreenOrientation.LandscapeRight) {
29.    
30.             if (_capturedDeviceCamera == DeviceCamera.FrontCamera) {
31.        
32.                 // normal
33.  
34.                 NatCam.SavePhoto (_photo, NatCamU.Extended.SaveMode.SaveToPhotoAlbum, Orientation.Rotation_180, callback);
35.             } else {
36.  
37.                 // mirrored
38.  
39.                 NatCam.SavePhoto (_photo, NatCamU.Extended.SaveMode.SaveToPhotoAlbum, Orientation.Rotation_0, callback);
40.             }
41.    
42.         } else if (_capturedScreenOrientation == ScreenOrientation.PortraitUpsideDown) {
43.    
44.             NatCam.SavePhoto (_photo, NatCamU.Extended.SaveMode.SaveToPhotoAlbum, Orientation.Rotation_270, callback);
45.    
46.         } else {
47.  
48.             NatCam.SavePhoto (_photo, NatCamU.Extended.SaveMode.SaveToPhotoAlbum, Orientation.Rotation_90, callback);
49.         }
50.  
51.         // Switch to AR camera
52.  
53.         _natCam.RestartPreview ();
54.     }
55.  

Edit: Reformat post and added 2nd question.

 

Hi Lanre,
I'm trying to build an Archive from Xcode for the first time.
I have this folowing error :
Bitcode could not be generated because "myNatCamProject" was built without fullbitcode. All Object files and libraries must be generated from Xcode Archive or install build for architecture arm64

I'm not very familiar with XCode but I understand that i have to disable bitcode right ?

 

Hi

I am building an augmented reality app where virtual objects must have a correct scale, I mean if a virtual object (in my Unity scene) is 1meter wide, it must appear the same size as a one-meter-wide real object that would appear on the video feed.
Do I simply need to give the scene camera the same field of view as reported by Natcam.DeviceCamera.HorizontalFieldOfView or VerticalFieldOfView depending on the device orientation?
Is the preview aspect always the same as the screen aspect?

Looking forward to the auto-rotation fix release also...

Thanks for the great support.

 

@Lanre When running the examples up in a fresh project the camera screen just flickers all the time. We have tried this on a Google Pixel XL and a Samsung Tab SM-T810. Any idea how to fix this? We have a critical project that we need to use this in.

 

Hello,
on iOS 10.3, my application is crashing when I switch from front to back camera.
The console give me this error:
malloc: *** error for object 0x1158f7f1: pointer being freed was not allocated

*** set a breakpoint in malloc_error_break to debug

(lldb)

I have test devices with iOS 10.1 and 10.2 and it's still working fine.
What could have change in iOS 10.3?
Thank you! Great asset!

 

@Lanre Any chance I could get the fix for this issue today?

 

Hello mimminito,

I understand that this bug only happens in Android and goes away by enabling multitrhead rendering in player settings.

 

Thank you for the tip, let me give it a go.

 

That might be something specific to how the native libraries handle it.

Absolute fastest is to use Graphics.Blit with a shader that does the flipping, then performing a readback using Texture2D.ReadPixels. I should note that this is an asynchronous operation. Without going through the complication of going through the GPU, you can use Array.Reverse on each row of the image. NatCam 1.5f3 is coming with a Utilities.RotateImage API that does this. The new version should be out early next week if you can wait (waiting for Asset Store review).

Actually, the natural camera is flipped for the rear camera and isn't for the front. This is what NatCam is consistent with.

A lot of this will be solved in NatCam 1.5f3. We introduced an integrated orientation pipeline in the Core spec. Now when you capture a photo, NatCam will also give you an orientation for that photo. You can then send this orientation to the NatCamPreview for displaying properly, to Utilities.RotateImage to physically rotate the image, and to NatCam.SavePhoto to save the photo correctly.

 

Yes, for now. NatCam 1.5f3 is coming with full bitcode support.

 

I would think so, but I am no expert in the mathematics of perspective projections and transformations. You would need to consult a computer vision expert.

Not at all. The preview aspect is that of the camera preview, whereas the screen is of the device screen.

 

NatCam 1.5f3 is coming with the full fix to this. The workaround is enabling multithreaded rendering in Player Settings.

 

Hi. Can you share a screenshot of the call stack when this error occurs? The call stack is located on the left of the screen and has a list of threads and routines that were running when the crash occurred.

 

Hi guys.

NatCam 1.5f3 is on its way to you! Here is the final changelog:

Core 1.5f3:

• NatCam now supports autorotation.
• Made preview rendering faster on Android.
• Made PhotoCallback return orientation information.
• NatCam.OnStart is now called when cameras are changed and when the orientation is changed.
• Added a unified orientation pipeline for captured photos.
• Added NatCamPreview component for properly displaying textures on UI panels with scaling and orientation.
• Added NatCamFocuser component for detecting focus gestures.
• Added Utilities.RotateImage(...) for rotating an image in system memory.
• Added bitcode support on iOS.
• Added support for zooming on a large amount of iOS and Android devices.
• Added DeviceCamera.MaxZoomRatio.
• Added tap-to-focus functionality to MiniCam example using NatCamFocuser.
• Added DeviceCamera.Framerate getter.
• Added Orientation struct for photo orientation.
• Added NatCamTransform2D shader for GPU-accelerated image transformation.
• Added ScaleMode.Letterbox for letterbox scaling.
• Added NatCam.Implementation.HasPermissions boolean.
• Refactored NatCam.OnPreviewStart, INatCam.OnPreviewStart, and NatCamBehaviour.OnPreviewStart to OnStart.
• Refactored NatCam.OnPreviewUpdate, INatCam.OnPreviewUpdate, and NatCamBehaviour.OnPreviewUpdate to OnFrame.
• Fixed bug where screen flickered when running on Android without multithreaded rendering.
• Fixed bug on Android where calling Play() after Release() or pausing and suspending the app does not work.
• Fixed bug on Android where camera does not pause when Pause() is called.
• Fixed crash when NatCam.Release() is called on Android.
• Fixed FocusMode.MacroFocus not working on Android.
• Fixed DeviceCamera.FocusMode getter causing an error.
• Fixed bug where DeviceCamera.SetFocus(..) ignored focus mode completely.
• Fixed bug where OnStart is not called when preview is resumed on Android.
• Fixed bug where setting high framerates rarely worked on iOS.
• Fixed bug where preview stretched when using NatCamScaler and autorotation.
• Fixed resource leak when NatCam.Release() is called after NatCam.Pause() on NatCamLegacy.
• Fixed memory leak when taking photos with the MiniCam example.
• Fixed potential namespace conflicts.
• Fixed resolution set for one camera being used by another on NatCamLegacy.
• Fixed IL2CPP build support on Android.
• Fixed crash on Android when NatCam is unable to open the camera.
• Fixed rare crash on Android when app lost focus with multithreaded rendering disabled.
• Fixed exception on Android when NatCam.Camera getter is called before NatCam.Camera has been set.
• Fixed DeviceCamera.SetFocus focus point being inverted on the Y axis.
• Fixed bug on Android where switching cameras caused preview to freeze on older devices.
• Fixed bug on Android where DeviceCamera.TorchMode getter always returned Off.
• Fixed bug where preview was incorrectly oriented after NatCam.Release() was called.
• Fixed bug on Android where preview would flip briefly when camera was switched.
• Fixed bug on NatCamLegacy where preview did not update after Pause() was called.
• Fixed bug on Android where setting a camera that did not exist caused a hard crash.
• Fixed bug on iOS where suspending the app caused system sounds to be distorted.
• Fixed bug on iOS where pausing the preview caused the orientation to be incorrect.
• Fixed bug where IDispatch did not invoke any delegates when one of multiple cameras in the scene is disabled.
• Fixed 'Camera is being used after Camera.release() was called' exception on Android.
• Fixed bug on Android where z-sorting was affected by the preview.
• Fixed build error when building Android project with NatCam and Android Ultimate Plugin.
• Deprecated INatCamMobile.HasPermissions.
• Deprecated INatCamMobile interface.
• Deprecated NatCamTransformation2D shader. Use NatCamTransform2D instead.
• Deprecated DeviceCamera.IsZoomSupported. Instead, check that DeviceCamera.MaxZoomRatio > 1.
• Renamed Minigram example to MiniCam.
• Renamed StartingOff example to PlainCam.
• Refactored INatCamDispatch to IDispatch.
• Refactored NatCamUtilities to Utilities.
• Refactored ScaleMode.FixedHeightVariableWidth to ScaleMode.AdjustWidth.
• Refactored ScaleMode.FixedWidthVariableHeight to ScaleMode.AdjustHeight.
• Made NatCamLegacy call OnPreviewStart only when Preview is not 16x16 (usually encountered on macOS).
• IDispatch has been further modularized.
• IDispatch will no more invoke a delegate more than once per Update.
• Implemented IDisposable interface for IDispatch.

Extended 1.5f3:

• NatCam Extended now features experimental support for Text Detection on Android.
• Greatly improved metadata detection speed on Android.
• Enhanced support for using libraries that depend on Google Play Services by avoiding clashes.
• NatCam.SavePhoto can now save photos with orientation on all platforms!
• Added IMetadata interface with rect and timestamp fields.
• Added MetadataCallback callback for use with IMetadata.
• Added NatCam.DetectBarcode (MetadataCallback).
• Added NatCam.DetectFace(MetadataCallback).
• Added overloads for equality, Equals() and GetHashCode() operations to Face struct.
• Added NatCamiOS.UseCoreImageMetadataBackend flag, allowing for more precise face tracking.
• Added bitcode support on iOS.
• Added FaceCam example showcasing face tracking.
• Added FaceCam.DrawMetadataRect function for drawing metadata positioning and scaling metadata rect on UI panel.
• Fixed crash after switching cameras while detecting metadata on Android.
• Fixed regression where metadata was not detected after the preview was resumed.
• Fixed bug where metadata was not detected after switching cameras.
• Fixed crash when using NatCam.SavePhoto on iOS 10+ due to missing NSPhotoLibraryUsageDescription.
• Fixed metadata position being mirrored on the X axis when using front camera.
• Fixed bug in NatCam.SavePhoto where photos are not saved if no callback is supplied.
• Deprecated Faces example.
• Deprecated NatCamPreview component. Use NatCamU.Core.UI classes instead.
• Deprectaed NatCam.RequestBarcode.
• Deprecated NatCam.OnBarcodeDetect and NatCam.OnFaceDetect.
• Deprecated BarcodeCallback and FaceCallback.
• Deprecated BarcodeRequest struct.
• Deprecated SaveOrientation struct. Use NatCamU.Core.Orientation instead.
• Renamed Scanner example to QRCam.

Professional 1.5f3:

• Added NatCam.PreviewMatrix(..) to greatly improve memory efficiency (so as not to allocate each time it is called).
• Added NatCam.PreviewFrame(..) to greatly improve memory efficiency (so as not to allocate each time it is called).
• Added VisionCam example to demonstrate using NatCam with OpenCVForUnity.
• Added ReadablePreview flag on NatCamAndroid. This is a workaround for the GPU driver bug that caused lag on the S7 Edge.
• Added bitcode support on iOS.
• Added NatCamToMatHelper script for OpenCV/DLibFaceLandmarkDetector.
• Exposed bitrates for configuring video encoders.
• Fixed crash when NatCam.StopRecording is called on Android.
• Fixed crash when NatCam.StopRecording is called on iOS.
• Fixed NatCam.PreviewMatrix having incorrect dimensions.
• Fixed null reference exception when NatCam.PreviewBuffer is called on NatCamLegacy.
• Fixed incorrect colors when using VisionCam example on iOS.
• Deprecated NatCam.PreviewMatrix property.
• Deprecated NatCam.PreviewFrame property.
• Deprecated Utilities.SaveVideoToGallery. Use NatCam.Implementation.SaveVideo(string, SaveMode) instead.

__________________________________________________

What's next?

• MoodCam VR App. This is a small VR application that showcases how to use NatCam for VR applications. This will be built for the Google Cardboard VR platform.
• NatCam 2.0. As we currently have it planned, the NatCam 2.x release cycle will bring no new features, but will bring new platforms. In NatCam 2.0, we will be adding native backends for the Windows and macOS platforms.

 

I'm currently having issues that apparently unity does not read the EXIF info of the
photo so the photos appear in wrong orientation.

Does NatCam allow the following:

1. Take a photo and display on texture2d in the correct orientation on both iOS and
Android, specifically the portrait issue where the photo is rotated 90 degrees in
wrong orientation?

2. Allow photo selection from users gallery and applies the same logic as point 1?

3. Provides me with the information of the taken or selected photo that I can
determine the photos correct orientation?

I have tried a number of assets that all fail the above and am struggling to find a
fix for this unity weirdness of photo orientation issues

 

Yes, NatCam supports this. NatCam returns a photo texture (not oriented) and the orientation that should be used to properly display the photo. It also comes with utilities to properly display the photo or to physically rotate the image using the orientation given in the photo callback.

NatCam is not a gallery photo picker.

It provides an Orientation flag enum for the captured photo.

 

Ah, thats a shame as I need the same functionality when choosing photos from the gallery as well. I'm surprised that something as simple as loading a photo is so problematic under Unity? If it just reference the orientation info life would be so much simpler ...

 

NatCam is a camera API, which is why it only exposes functionality that centers around using the camera. There might be assets on the Asset Store that provide what you need. If not, you can create a native plugin with the exact functionality you want.

 

Thanks, alas I do not know how to create native plugins for either iOS or Android...

 

Hi Lanre !
Youpi the new version is available ! I could adapt it to my project on android and it works great.
Tomorrow I will try with ios

Is there an a way to mirror the photo witin natCam?

Do you plan to add later a zoom fonction ?

Regards,
Seb

 

Hi @Lanre Thanks for posting the new changelog. When using Orientation like:

Code (CSharp):

1. NatCam.SavePhoto(photo, SaveMode.SaveToPhotoGallery, Orientation.Rotation_180, OnSave);

I have encountered the following error:

Code (csharp):

1. Internal_GetWidth can only be called from the main thread.
2. Constructors and field initializers will be executed from the loading thread when loading a scene.
3. Don't use this function in the constructor or field initializers, instead move initialization code to the Awake or Start function.
4. UnityEngine.Texture:get_width()
5. NatCamU.Core.Utilities.<RotateImage>c__AnonStorey0:<>m__1() (at Assets/NatCam/Core/Plugins/Managed/Utilities/NatCamUtilities.cs:99)
6. NatCamU.Dispatch.IDispatch:<Update>m__0(Action) (at Assets/NatCam/Core/Plugins/Managed/Dispatch/IDispatch.cs:65)
7. System.Collections.Generic.List`1:ForEach(Action`1)
8. NatCamU.Dispatch.IDispatch:Update() (at Assets/NatCam/Core/Plugins/Managed/Dispatch/IDispatch.cs:65)
9. NatCamU.Dispatch.ConcurrentDispatch:Update() (at Assets/NatCam/Core/Plugins/Managed/Dispatch/ConcurrentDispatch.cs:58)

There is no error when not using rotation

Code (CSharp):

1. NatCam.SavePhoto(photo, SaveMode.SaveToPhotoGallery, Orientation.Rotation_0, OnSave);

UPDATE: I also tried the Utilities.RotateImage method (to correctly orient the Texture2D) and received the same error.

 

I finally got it to work taking a photo (documentation should be updated really) and get a photo plus orientation information back.

I see as others point out that if in landscape the orientation states Mirror or Rotate_180, Mirror and if in portrait is states Rotate_90, Mirror.

So orientation returns multiple entries in it? I would expect to test for one or the other?

Did I miss something, does NatCam not have a utility built in to correct this or do I have to use 3rd party code to flip then rotate?

And why is the photo mirrored in the first place?

Thanks

 

Use the Utilities.RotateImage function in NatCam Core.

You can zoom using DeviceCamera.ZoomRatio.