Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

NatDevice - Media Device API

Discussion in 'Assets and Asset Store' started by Lanre, Dec 17, 2015.

?

Should we add exposure controls in v1.3? This means dropping support for iOS 7

Poll closed Jun 10, 2016.
  1. Yes

    9 vote(s)
    75.0%
  2. No

    3 vote(s)
    25.0%
  1. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,970
    Now you don't have to settle for the slow, lacking WebCamTexture API. The NatCam API provides access to device camera functions like high resolution previews (at high frame rates), exposure, focusing, zooming, flash, torch, photo capture, VR support, OpenCV support, and so much more. We used it in our upcoming project and it works absolutely great.

    Get NatCam on the Asset Store.



    NatCam Features Include:
    - Blazing fast camera preview (especially on Android).
    - HD. You can view what your phone sees in HD, or even higher resolutions if you wish to. All at a solid framerate.
    - Custom resolutions. NatCam allows you to set the camera's resolution. There are also resolution presets for ease of use.
    - Photo capture. NatCam allows you to capture high-resolution insta-worthy photos. You also have control over lighting with flash mode access.
    - Focus. Focus on anything you see. It also features focus modes (like autofocus, tap to focus, or even off).
    - Exposure. Set custom exposure bias for properly exposing those under- and over-exposed scenes your app will face. It also features exposure modes (like auto exposure and locked exposure).
    - Zoom. NatCam supports hardware optical zoom.
    - Torch. Switch on and off the device's LED torch.
    - VR support. NatCam is compatible with GearVR and Google Cardboard. Check out a Cardboard VR example here.
    - Preview Data. NatCam gives you access to the preview data for performing software operations.
    - OpenCV support. NatCam has built-in support the OpenCVForUnity API. Check out the official examples.
    - Minimalist package. NatCam is a very small, specialized package. It is specifically designed not to add unnecessary overhead to your project.
    - Instant support. We respond to support emails usually within minutes.

    See the documentation and Github.
     
    Last edited: May 25, 2020
  2. ikazrima

    ikazrima

    Joined:
    Feb 11, 2014
    Posts:
    320
    can you provide performance comparison?
     
    undyingwill likes this.
  3. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,970
    On my galaxy S3 pictured above, I can play a full HD preview (1080p) from the camera between 24-30FPS. As for Unity's WebCamTexture API, we all know how slow it is. WebCamTexture will only yield 3-4FPS at such a resolution.
     
    Last edited: Oct 6, 2016
    MrEsquire likes this.
  4. ikazrima

    ikazrima

    Joined:
    Feb 11, 2014
    Posts:
    320
    Hey @Lanre any update on this? :)
    Just curious since I retested my own app with Redmi Note 2 and the webcamtexture is performing poorer than my previous phone.
     
  5. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,970
    Hello @ikazrima . The package is still pending review; this is probably due to the Christmas and New Year holiday. I would guess that it would be reviewed within the next week. I'm very eager to see how Unity Devs use it.
     
  6. Dotby

    Dotby

    Joined:
    Nov 12, 2013
    Posts:
    32
    Oh. I found this. When I came across this thread, I almost cried like a little girl. A week I struggled with Unity camera and began to teach Java to write the wrapper. It would take a lot of time and I had almost given up hope that I would find a solution. But NatCam save my hope! Thank you, Larne.
     
    Last edited: Jan 1, 2016
  7. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,970
    You're welcome Dotby :D .

    I would still advise you to get familiar with Android Java in your free time because the sources are included. That way, if you want to add some custom functionality, like scanning a new type of barcode, you can do so easily. I have included a little readme for the sources and will be happy to discuss Java programming with you.

    But yes, NatCam is made for extreme convenience. You sure would love it!
     
    Cromfeli likes this.
  8. bariscigal

    bariscigal

    Joined:
    Oct 26, 2009
    Posts:
    46
    Ok refreshing "new-on-asset-store" page continously. Come on Unity. Release this already.
     
    Dotby likes this.
  9. therobear

    therobear

    Joined:
    Sep 28, 2010
    Posts:
    22
    Haha, I am doing the same thing, but on this thread.
     
  10. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,970
    Hi guys! I'm sorry that I have been incommunicado for the past few days; My team is about to launch an app so we're performing last-minute adjustments.

    I have bitter-sweet news (more sweet than bitter). I must inform you that Unity still has my package in Pending Review. Because it's taking so long, I have asked Unity to cancel it. In order to make up for it, I plan to add even more features and release a 100% complete featureset. These new features will include OpenCV support, video capture support, front camera support, zoom, torch, custom camera resolutions, and quite (very) possibly, photo filters. I plan to be done with all this in about 2 weeks. Then I'll send it back to Unity for review.

    I am really really really sorry that all this is taking a while, but it is completely worth the wait. You won't ever need another device camera solution again. Thanks for your patience!
     
    ikazrima and jcarpay like this.
  11. Dotby

    Dotby

    Joined:
    Nov 12, 2013
    Posts:
    32
    Thank u Lanre! This is very good news)
     
  12. ikazrima

    ikazrima

    Joined:
    Feb 11, 2014
    Posts:
    320
    The OpenCV support is music to my ears ;)
     
    FaberVi likes this.
  13. therobear

    therobear

    Joined:
    Sep 28, 2010
    Posts:
    22
    Sounds awesome! Take as much time as you need.
     
  14. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,970
    FaberVi and jcarpay like this.
  15. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,970
    Hello guys. I've been working on OpenCV support for Android NatCam. The biggest hurdle is returning a texture that lives in system memory to OpenCV. The issue with that is that getting the camera preview to Unity requires using GPU processing (to convert from YUV to RGB). That's what makes NatCam so fast. Now getting that texture data back to system memory is the huge challenge. The only option is to use glReadPixels, which is notoriously slow. It sends framerates from ~30FPS to ~8FPS (rendering a 1920x1080 camera preview). That is totally unacceptable. I don't want us devs to have to sacrifice quality resolutions for speed. There are two possible compromises that can be made:

    1. For OpenCV, supplying a camera preview with a very low frame rate (~5FPS) while displaying the fluid preview to the user. This is how NatCam performs Barcode Detection on Android. Does OpenCV require a fluid 30FPS stream? I would love to know as I have not worked with OpenCV myself.

    2. ...

    EDIT: An optimized method has been found. Stay tuned.
     
    Last edited: Jan 16, 2016
    yinczar and FaberVi like this.
  16. bariscigal

    bariscigal

    Joined:
    Oct 26, 2009
    Posts:
    46
    5 fps is pretty low for opencv. And also i think it should be close to the real capture fps, because the tracking would delay on the real image.
    But the good thing is while using the opencv i currently using a resolution of ~240x180 and getting really good results from a processed image. (get the data array > rescale the image (pyrDown) > converting grayscale > applying histogram) Also all this happens using opencv which takes place on the data array.
    So i think rescaling the opencv image on the gpu and using it with opencv would be a great solution.
     
  17. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,970
    NatCam should now support fluid OpenCV on iOS. Android OpenGLES2 coming up next! Then Android OpenGLES3. :D
     
  18. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,970
    NatCam should now support OpenCV on Android OpenGLES2 devices with ARM CPU's (which account for the majority of mainstream Android phones). It currently doesn't support x86 CPU's (it will during release, but will be marked 'Experimental'). I get between 18-22FPS on my Galaxy S3 running a 1920x1080 preview. Pretty decent performance. Now on to OpenGLES3! :D

    EDIT: Also in the works in Android IL2CPP support. Currently working with a few issues. Will all be resolved in the release.

    EDIT: NatCam should now work with OpenCV on Android GLES2 devices. On both ARM and x86 CPU's.
     
    Last edited: Jan 15, 2016
  19. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,970
    Hello guys. Unfortunately, NatCam v1.0f1 will not include video recording support. This decision was made considering the amount of development time, and also challenges that will arise in video playback.
     
  20. shawnblais

    shawnblais

    Joined:
    Oct 11, 2012
    Posts:
    324
    Wow, I can't believe my luck, I'm trying to build an image scanning app, and have been working on a native plugin for a few days, and really getting frustrated with the lack of progress. The fact you're working on one as we speak is pretty damn amazing!!

    My needs are simply a performant camera preview, with the ability to take a hi-resolution auto-focused picture when the user taps a button, along with optional flash support. I'm using OpenCV to detect scanned documents, and would only need to run it on the final, captured image, not on the entire image preview.

    Any ideas on a timeline for release? Please hurry up and take my money!
     
  21. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,970
    Hi shawnblais. Let's look through your needs:

    0. Performant camera preview. CHECK
    1. Take hi-resolution stills. CHECK
    2. Auto-focus. CHECK
    3. Flash support. CHECK
    4. Live OpenCV support. Almost Completed.

    I'm targeting an Asset Store resubmission on the 21st. I don't know how long it'll take Unity to review it, probably about 5-7 days. I'll keep this thread updated.
     
    shawnblais likes this.
  22. shawnblais

    shawnblais

    Joined:
    Oct 11, 2012
    Posts:
    324
    You're a savior!

    Just to confirm, would you be adding support for the the AutoFocus callback in Android? In Java it would be:
    Code (Java):
    1.  
    2.     mCamera.autoFocus(new Camera.AutoFocusCallback() {
    3.         public void onAutoFocus(boolean success, Camera camera) {
    4.           if (success) {
    5.              camera.takePicture(null, rawCallback, jpegCallback);
    6.           }
    7.        }
    8.     });
    9.  
    Basically I'm building a custom CameraUI, and want to replicate the behavior of the native camera when taking stills.
     
    Last edited: Jan 17, 2016
  23. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,970
    This functionality was completed in the earliest NatCam beta. It's fully supported no worries. And there's tap to focus, which will automatically fade into auto-focus like iOS cameras behave.
     
    Dotby and shawnblais like this.
  24. schpinn

    schpinn

    Joined:
    May 13, 2015
    Posts:
    3
    Is there currently any way to obtain NatCam? Also, if you need more beta testers, I'd be more than happy to help you test it out, I'm currently working on a project involving OpenCV so I could help you test OpenCV integration as well.
     
    Dotby likes this.
  25. shawnblais

    shawnblais

    Joined:
    Oct 11, 2012
    Posts:
    324
    Ya, +1 to that, I'd be happy to help test, and anxiously await the release :)
     
  26. Dotby

    Dotby

    Joined:
    Nov 12, 2013
    Posts:
    32
  27. EddieOffermann

    EddieOffermann

    Joined:
    Sep 13, 2015
    Posts:
    13
    Anxious to give this a look - I'm curious how close the API is to being at least a partial drop-in replacement to WebCamTexture for integration to other APIs
     
  28. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,970
    Hi guys. NatCam's development was slowed down due to an issue on iOS with OpenCV compatibility (making the textures readable). It has been fixed. We are currently rebuilding the codebase (out of code we already have, so we're basically copying, pasting, and cleaning up). I approximate that we'll be done in 5 days.

    @EddieOffermann I would say that the NatCam API can completely replace the WebCamTexture API. The only issue is that it is built with iOS and Android in mind, but the last thing on our task list is adding a wrapper around the WebCamTexture API so that WebCamTexture-supported calls to NatCam on unsupported platforms will be forwarded to the WebCamTexture API.
     
    FyreDogStudios likes this.
  29. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,970
    Sunday Update:

    -NatCam iOS is almost 100% complete, then I'll begin working on NatCam Android (which in itself is about 45% complete).
    -NatCam WebCamTexture wrapper is now about 10% done. It won't take much time to complete.
    -LUT Filters have been postponed to NatCam v2.0 because of my team's time considerations with other projects.
    -Documentation has begun. I'll publish the link to it in about 2 days. It's a small API so it won't be intimidating.
    -I will also be releasing an APK with the documentation so you can adb it to your device and see for yourself.

    I only have 4 pages of task list left :D. NatCam coming very soon.

    EDIT:
    -We've decided to push exposure control to v2.0 also as it is not a core feature (currently NatCam will set auto exposure which will work in 99.9% percent of use cases), and it isn't worth spending time on it now because that time has opportunity cost for us and you all. Stay tuned.
     
    Last edited: Jan 24, 2016
    jcarpay likes this.
  30. Dotby

    Dotby

    Joined:
    Nov 12, 2013
    Posts:
    32
    Thanks for the news)
     
  31. shawnblais

    shawnblais

    Joined:
    Oct 11, 2012
    Posts:
    324
  32. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,970
    I am happy you asked this question so as to avoid confusion.

    Initially, NatCam provided a camera preview texture that existed solely on the GPU. This means that the pixel data of the camera preview was not available to software processes like GetPixel(s), EncodeToPNG/JPG, and as a result, OpenCV applications.

    NatCam now supports OpenCV as pixel data from the camera is now fully accessible for processing by your favorite OpenCV library.

    This DOES NOT mean that NatCam includes OpenCV libraries. It will probably not be included in future releases as there are already full packages on the Asset Store.

    Using an Asset Store OpenCV package or a similar C# implementation should work seamlessly now. I would confirm this with beta testers once the beta is completed and sent out.
     
  33. shawnblais

    shawnblais

    Joined:
    Oct 11, 2012
    Posts:
    324
    Very cool! So will this just be a raw array of pixel data? Or available as a Texture2d? In order to work with "OpenCV for Unity", we need to convert Texture2d to a Matrix, which is a performance drain, so an array of raw pixel data would be ideal.
     
  34. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,970
    I'm glad that you've brought this to my attention. Currently NatCam has the preview's pixel data in a Texture2D. But I would like to know:

    -Instead of converting a Texture2D to a Matrix because you don't have raw pixel data, couldn't you simply call Texture2D.GetRawTextureData? (this returns a byte[] with pixel data)

    -Does your OpenCV package support pixel buffers in BGRA format? Can you define what format your pixel buffer is? BGRA is the format iOS supports, and converting it to ARGB32 or RGBA32 will have a significant performance impact.
     
  35. shawnblais

    shawnblais

    Joined:
    Oct 11, 2012
    Posts:
    324
    Currently the package provides utility functions, matToTexture2D and texture2DToMat, which call native code to do the conversion. Converting a 512x512 texture to mat costs about 8ms on my Android Phone.

    It looks like it converts textures to mats in RGBA, and I don't see anywhere where you can define the default type (though I could easily be missing it).

    I'll PM you the Utils code from the library with all the import functions, maybe it will help shed some more light on how it works.
    Thanks!
     
  36. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,970
    Tuesday Update:

    NatCam is almost complete. Just about 7 things left. It will be completed sometime later today.
     
    FyreDogStudios and jcarpay like this.
  37. FyreDogStudios

    FyreDogStudios

    Joined:
    Aug 23, 2015
    Posts:
    97
    So how would this thing work?

    Will it be supported with the UI system? Could a stick a script on a GameObject with an image component on it?
     
  38. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,970
    @FyreDogStudios NatCam is a static class. It has a 'Preview' field which gives you access to the camera's preview texture (similar to WebCamTexture). This is our current setup:

    Code (CSharp):
    1.  
    2.     public RawImage preview;
    3.  
    4.     // Use this for initialization
    5.     void Start () {
    6.        //Initialize NatCam
    7.        NatCam.Initialize(PreviewType.NonReadable, PhotoSaveMode.SaveToPhotos);
    8.        //Set HD Resolution
    9.        NatCam.SetResolution(ResolutionPreset.HD);
    10.        //Start the preview
    11.        NatCam.Play();
    12.        //Project the camera preview texture on the UI Panel RawImage
    13.        preview.texture = NatCam.Preview;
    14.        //We want control over focusing
    15.       NatCam.tapToFocus = Switch.On;
    16.     }
     
    FyreDogStudios likes this.
  39. Dotby

    Dotby

    Joined:
    Nov 12, 2013
    Posts:
    32
    I believe in NatCam)
    So far, NatCam is best solution that I have found. The only possible solution.
    Please, give it to us. Can not wait to try, Lanre)

     
  40. shawnblais

    shawnblais

    Joined:
    Oct 11, 2012
    Posts:
    324
    haha, I too am a believer. Also, while it's running through approval, I'd be happy to pay you via paypal in order to get early access to the code. You'd get an extra 30% and some free beta testing. As long as I can eventually get a license to the Asset Store version.
    Cheers,
     
    Dotby likes this.
  41. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,970
    Hi guys. Sorry I haven't been here for a while. NatCam is literally 99.9% done. It is completed (I have a final package sitting on my computer) but I can't submit it yet because one of our betas reported an issue where the preview won't work on Android KitKat. I expect to fix it in the next 2 days.
     
    Last edited: Feb 28, 2016
  42. Dotby

    Dotby

    Joined:
    Nov 12, 2013
    Posts:
    32
    Black screen preview on Android 6.0.1
     
  43. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,970
    @Dotby care to be a beta? If so I'll dm you for more details and get the issue fixed. Thank you!
     
  44. schpinn

    schpinn

    Joined:
    May 13, 2015
    Posts:
    3
    Android 6.0.1 here as well, everything works as expected.
     
  45. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,970
    The unpredictable world of Android. This issue, and the one on KitKat are all that are left before we send in NatCam. We are investigating both right now.
     
  46. shawnblais

    shawnblais

    Joined:
    Oct 11, 2012
    Posts:
    324
    Testing on Kindle Fire HDX (3rd Gen), I get the Unity Splash Screen, and then the app shuts down.

    This uses FireOS 4.5.x, which wikipedia tells me is a fork of Android 4.4.2.

    I was able to pull these errors from Logcat: http://screens.gskinner.com/shawn/TmO.png
     
  47. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,970
    Hi @shawnblais. Thanks for the log; we have addressed the issue. On some devices, it throws NullPointerExceptions when it is not supposed to.

    We are still investigating the KitKat/Marshmallow issue. Once that's been resolved, NatCam will be on its way to the Asset Store. I'll upload a new APK when the issue is fixed.
     
  48. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,970
    The KitKat issue has been solved. Now, the Marshmallow issue is all that's left.
     
    jcarpay likes this.
  49. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,970
    Marshmallow issue solved. We'll be sending in NatCam for review on Monday. Yaay!
     
    jcarpay and Dotby like this.
  50. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,970
    Now the ball is in Asset Store Review's court
     
    Dotby likes this.