[Tutorial] Using C++ OpenCV within Unity

I've never worked with the Hololens myself, but I don't see a reason why it wouldn't work. You'll have to post which errors you are getting in order for us to be able to help you, however.

AllowUnsafeBlocks is in the .csproj of your Unity project root folder (the one above the Assets folder). Right click and open with notepad to be able to edit them. You only need to change those values to be able to attach Visual Studio to Unity for debugging though, it will not cause any errors.

I'll need to see your code in PositionAtFaceScreenSpace to be able to get an idea what's giving you the null reference.

 

@Thomas-Mountainborn thank you for quicly reply,

At the moment, I have no way to post the error that I'm getting, because I've got the project in my company.
However, what do you think about dlls? Is it possibile that I'm getting these errrors because the dlls are in x64 and the hololens are in x86? If it's so, do I have to do everything again in x86 rather than in x64? Tomorrow I will post the errors that I'm getting, thanks anyway!!

 

If you are using the 32 bit Unity editor, then yes, you will need to compile 32 bit dll's. Otherwise though, I believe it should be just fine in x64.

 

Well, so it depends on unity architecture even if the hololens are in x86.

 

Good morning Thomas,

The error that I'm getting is:
Loading opencv_superres310.dll

Failed to load 'opencv_superres310.dll', expected x86 architecture, but was Unknown architecture. You must recompile your plugin for x86 architecture.

This error is for all dll, not only for this, and it happens when I try to run the app.

[EDIT]:
I tried to compile my dll for x86, but it's the same. I don't know what to do, vs doesn't recognize the architecture

 

Ah, in that case you will indeed have to compile a 32 bit (= x86) OpenCV project. In part one of the tutorial, you will have to change all actions relating to 64 bit to 32 bit (notably step 7 and step 9). It's probably easiest if you go through the entire set-up again in a new folder for 32 bit, and then copy the 32 bit OpenCV dlls to your C++ project, which you then in turn build for x86 architecture.

 

Thank you for quickly reply, I'll try to do that! I'll let you know.

 

@Thomas-Mountainborn I did it, but nothing rgiht, I'm always getting the same errror : Failed to load 'opencv_superres310.dll', expected x86 architecture, but was Unknown architecture. You must recompile your plugin for x86 architecture.

if you look closely, you note "Unknown architecture". it doesn't recognize the architecture because the common error is:

 

You will need to download the 32 bit Unity editor if you want to use 32 bit libraries (download link taken from this page). Be sure to set your target architecture to x86 in the build settings as well.

 

Nothing changed....

 

Here it is

 

Hmm, I don't know what else it could be... seems like you have an afternoon of trial & error and Googling ahead of you.

 

Hello Thomas, I got a new error and I think you can help me! Compiler reports that it can't load "dll name" because the specified module could not be found! In the script (OpencvFaceDetection.cs) we imported the dll with dllImport("dll name"), and then put the dll in the root folder where the script was located. But when you build the app, the dll is in a different place and compiler couldn't find it. Do you know how to do? Thanks anyway.

 

Hi Thomas

I have a great deal of experience with image processing and open CV, however i am new to Unity. I have managed to make all the .dll files but i have no idea on how to start or create the unity project. Do you have a copy of the project that you used in the example that i could import and learn from?

 

Hi steve,

Here's a download link to the sample project I made for the tutorial.

Oiram, in Unity plugins don't go in the root folder where the script is located, but in the Plugins folder. You can check the sample project to see the folder structure I use.

 

That's perfect, thank you Thomas.

 

I am having serious trouble in your tutorial part 1 @Thomas-Mountainborn implementing it. I tried sorting it out but can't figure it out because I literally don't know what I am doing. Please help
https://stackoverflow.com/questions...n-cmake?noredirect=1#comment77077586_45049683

 

Solved the above problem by re-installing visual studio and all its components. But, I am facing the wall again. Can anyone help?
https://stackoverflow.com/questions...ile-building-opencv-sln-in-visual-studio-2017

 

@Thomas-Mountainborn Solved the above problem. Used a different version of OpenCV and Built a .dll file successfully.
I followed the tutorial till part 3 and have created the two scripts. I have attached OpenCVFaceDetection.cs with an empty game object and the other with the main camera. When I try to run it, I get the following error:-
NullReferenceException: Object reference not set to an instance of an object
PositionAtFaceScreenSpace.Update () (at Assets/PositionAtFaceScreenSpace.cs:14)
Please Help

 

Are you sure there's an instance of OpenCVFaceDetection in the scene somewhere? If you left the scripts unaltered, you should not get a null reference exception at that line unless the script is not placed in the scene. You can download the full sample project from the link I posted above and check out how it should be set up.

Also, PositionAtFaceScreenSpace should go on an object you wish to position at the tracked face position, not the camera.

 

@Thomas-Mountainborn Thank you for your reply. I downloaded your code and ran it. Gives a black output in a seperate window. How do I make it show the actual image?
I want to make an application where I take the stream from a camera and display filters on the user's faces like msqrst or snapchat filters. Any help in this direction would be greatly appreciated. Thanks in advance

 

If the camera preview window is black, the stream probably failed to open. I'll check if the sample project still tries to open the capture on -1 instead of 0 when I get the chance.

You should can still see how the scripts are used in the scene though, and use that knowledge with your own OpenCV dll's.

 

@Thomas-Mountainborn I deleted everything and did everything accurately from scratch. Copied both the scripts as it is and attached to different objects as in your project. Still getting the NullReferenceException I can't figure out why. Also, your sample project is doing the same. I sincerely dont know what is wrong.

 

@Thomas-Mountainborn I am really stupid lol. Figured out my mistake. I was specifying the wrong .dll file to read the data from. Thank you. Got the camera streaming and detecting my face perfectly.

Can you reply to this:-
I want to make an application where I take the stream from a camera and display filters on the user's faces like msqrst or snapchat filters. Any help in this direction would be greatly appreciated. Thanks in advance

And how can I measure the delay in detecting the face from the time the data was sent to OpenCV?

 

The delay is easy - just get the current time before you send your data to OpenCV, and then compare it to the time after OpenCV returns.

As for the filters, I've no experience with that myself. There are some interesting tutorials on the subject, and there are also libraries which are fully focused on face pose estimation and expression recognition.

The subject is quite advanced though - considering it already took you considerable effort just to get the sample to run, maybe it would be wise to set your aim a little lower at first so you don't get burned out.

 

Indeed, the delay was simple.
As for the filters, I am aware of the advanced-ness of the topic but it would be appreciated if you could break down the task into minor checkpoints so that:-
1) I know i'm moving in the right direction
2) I know what to achieve in that day

I did a bit of search and it seems that face landmarking is a station on my journey so I'm gonna get started on that until I get some more guidance. The problem is actually that I'm actually a java guy and I have no experience handling c++ or c# code or libraries. Hence my posts on various forums like here. Thanks for your Co-operation. Anticipating a quick reply

 

Thanks @thomas-Mountain , First of all lot thanks for the tutorial.

For me build/compilation of all dlls both open cv and unity is done.
But When I pressed play button, nothing seems coming out.
I want to test the same with various inputs. Can you please let me know how to proceed further.

thanks and regards,

 

Hi, @Thomas-Mountainborn

First of all thanks alot for your tutorial.
How to test this with various inputs?
Do I need to create game object in Unity ?
Usually, in unity we create game objects and then attach script compoent( in this case C# script) Right?
Am I missing something?

Please let me know.

Thanks,

 

Hi

Hi,
I feel, I am also facing similar issues like you. Please correct me if I am wrong.
There are two C# scripts (one is faceDetection and other is position at screen face).
So which/what kind of game objects to be created in Unity ?
And to which objects above said scripts to be attached ?

Thanks and regards,

 

You can download my full project here and check it out.

 

Hi Thomas

I'm having great success with your tutorial so far. The next stage i need is to pass video data from the opencv dll and display it within unity. I think you mentioned earlier in a post that "part 4" would be dealing with this but while that is being written you have some code snippets that may help. Could you please help me with whatever you have?

Thanks

Steve

 

Hi Steve, glad to hear the tutorial is working out for you. Sadly, I've still not gotten around to writing the next part. I'll do my best to make some time for it.

 

Hello @martejpad,

I do all steps correct as you write and then build an apk with unity, but when i try to call function from plugin application crashes without any errors.

To be sure i build a simple plugin without opencv libraries and application works well.

 

Thanks alot @Thomas-Mountainborn , its working for me now. I compared your program and my program. Some build options were different.

Is there any possibility to render the streaming content onto unity3d camera objects?

Thanks and regards,

 

Hey @N0rt95 I'd like to help you, could you please give me more details?

 

Hi @Thomas-Mountainborn, do you know if this works well on iOS?
Has anyone on this forum tried to run it on iOS? Trying to figure out how much work would be required and if it's faster to just switch to Android instead and use @martejpad's instructions.
Thanks

 

Hi @Thomas-Mountainborn , thanks you for great tutorial. Im very anticipate for your part 4 tutorial about the optimized way to passing streaming webcamtexture to c++ native code using pointer

 

Hey s_unity, unfortunately I do not have any experience with iOS development.

 

Hello Thomas and everyone who has selflessly helped out the rest of us.

I wanted to thank you all. Because of Thomas' blog posts and other posters here,I was able to gather courage to step into the land of Unity Plugin development.

I came here looking for a way to use opencv with IOS ARkit ( because unity does not ( yet ) have a wrapper for apple's ML kit which includes face / object detection.
I found so much more...so thank you ALL !

Thomas if there is any way I can help you back please let me know. I have over 15 years of experience in the Hollywood VFX industry so I am happy to share my knowledge.

Cheers & look forward to part 4
b

 

Hi Behram,
Got here for the exact same reason (OpenCV on iOS and ARKit). Did you have any success with this ? In that case I'll be interested in knowing more about the way you did it.
Thanks

 

Hello Aurecon,
I'm following the steps outlined by @martejpad and things are still WIP but going as planned as of now.
The funny thing is I got in wanting a quick solution and have learned more than I bargained for !

The ability to inter-op different pieces of software (even hardware) are very liberating.It's intimidating yes,but the payoff is huge in the long run. I can only hope someone from Unity (@mikegeig) is watching and they decide to do more tutorials in managed and un-managed plugin dev.

Cheers
b

 

hi @Thomas-Mountainborn how would you go around detecting objects in unity using opencv ? How can you make unity camera as input for opencv? Having difficulties figuring it out.

 

I've now got my opencv plugin working in MacOS, Windows, Android, and iOS and I would say that iOS was by far the easiest to set up.

But the plugin creation process works completely differently in iOS compared to the others.

Instead of compiling an opencv plugin library up front, which you then import into the unity project as an external resources, for iOS you copy your own C++ source files to the unity project's Plugins/iOS folder. Then, when you get Unity to build the xcode project, it adds those source files to the project. XCode can then compiles your plugin code into the app along with the rest of it.

Before getting XCode to do the build, you need to do a couple of tweaks to the xcode project:

• adding the opencv framework to the project and
• adding to the search paths so that it finds

1. that framework
2. your include files and
3. the opencv include files.

​

There is also one change needed in the Unity code.

Because the plugin is built in (rather than loaded from an external library), you have to declare the api functions using

[DllImport("__Internal")]

rather than

[DllImport("NameOfLibrary")]

And that's everything.

Huge thanks to Thomas and martejpad, by he way. This page was very helpful indeed when I was doing my Android version, in particular.

 

@Thomas-Mountainborn Thanks for the awesome tutorial! I've found that I have an unusual issue that I don't think others have had. Using a plugin setup nearly identical to the tutorial (I'm using the camera for optical flow instead of object detection) I've found that I am able to run the game in the editor once, but if I stop it and run it again the editor crashes. Have you or has anyone else run into this issue? Unity version is 5.6.2f1

 

There's possibly a stream that you're not closing, and the plug-in raises an exception when you try to open it again the second time. That's what I think off the top of my head anyway, we might be able to give more advice if we see the code in question.

 

@Thomas-Mountainborn
Here's the C# code:

Code (CSharp):

1.  
2. internal static class OpenCVInterop {
3.     [DllImport("OpenCVCameraController")]
4.     internal static extern int Init (bool viewable);
5.  
6.     [DllImport("OpenCVCameraController")]
7.     internal static extern void Close ();
8.  
9.     [DllImport("OpenCVCameraController")]
10.     internal static extern void StartCalibration(int num_clusters);
11.  
12.     [DllImport("OpenCVCameraController")]
13.     internal static extern int UpdateInputs(int threshhold);
14.  
15.     [DllImport("OpenCVCameraController")]
16.     internal static extern MagTemp GetMagnitudeInputs();
17.  
18.     [DllImport("OpenCVCameraController")]
19.     internal static extern VecTemp GetVectorInputs();
20. }
21.  
22. public class OpenCVCameraReceiver : MonoBehaviour {
23.     /* extra variables */
24.     public bool ready = false;
25.  
26.     void Start() {
27.         ready = OpenCVInterop.Init (false) == 0;
28.     }
29.  
30.     void Update() {
31.         if (ready) {
32.             OpenCVInterop.UpdateInputs (5); //Updates optical flow values (gets new frame and performs calculations)
33.             magInput = new ClusterInputMag (OpenCVInterop.GetMagnitudeInputs ()); //Gets input
34.             vectInput = new ClusterInputVect (OpenCVInterop.GetVectorInputs ()); //Gets more input
35.         }
36.     }
37.  
38.     void OnApplicationQuit() {
39.         Debug.Log ("Close Camera!");
40.         if (ready) {
41.             OpenCVInterop.Close ();
42.         }
43.     }
44. }

And here's the C++ code:

Code (CSharp):

1.  
2. #include <opencv2\opencv.hpp>
3.  
4. /* ...Global Variables... */
5.  
6. VideoCapture vc;
7. Mat gray, prevGray, image, frame;
8.  
9. bool calibrate = false;
10. bool viewable = false;
11.  
12.  
13. //Structures to pass data from C++ to Mono
14. struct ClusterInputMag {
15.     ClusterInputMag(float * _magnitudes, int length) {
16.         for (int i = 0; i < length; i++) {
17.             magArray[i] = _magnitudes[i];
18.         }
19.         for (int i = length; i < MAX_CLUSTERS; i++) {
20.             magArray[i] = 0;
21.         }
22.     }
23.     float magArray[5];
24. };
25.  
26. struct ClusterInputVect {
27.     ClusterInputVect(float * _vectors, int length) {
28.         for (int i = 0; i < (length * 2); i += 2) {
29.             vectArray[i] = _vectors[i];
30.             vectArray[i + 1] = _vectors[i + 1];
31.         }
32.         for (int i = length * 2; i < (MAX_CLUSTERS * 2); i += 2) {
33.             vectArray[i] = 0;
34.             vectArray[i + 1] = 0;
35.         }
36.     }
37.     ClusterInputVect(InputVector * _vectors, int length) {
38.         for (int i = 0; i < length; i++) {
39.             vectArray[i * 2] = _vectors[i].x;
40.             vectArray[(i * 2) + 1] = _vectors[i].y;
41.         }
42.         for (int i = length * 2; i < (MAX_CLUSTERS * 2); i += 2) {
43.             vectArray[i] = 0;
44.             vectArray[i + 1] = 0;
45.         }
46.     }
47.     float vectArray[10];
48. };
49.  
50.  
51. //Initialization function for video capture and optical flow processing
52. extern "C" int __declspec(dllexport) __stdcall Init(bool _viewable) {
53.     viewable = _viewable;
54.     vc.open(0);
55.     if (!vc.isOpened()) {
56.         return -1;
57.     }
58.     // Other init stuff...
59.     return 0;
60. }
61.  
62. //Releases use of the webcam
63. extern "C" void __declspec(dllexport) __stdcall Close() {
64.     vc.release();
65. }
66.  
67. //Initializes calibration process to begin next Update
68. extern "C" void __declspec(dllexport) __stdcall StartCalibration(int num) {
69.     ///...Starts "calibration" for optical flow clusters
70. }
71.  
72. //Updates the cluster inputs
73. extern "C" int __declspec(dllexport) __stdcall UpdateInputs(float threshold) {
74.     vc.read(frame);
75.     if (frame.empty()) {
76.         return -1;
77.     }
78.  
79.     //Optical Flow
80.     frame.copyTo(image);
81.     cvtColor(image, gray, COLOR_BGR2GRAY);
82.  
83.     //Do optical flow calculations and updates global input variables
84.  
85.     if (viewable) {
86.         //Draws text, dots, and arrows onto image to display via imshow() if viewable set true
87.     }
88.    
89.     return 0;
90. }
91.  
92. //Returns the magnitude inputs calculated using optical flow and clustering
93. extern "C" ClusterInputMag __declspec(dllexport) __stdcall GetMagnitudeInputs() {
94.     //Returns magnitude input for each cluster via ClusterInputMag struct
95. }
96.  
97. //Returns the vector inputs calculated using optical flow and clustering
98. extern "C" ClusterInputVect __declspec(dllexport) __stdcall GetVectorInputs() {
99.     //Returns vector input for each cluster via ClusterInputVect struct
100. }

That's the basic version of the code, with the unrelated pieces left out. The UpdateInputs() function is essentially doing what Detect() does in the tutorial: processing a single frame and calculating some kind of output. Rather than using a fixed pointer though it just stores the float values in a global variable which can be retrieved later via GetMagnitudeInputs() and GetVectorInputs(). Everything works correctly with no error messages until the crash. Again, thanks for the help!

 

@Thomas-Mountainborn
I've taken a look at the log files and it turns out that Unity crashes due to an Access Violation by opencv_world300.dll.

Code (CSharp):

1. opencv_world330.dll caused an Access Violation (0xc0000005)
2.   in module opencv_world330.dll at 0033:de3dbb80.

I noticed that another user had this issue on the previous page but I've double checked that I've built in Release and that the dll is the release dll (with no Debug dll) but it still happens. I've also tried building in Debug mode and switching the dll but to no avail.

Edit: I also just realized that my definition for UpdateInputs had int for the input in C# but float for the input in C++. I fixed that but it still did not remedy the problem. I have narrowed it down to the UpdateInputs method however...commenting it out in the C# script makes the script work multiple plays.

 

I'm afraid I can't give you any advice, other than to keep narrowing down the UpdateInputs method until you discover the root cause.

 

@Thomas-Mountainborn
Solved! Finally... haha
I am using vectors of Point2f (OpenCV) for optical flow calculations and I did not clear the vectors when calling Close(). My Init function would push_back() new points each time it ran, so it would have twice the number of points in it, causing a for loop later to access an invalid part of an array that is only meant to have the size of one set of points. You'd think after studying CS for so long I'd find these things quicker... thanks again!

 

I'm trying to do an app for Android using Unity, so I followed @martejpad steps, but I have some questions after doing them.

Doing this way, do I need to set the System Variables of step 1 and build as in step 2, or can I just to skip to step 3? And the includes in C++ (#include "opencv2/objdetect.hpp", #include "opencv2/highgui.hpp" , #include "opencv2/imgproc.hpp"), will they be those in app\src\main\cpp\include?
I'm kind of new to this stuff of creating plugins, so sorry if they are silly questions. Thanks you.