[RELEASED] OpenCV for Unity

Thanks Enox. I've moved the face detection to another thread which has improved performance slightly. It's taking ~300ms to detectMultiScale. Are there any alternatives to detect faces that runs better?

Analysis in the following article shows a much superior performance on Android:
http://ngvb.net/wp-content/uploads/2014/11/cobarzan_androidvsios.pdf

 

this Assets uses the same native code in Android and iOS.However, it seems there is a difference in processing speed.
Let me confirm.

 

Does anyone have scripts to automate the build and run to iOS ? would prefer not to have to link the library to the binary everytime i need to test on device.

 

In the case of Unity5.0.1 or more, it will not occur link error by placing the following code in the ”Assets / Editor ”folder.

1. Rename "OpenCVForUnity/iOS for Xcode/" to "OpenCVForUnity/OpenCVforXcode/".
   
2. Add "iOS_BuildPostprocessor.cs" in ”Assets / Editor ”folder.

iOS_BuildPostprocessor.cs

Code (CSharp):

1. using UnityEngine;
2. using UnityEditor;
3. using UnityEditor.Callbacks;
4. using UnityEditor.iOS.Xcode;
5. using System.Collections;
6. using System.IO;
7.  
8. public class iOS_BuildPostprocessor : MonoBehaviour
9. {
10.    
11.     [PostProcessBuild]
12.     public static void OnPostprocessBuild(BuildTarget buildTarget, string path)
13.     {
14.         if (buildTarget == BuildTarget.iOS)
15.         {
16.  
17.             string projPath = path + "/Unity-iPhone.xcodeproj/project.pbxproj";
18.            
19.             PBXProject proj = new PBXProject();
20.             proj.ReadFromString(System.IO.File.ReadAllText(projPath));
21.                    
22.             string target = proj.TargetGuidByName("Unity-iPhone");
23.  
24.             // Add our framework directory to the framework include path
25.             proj.SetBuildProperty(target, "FRAMEWORK_SEARCH_PATHS", "$(inherited)");
26.             proj.AddBuildProperty(target, "FRAMEWORK_SEARCH_PATHS", "$(PROJECT_DIR)/Frameworks/OpenCVForUnity/iOSforXcode");
27.            
28.  
29.             File.WriteAllText(projPath, proj.WriteToString());
30.         }
31.     }
32. }

 

By changing the cascadefile, performance has been greatly improved.(tested in Nexus7)
In OpenCV Face Detection Sample by OpenCV.org, it seems to use the "lbpcascade_frontalface.xml".

Code (CSharp):

1. //cascade = new CascadeClassifier (Utils.getFilePath ("haarcascade_frontalface_alt.xml"));
2. cascade = new CascadeClassifier (Utils.getFilePath ("lbpcascade_frontalface.xml"));

lbpcascade_frontalface.xml
https://github.com/Itseez/opencv/bl...-detection/res/raw/lbpcascade_frontalface.xml

 

Thanks Enox. I'll give this a try.

 

Hello Enox, Thank you very much for the code. I appreciate it. It would be very helpful if the readme is updated with this code and the folder path is also changed from the start. The only thing I need to change from your code in order to make it work is the FRAME_WORK_SEARCH_PATHS (and apparently, you cannot use space for the path)

 

Hello, several questions before buying this package.
Do you have a demo?
Can we export to webgl?
thank you

 

The demo version is not available.
Since the Unity Web Player does not support the native plugin, "OpenCV for Unity" does not support "WebPlayer　Platform".

 

It would be better if you also put stuff like this in the readme. It might save couple of hours of problem

 

Hi,

I have a pre-sale question: is it possible to build an app like

that will run on Surface Pro 3 (Windows 8.1) with the help of OpenCV for unity?

The app would need to:
- recognize a marker which would be a image of a product (in order to know what product should be displayed)
- display a model correctly positioned on a marker placed in a real world

Thanks!

 

I think that it is possible basically with "OpenCVForUnity" if it is possible with "OpenCV Java".
- recognize a marker which would be a image of a product (in order to know what product should be displayed)
it is possible by using the DescriptorMatcher.
- display a model correctly positioned on a marker placed in a real world
Please look for an example of Marker-less AR Sample with "OpenCV Java".

 

Hello there,

In my internship we bought this cool asset and I have developed several apps demos for AR, so I will share a chip of the code examples I developed : Homography with KnnMatch http://pastie.org/private/d9rd4ylecnu4hqcjpdow
You can use it to find the camera position, that is at your choice ;D

In the other hand, I want to know if instead of c++ version of findContours() we can access the c version, because the vector management of the STL is so slow I have many problems to achieve realtime on pc, and it's so slow on mobile.

If there's not I will share the solution later (Maybe a C# translation of the C version ?).

Thanks in advance.

 

I have Buy the package,but When i run sample,there is always error imformation,like this
DllNotFoundException: opencvforunity
OpenCVForUnity.Mat..ctor (Int32 rows, Int32 cols, Int32 type) (at Assets/OpenCVForUnity/org/opencv/core/Mat.cs:520)
OpenCVForUnitySample.WebCamTextureDetectFaceSample+<init>c__Iterator3.MoveNext () (at Assets/OpenCVForUnity/Samples/WebCamTextureDetectFaceSample/WebCamTextureDetectFaceSample.cs:137)
UnityEngine.MonoBehaviour:StartCoroutine(IEnumerator)
OpenCVForUnitySample.WebCamTextureDetectFaceSample:Start() (at Assets/OpenCVForUnity/Samples/WebCamTextureDetectFaceSample/WebCamTextureDetectFaceSample.cs:78)

 

Thank you for the great code example.That is very helpful.

Unfortunately, there is no way to access the c version.

 

Plugin does not seem to be loaded correctly. Please check the setup procedure.

 

I just found an issue compiling the latest version of the plugin on a non-pro install of Unity4.x. The new face recognition functions createLBPHFaceRecognizer, createEigenFaceRecognizer, etc, their body is in a #if UNITY_PRO_LICENSE block. In the else case of that #if block, they all return false, yet the functions return an object. This won't compile. It's a simple fix, but requires changing the plugin code base with the updated plugin.

 

I am unable to get VideoCapture working on android devices. I see errors in the android console when it is loading all of the libnative_camera_r* libraries, and get this error for each version:

dlerror="dlopen failed: cannot locate symbol "_ZN7android11BufferQueueC1EbRKNS_2spINS_19IGraphicBufferAllocEEE" referenced by "libnative_camera_r4.3.0.so"..."

One of those for every version of the libnative_camera library. I would prefer using VideoCapture over WebCamTexture, since the former is faster. It works on iOS.

 

Thank you for bug report.
I will fix this bug in a future version.

 

Currently OpenCV does not support for nativecamera(VideoCapture) on Android5.0.
and There does not seem to be the plan to support in the future.
http://code.opencv.org/projects/opencv/wiki/ChangeLog

• The other, a bit sad but inevitable news - we had to drop support for nativecamera on Android. It used some undocumented API and never worked stable.

Please use the WebCamTexture instead of VideoCapture on Android.

 

Finally we decided not go in that direction(Binary Markers) and use a photo marker.

No problem, you can add it to the sample codes, just quote me as the coder

.

 

Hi i want to do wand magic tool is this doable in this tool

 

Hi EnoxSoftware,

I use OpenCV for Unity in my project, which works under Windows and Android (for face detection). I wanted to try it on iOS but I have linker errors in the XCode project. It seems that opencv2.framework needs the target to be libstdc++ while the Unity library libiPhone.a needs it to be libc++, so I get missing symbols in both cases. This post seems to be similar but doesn't provide any checked solution.
Do you know if there is any solution to build the XCode project successfully? If you need more detailed logs or anything I'd be happy to bring them.

Thanks in advance
Unity 5.0.1f1, XCode 6

 

In Unity4 and Unity5 the different files to be imported(opencv2.framework and libopencvforunity.a). When you update the project that you made in Unity4(libstdc++) to Unity5(libc++), please import again this asset in Unity5.

 

OpenCV for Unity
Released Version 1.1.5

Version changes1.1.5
[Common]Add Beta Version of “OpenCV for Untiy” based on “OpenCV3.0.0-rc1”(support Unity5).
[Android]Fix Utils.getFilePath().
[Common]Add WebCamTextureAsyncDetectFaceSample.
[iOS]Change folder name from “OpenCVForUnity/iOS for Xcode/” to “OpenCVForUnity/iOSforXcode/”.Add iOS_BuildPostprocessor.cs.

Add Beta Version of “OpenCV for Untiy” based on “OpenCV3.0.0-rc1”. (WindowsStoreApp8.1 & WindowsPhone8.1 support)

 

It solved the problem, thanks a lot!

 

Hi,

im trying to clear out the undefined depth pixels from a kinect image (grayscale image, with pixels with value 0 being undefined). I tried to do it using morphologyEx which seems a good method to fill the gaps nicely.

Code (CSharp):

1. // fill holes with spheres
2. int counter = 0;
3. int morph_size = 2;
4. while ((Core.countNonZero(inputMat) != (inputMat.width() * inputMat.height())) && (counter < maxMorphIterations)) {
5.     int morph_elem = 2;
6.     Mat kernelMat = Imgproc.getStructuringElement( morph_elem, new Size( 2*morph_size+1, 2*morph_size+1 ), new Point( morph_size, morph_size ) );
7.     Imgproc.morphologyEx (inputMat, inputMat, Imgproc.MORPH_CLOSE, kernelMat);
8.     //inputMat = inputMat.clone();
9.     counter ++;
10.     }
11. Debug.Log ("took " + counter + " turns to remove empty space " + Core.countNonZero(inputMat) + " "+ counter);

my problem is that the loop does not fill up the holes continuously. It stays the same and is only dependant on the morph_size.

Anybody knows why this is the case?

 

Hi.
I want detect reactangular object and I want draw object's contourns.
This is my code but I have an error:


List<MatOfPoint> contours = new List<MatOfPoint>();
Mat hierarchy = new Mat();

Imgproc.cvtColor(rgbaMat, rgbaMat, Imgproc.COLOR_RGBA2GRAY);
Size blurSize = new Size(5,5);
Imgproc.blur(rgbaMat, rgbaMat, blurSize);
Imgproc.Canny(rgbaMat, rgbaMat,0, 100);
Imgproc.dilate(rgbaMat, rgbaMat, new Mat());

Imgproc.findContours(rgbaMat, contours, hierarchy, Imgproc.RETR_TREE, Imgproc.CHAIN_APPROX_SIMPLE);

List<MatOfPoint2f> squares = new List<MatOfPoint2f>();

int j = 0;

for(int k = 0; k < contours.Count; k++)
{
MatOfPoint2f contour2f = new MatOfPoint2f(contours[k].toArray());
double approxDistance = Imgproc.arcLength(contour2f, true) * 0.02f;
MatOfPoint2f approxCurve = new MatOfPoint2f();
Imgproc.approxPolyDP(contour2f, approxCurve, approxDistance, true);

if(approxCurve.toArray().GetLength (0) == 4 && Imgproc.contourArea(approxCurve) > 1000f && Imgproc.isContourConvex(new MatOfPoint(approxCurve)))
{
Debug.Log("OK");
}
}

Error is:
CvException: Incompatible Mat
OpenCVForUnity.MatOfPoint..ctor (OpenCVForUnity.Mat m) (at Assets/OpenCVForUnity/org/opencv/core/MatOfPoint.cs:32)

Any suggest?

Then, How can I draw contourns?

Thanks in advance.

 

MatOfPoint and MatOfPoint2f are different CvType.
Please change code as follows.

Code (CSharp):

1. if(approxCurve.toArray().GetLength (0) == 4 && Imgproc.contourArea(approxCurve) > 1000f && Imgproc.isContourConvex(new MatOfPoint(approxCurve.toArray())))
2.  {
3.              Debug.Log("OK");
4.  }

 

Hi! Thanks for the great wrapper!

I have some problems running a simple K-nearest search on the other hand.

Code (CSharp):

1.  
2.  
3. using UnityEngine;
4. using System.Collections;
5.  
6. using OpenCVForUnity;
7.  
8. public class deneme : MonoBehaviour {
9.  
10.     void Start () {
11.  
12.         Mat sampleMat = new Mat (3, 2, CvType.CV_32FC1);
13.         Mat responseMat = new Mat (3,1, CvType.CV_32FC1);
14.  
15.         sampleMat.put (0, 0, 2);
16.         sampleMat.put (0, 1, 3);
17.  
18.         sampleMat.put (1, 0, 1);
19.         sampleMat.put (1, 1, 1);
20.  
21.         sampleMat.put (2, 0, 5);
22.         sampleMat.put (2, 1, 3);
23.  
24.         responseMat.put (0, 0, 3);
25.         responseMat.put (1, 0, 4);
26.         responseMat.put (2, 0, 5);
27.  
28.         Mat sampleIdMat = new Mat (3, 1, CvType.CV_8UC1, new Scalar (1));
29.  
30.         Mat testMat = new Mat (1, 2, CvType.CV_32FC1, new Scalar (1));
31.  
32.  
33.         CvKNearest KNN = new CvKNearest (sampleMat, responseMat, sampleIdMat, true, 10);
34.  
35.         KNN.train (sampleMat, responseMat, sampleIdMat, true, 1, false);
36.  
37.  
38.  
39.  
40.         Mat predictMat = new Mat (1, 1, CvType.CV_32FC1, new Scalar (7));
41.         Mat neighborMat = new Mat ();
42.         Mat distMat = new Mat ();
43.  
44.  
45.         KNN.find_nearest(testMat,1,predictMat,neighborMat,distMat);
46.  
47.         Debug.Log (ans + "::" + predictMat.dump () + "::" + predictMat.ToString() );
48.  
49.         KNN.save ("a.xml");
50.  
51.     }
52.  
53.     void Update () {
54.  
55.     }
56. }
57.  

I generate same sample data and fill the sample and response matrix, init the search space (train) and call find_nearest.

I expect predictMat to be filled with the output (nearest neighbor's value) but it stays as it is. If I dont init predictMat to 7s it reads random memory data.

Additionally save method generates the file but it is empty.

Are these functionality not supported or am I missing something?

Edit:

Using only KNN.train (sampleMat, responseMat) works OK, but i want to use regression, not vote-based classification. Guess there is smt wrong with sampleIdx. Both CV_8UC1 with 1s or 255s and CV_32SC1 with indices does not work. Is it possible to pass null pointers somehow?

save still does not work.

These problems and some other missing functionality, I guess I have to switch to another solution...

 

Hi Enox Software, I encountered some problems while using the combination of Vuforia and OpenCVForUnity.
First, In Unity 4.6.6., I got black screen video from ARCamera Prefab when I built an iOS app with only these two libraries. It's workable while using Vuforia only without OpenCVForUnity. So I wondered that Vuforia's ARCamera is not workable if these two libraries are used in the same time in iOS version. There is no problem in Android version.
Second, In Unity 5.1.1, it worked smoothly on iOS version this time. But, there existed the same problem, black screen problem, in Android version. But I also find a workaround right now that it could be also workable on Android version. The workaround is that not use development build mode in Android version. Please check these issues. Thx.

 

Thank you for a bug report.
Please let me confirm.

 

Do I need to file a bug report independently? Or is the above message is sufficient?

 

In Unity 4.6, It seems the black screen problem occurs while using the combination of Vuforia and OpenCVForUnity.
I am investigating the cause.Unfortunately, please use the Unity5 at the moment.

 

When I tested the same code in "OpenCV for Android"(http://docs.opencv.org/doc/tutorial...d_binary_package/dev_with_OCV_on_Android.html), the same error on the following page occurred.
http://answers.opencv.org/question/13418/machine-learning-saveload-problem/
This bug seems to be a bug in OpenCV2.4.10.


Additionally, the following error has occurred on this line.

KNN.train (sampleMat, responseMat, sampleIdMat, true, 1, false);

07-02 02:07:41.844: E/cv::error()(14425): OpenCV Error: One of arguments' values is out of range (k must be within 1..max_k range) in virtual float CvKNearest::find_nearest(const CvMat*, int, CvMat*, float const**, CvMat*, CvMat*) const, file /home/reports/ci/slave_desktop/50-SDK/opencv/modules/ml/src/knearest.cpp, line 384

 

Hello, can not operate facetrakersample I tested on unity 4.63 / 5.00 / 5.1.1! I do not have a camera and an error (image below.
I also sent an email on your media and still no answer!
I am very disappointed for a pakage has € 101!

 

how about adding [System.Serializable] to the Point Class?

Code (CSharp):

1.  
2.         [System.Serializable]
3.         public class Point
4.         {
5.  
6.                 public double x, y;
7.  

 

I am terribly sorry for the late response.

According to the error message, it seems to fail to the loading of the file.
Please Copy from “FaceTrackerSample/StreamingAssets/” to “Assets/StreamingAssets/” folder.

 

thank you it works, I had seen the manipulation but I moved the file in opencv.
I wish I have OpenCVForUnity package without exporting via the store asset in order to import it on a computer not connected to the networks. Can you make me follow the package or give me a solution? Sorry for my English I pass by google translator.

 

Unfortunately,There is no other distribution methods.
please download from the asset store.

 

I am running Unity 5.1.1f free version on a Windows 7 computer. I imported your latest version of OpenCV for Unity and Facetracker from the asset store. I followed your instructions and was able to run the sample scenes from OpenCVforUnity in the Unity editor, as well as on my Android device with Unity Remote 4 (except for the web camera display). When I try to build and run on an Android device, however, I get the following error:
Assets/OpenCVForUnity/iOSforXcode/iOS_BuildPostprocessor.cs(4,7): error CS0246: The type or namespace name `UnityEditor' could not be found. Are you missing a using directive or an assembly reference?

This is strange, because I carefully checked that I have the Android platform checked in the Android Plugins files, as you describe in the installation instructions. I also checked that Player settings is set to Android, and that the Platform is set to Android in the Build Settings dialog.

 

@EnoxSoftware

double d = Imgproc.matchShapes("some mat_0", "some mat_1", Imgproc.CV_CONTOURS_MATCH_I1, 1); is always return d = 0

please some advise or sample code!

thanks for read!

 

If Unity5, Please Move “OpenCVForUnity/iOSforXcode/iOS_BuildPostprocessor.cs” to “Assets/Editor/”folder.
Description of Readme.pdf was the insufficient.It will be fixed in the next version up.

 

I made the sample scene using Imgproc.matchShapes before.
I was referring to the following pages.
http://answers.opencv.org/question/14804/matchshapes-always-returns-0/

Code (CSharp):

1. using UnityEngine;
2. using System.Collections.Generic;
3.  
4. using OpenCVForUnity;
5.  
6. namespace OpenCVForUnitySample
7. {
8.         /// <summary>
9.         /// MatchShapes Sample
10.         /// </summary>
11.         public class MatchShapesSample : MonoBehaviour
12.         {
13.      
14.                 // Use this for initialization
15.                 void Start ()
16.                 {
17.          
18.                         Texture2D imgTexture = Resources.Load ("alphabet1") as Texture2D;
19.          
20.                         Mat imgMat = new Mat (imgTexture.height, imgTexture.width, CvType.CV_8UC4);
21.  
22.                         Utils.texture2DToMat (imgTexture, imgMat);
23.                         Debug.Log ("imgMat dst ToString " + imgMat.ToString ());
24.  
25.  
26.                         Mat srcMat = new Mat ();
27.  
28.                         Imgproc.cvtColor (imgMat, srcMat, Imgproc.COLOR_RGBA2GRAY);
29.  
30.                         Imgproc.threshold (srcMat, srcMat, 0, 255, Imgproc.THRESH_BINARY);
31.  
32.  
33.                         Mat dstMat = srcMat.clone ();
34.  
35.                         Debug.Log ("srcMat ToString " + srcMat.ToString ());
36.                         Debug.Log ("dstMat ToString " + dstMat.ToString ());
37.  
38. //            Core.flip (dstMat,dstMat,-1);
39.                         Core.rectangle (dstMat, new Point (50, 50), new Point (100, 100), new Scalar (0), 5);
40.  
41.  
42.  
43.                         List<MatOfPoint> srcContours = new List<MatOfPoint> ();
44.                         Mat srcHierarchy = new Mat ();
45.  
46.                         /// Find srcContours
47.                         Imgproc.findContours (srcMat, srcContours, srcHierarchy, Imgproc.RETR_CCOMP, Imgproc.CHAIN_APPROX_NONE);
48.  
49.  
50.                         List<MatOfPoint> dstContours = new List<MatOfPoint> ();
51.                         Mat dstHierarchy = new Mat ();
52.          
53.                         /// Find dstContours
54.                         Imgproc.findContours (dstMat, dstContours, dstHierarchy, Imgproc.RETR_CCOMP, Imgproc.CHAIN_APPROX_NONE);
55.  
56.  
57.                         double returnVal = Imgproc.matchShapes (srcContours [0], dstContours [0], Imgproc.CV_CONTOURS_MATCH_I1, 0);
58.                         Debug.Log ("returnVal " + returnVal);
59.  
60.  
61.          
62.                         Texture2D texture = new Texture2D (dstMat.cols (), dstMat.rows (), TextureFormat.RGBA32, false);
63.          
64.          
65.                         Utils.matToTexture2D (dstMat, texture);
66.          
67.                         gameObject.GetComponent<Renderer> ().material.mainTexture = texture;
68.          
69.                 }
70.      
71.                 // Update is called once per frame
72.                 void Update ()
73.                 {
74.          
75.                 }
76.      
77.                 void OnGUI ()
78.                 {
79.                         float screenScale = Screen.width / 240.0f;
80.                         Matrix4x4 scaledMatrix = Matrix4x4.Scale (new Vector3 (screenScale, screenScale, screenScale));
81.                         GUI.matrix = scaledMatrix;
82.          
83.          
84.                         GUILayout.BeginVertical ();
85.                         if (GUILayout.Button ("back")) {
86.                                 Application.LoadLevel ("OpenCVForUnitySample");
87.                         }
88.          
89.          
90.                         GUILayout.EndVertical ();
91.                 }
92.         }
93. }

alphabet1.png

 

OpenCV for Unity
Released Version 1.1.6

Version changes1.1.6
[Common]Fix FaceRecognizer.cs(Compile Error had happened only in the case of Unity4 Non Pro License).

 

It is a good idea.
It might be added in future versions.

 

Open CV Unity can you use Optical character recognition (OCR)

 

My bad, I've updated the post, KNN.find_nearest(testMat,10,predictMat,neighborMat,distMat); <-- 10 should be 1, problem still exists. How do you get the detailed error information? it may help...

 

There is an example of the OCR using CvSVM class.
https://github.com/MasteringOpenCV/code/tree/master/Chapter5_NumberPlateRecognition
Perhaps,this example can be converted in the "OpenCV for Unity".

 

Error message of C++ is displayed by LogCat of Android or Console of Xcode if I turn on check box of "Development Build".
"printf" is used to output an error in Win and Mac Standalone.
but,I do not know how to display in the case of a run on the Editor.