[RELEASED] OpenCV for Unity

My English may also be a little bad

Thank you for your help. Thanks to your new markers I think I found why my markers had a problems. If I will find exact reason I will share it here.

 

I found a pretty interestig bug. Not sure if it's only my unity instance problem.

I'm trying to write my own marker manager which will take images from file, binarize it etc.
I'm loading resources dynamically

Code (CSharp):

1.     Object[] markers = Resources.LoadAll ("Markers");
2.         Object[] gamePrefabs = Resources.LoadAll ("GameObjects");
3.      
4.         Mat tempMat = new Mat (256, 256, CvType.CV_64FC1);
5.         int[,] bitMatrix = new int[5,5];
6.  
7. //This part is for testing dynamic load of textures it works correctly
8.         GameObject go = GameObject.CreatePrimitive(PrimitiveType.Plane);
9.         Renderer rend = go.GetComponent<Renderer>();
10.         rend.material.mainTexture = markers [0] as Texture;
11.  
12.  
13.         foreach (Object marker in markers) {
14. //And here it crashes
15.             Utils.textureToMat(rend.material.mainTexture, tempMat);
16. }

The part with rendering is just to test if it's all ok with loaded image. I can display it on plane but when I try to use
Utils.textureToMat(rend.material.mainTexture, tempMat);

It crashes Unity.

@Edit:

The main problem is this method
private static extern void OpenCVForUnity_LowLevelTextureToMat (IntPtr texPtr, int texWidth, int texHeight, IntPtr mat);

 

Please try the following code.

Code (CSharp):

1. Object[] markers = Resources.LoadAll ("Markers");
2.         Object[] gamePrefabs = Resources.LoadAll ("GameObjects");
3.    
4.         //Mat tempMat =new Mat (256, 256, CvType.CV_64FC1);
5.         Mat tempMat = new Mat (256, 256, CvType.CV_8UC1);
6.         int[,] bitMatrix = new int[5,5];
7. //This part is for testing dynamic load of textures it works correctly
8.         GameObject go = GameObject.CreatePrimitive(PrimitiveType.Plane);
9.         Renderer rend = go.GetComponent<Renderer>();
10.  
11.         //rend.material.mainTexture = markers [0] as Texture;
12.         rend.material.mainTexture = markers [0] as Texture2D;
13.         foreach (Object marker in markers) {
14. //And here it crashes
15.             //Utils.textureToMat(rend.material.mainTexture, tempMat);
16.             Utils.texture2DToMat(rend.material.mainTexture, tempMat);
17. }

 

I’m trying to get a second camera to follow the rectangle drawn from face recognition
So if a face is detected it shows a up-close view of the persons face, the camera is following but way off the target.

Code (CSharp):

1.  
2.                     //loop and place rectagles
3.                     for (int i = 0; i < rects.Length; i++)
4.                     {
5.                        Core.rectangle(rgbaMat, new Point(rects[i].x, rects[i].y), new Point(rects[i].x + rects[i].width, rects[i].y + rects[i].height), new Scalar(255, 0, 0, 255), 2);
6.  
7.                        Vector3 vectorWorldSpace = new Vector3(rects[i].x , rects[i].y , 0f);
8.                        Vector3 pointWorldSpace = Person1.transform.InverseTransformDirection(vectorWorldSpace);
9.                         Person1.transform.position = pointWorldSpace;
10.                     }

 

Released Version 1.1.2

Version changes
1.1.2
[Common]Fix the direction of rotation of the mat that is converted from WebCamTexture.

 

Hi, any plans to support Web platform?

 

Since the Unity Web Player does not support the native plugin, "OpenCV for Unity" does not support "WebPlayer　Platform".

 

Hello,

This is a very nice asset, thanks for making it.

I have read the previous posts on changing the marker.

Is there a way to have multiple markers in one scene displaying different game assets?

 

Hi,

Where WebCamTextureMarkerBasedARSample.cs load the Axes_X, Axes_Y, Axes_Z and cube?

Thx

 

Unfortunately,The current sample code can detect only a single marker.

 

AR object is placed on the scene, it will be displayed by the ARCamera When the marker is detected.

 

I've once again just made a new unity projected added the OpenCV framework, published it to Xcode and got it working perfectly on my phone and iPad however when i click submit to the app store i get the same error above

I tried it on a new Xcode project without the openCV framework just to see if its Xcode however any other project varifies fine,

Really stuck on this need to upload an app for a client ASAP


Here is the log
<IDEDistributionIssue: severity(error), error(Error Domain=ITunesSoftwareServiceErrorDomain Code=-21013 \"The archive is invalid. /var/folders/1_/syt94vl56ts80r8cl5tk1n280000gn/T/XCodeDistPipeline.MED/Unity-iPhone.ipa does not exist.\" UserInfo=0x7f8f09c7f550 {NSLocalizedDescription=The archive is invalid. /var/folders/1_/syt94vl56ts80r8cl5tk1n280000gn/T/XCodeDistPipeline.MED/Unity-iPhone.ipa does not exist., NSLocalizedFailureReason=Unable to validate your application.})>"
)
2015-05-11 11:11:35 +0000 [MT] Canceled distribution assistant


Any help anyone?

 

Thank you for a report.

It may be "Codesign faild Error".
Please check the "Code Signing" setting in Xcode.


I was able without problems Validate in my environment.

OSX 9.6.4
Unity 5.0.0f4
Xcode 6.1.1

1. Open new project in Unity.
2. Import "OpenCVForUnity 1.1.2" from AssetStore and Setup.
3. Export Xcode project.(BuildSettings-Build)
4. Setup "Code Signing".
5. Archive and Validate in Xcode.

Validate Error will occur if it contains "opencvforunity.bundle” is in the archive. "OpenCVForUnity/Plugins/opencvforunity.bundle" is for mac Standalone platform.
Please set as shown in the following screen shot.I'm sorry, it was not written to Readme.pdf.

 

Thank you for making this plugin, it is really well done.

1) I am confused about the API, is all of opencv available or only a subset which you expose through your API listed here? http://enoxsoftware.github.io/OpenCVForUnity/doc/html/annotated.html

I ask because I want to use the VideoWriter and I think I see that it is part of the opencv for ios?
http://docs.opencv.org/modules/highgui/doc/reading_and_writing_images_and_video.html

And I do not see it listed in your API,

2) what is the best way I could save or render Video to a file?

Thanks again for making this great plugin

 

Hallo,

hello,
I have something on the internet reading about the further processing of the detected faces or other objects.
Do you have a good tip which software or service I can use, or should I build a database itself?

Question 2: Is it possible the marker with the distance of the objects in the video to make larger. A person comes to the camera and the marker is greater with each step. (Often seen in movies)

THX

 

Do you have an example of FaceRecognizer?

 

1)API that can be used in "OpenCV for Unity" can be found in the following pages.
VideoWriter class is not implemented.

2)Sorry,I do not know whether it is possible in Unity.

 

I have question how do we doing on auto focus to disable?

 

Hi,
Thanks for the good work!
Just one dummy question: in the 2d features examples, I always have 500 matches; whatever the images I use.
Is there a way to get only the "good" matches and not the best. Threshold?

Thanks,
Michaël

 

Currently, there is no demo code.
Because ”OpenCVforUnity” is almost a clone of "OpenCV java", you can be in reference to the example of "OpenCV java".

 

Thank you for the reply.
I'm getting a object is null, I'm guessing we need the "FisherFaceRecognizer" or what I'm I missing:

Declare FaceRecognizer:
FaceRecognizer FaceRecognizers;

Loop to find faces and crop:

OpenCVForUnity.Rect[] rects = faces.toArray();
for (int i = 0; i < rects.Length; i++)
{
OpenCVForUnity.Rect Headarea = new OpenCVForUnity.Rect(rects[0].x, rects[0].y, rects[0].width, rects[0].height);
crop = imgMat.submat(Headarea);
if (StoredFaces.ToArray().Length != 0)
{
FaceRecognizers.train(StoredFaces, crop);
FaceRecognizers.predict(StoredFaces, predicted_label, predicted_confidence);
FaceRecognizers.save("Foundfaces.XML");
}
}

 

Q1:
I know fix the direction of rotation of the mat in the V1.1.2. But the scene"FaceTrackerARSample" of the example"FaceTrackerSample" has also same problem.I hope can show me how fix it.Thanks.

Q2:
[Unity5.0.1 + OpenCVforUnity1.1.2]
When I Build in the Xcode6.3.1,the link erroe occurs.I had set it follow the ReadMe.pdf step by step.I don't know why.

 

did you added this on your current update?

 

I think that there is no way to disable the autofocus of WebCamTexture.

 

A1:
Error reporting Thank you.
Please be replaced with the following code.
It will be fixed in the next version.

Code (CSharp):

1.  
2. //flip to correct direction.
3.                                 if (webCamTexture.videoVerticallyMirrored) {
4.                                         if (webCamDevice.isFrontFacing) {
5.                                                 if (webCamTexture.videoRotationAngle == 0) {
6.                                                         Core.flip (rgbaMat, rgbaMat, -1);
7.                                                 } else if (webCamTexture.videoRotationAngle == 180) {
8.                                                         Core.flip (rgbaMat, rgbaMat, 0);
9.                                                 }
10.                                         } else {
11.                                                 if (webCamTexture.videoRotationAngle == 0) {
12.                                                        
13.                                                 } else if (webCamTexture.videoRotationAngle == 180) {
14.                                                         Core.flip (rgbaMat, rgbaMat, 1);
15.                                                 }
16.                                         }
17.                                 } else {
18.                                         if (webCamDevice.isFrontFacing) {
19.                                                 if (webCamTexture.videoRotationAngle == 0) {
20.                                                         Core.flip (rgbaMat, rgbaMat, 1);
21.                                                 } else if (webCamTexture.videoRotationAngle == 180) {
22.                                                         Core.flip (rgbaMat, rgbaMat, 0);
23.                                                 }
24.                                         } else {
25.                                                 if (webCamTexture.videoRotationAngle == 0) {
26.                                                        
27.                                                 } else if (webCamTexture.videoRotationAngle == 180) {
28.                                                         Core.flip (rgbaMat, rgbaMat, -1);
29.                                                 }
30.                                         }
31.                                 }
32.  

A2:
Did you try the following procedures?
Link “OpenCVForUnity/iOS for Xcode/opencv2.framework” to Xcode project. (in Xcode project. Build Phases > Link Binary with Libraries > Add opencv2.framework . When a link error occurs, please add framework after delete once. recommend to use PostprocessBuildPlayer.)

 

Hi EnoxSoftware,

I am using unity 4.6.1f1. Can I test the samples from unity editor ? I opened every scenes, and i always get this error :

ArgumentException: The output Mat object has to be of the same size
OpenCVForUnity.Utils.webCamTextureToMat (UnityEngine.WebCamTexture webCamTexture, OpenCVForUnity.Mat mat, UnityEngine.Color32[] bufferColors) (at Assets/OpenCVForUnity/org/opencv/unity/Utils.cs:517)
OpenCVForUnitySample.WebCamTextureDetectFaceSample.Update () (at Assets/OpenCVForUnity/Samples/WebCamTextureDetectFaceSample/WebCamTextureDetectFaceSample.cs:193)

or

ArgumentException: The output Mat object has to be of the same size
OpenCVForUnity.Utils.texture2DToMat (UnityEngine.Texture2D texture2D, OpenCVForUnity.Mat mat) (at Assets/OpenCVForUnity/org/opencv/unity/Utils.cs:433)
OpenCVForUnitySample.DrawingSample.Start () (at Assets/OpenCVForUnity/Samples/DrawingSample/DrawingSample.cs:21)

It seems that the code in the Mat.cs does not get called ? for example,
Mat.rows() code :

int retVal = core_Mat_n_1rows (nativeObj);

return retVal;

is wrapped in this macro :
#if UNITY_PRO_LICENSE || ((UNITY_ANDROID || UNITY_IPHONE) && !UNITY_EDITOR) || !(UNITY_4_5 || UNITY_4_6)

So. does it mean you only support unity 5? I read in your asset store description that you support unity 4.5.5 or higher.

 

Thanks reply.I know this code change,but I can't fix the "ARCamera".Maybe I must wait your next version of "FaceTrackerARSample".The Q2,I had followed the steps but use PostprocessBuildPlayer.It also occurs error.I don't know why.I used old version in unity4.5 and xcode 5.x,it work well.I use unity5.0.1,I have not use Unity5.0.0.

 

Q2.
Imported "OpenCVForUnity/iOS for Xcode/opencv2.framework" file from AssetStore in Unity4 is not available in Unity5.
Please re-import the "OpenCVForUnity" from AssetStore in Unity5.

 

OpenCV for Unity
Released Version 1.1.3

Version changes
1.1.3
[Common] Fix SampleScene.
[Common] Change Property of Platform Dependent Compilation from UNITY_IPHONE to UNITY_IOS.

 

Hi Enox, could you response to my question (http://forum.unity3d.com/threads/released-opencv-for-unity.277080/page-4#post-2118653) ?

 

This error occurs when you have failed to set up the plugin or you do not have a Pro license of Unity.
To run the NativePlugin in Editor in Unity4, you need the Pro license.
If you do not have a Pro license, please try to run in Unity5.

 

Thank you for bug report.
I'm sorry.The subclass of FaceRecognizer was not implemented in careless mistake.
It will be fixed in the next version up.

 

I know that OpenCV does not support DX-based devices, but do you have a plan to extend your plugin so as to support WP8 devices and Surface tablets?

 

Ah i see.. Thanks

 

@EnoxSoftware ?

 

I am going to support WindowsStoreApp and WindowsPhone8.1 in the version that implemented OpenCV3.0.0 under development now.

 

Great!!! ETA? (in case you know it)

 

Enox has amazing plugins and love their support!!!!!!!!!

 

Thanks for releasing this, great plugin.

I'm using this in a game for face recognition. It work's fine on my Mac, and my iPhone 6, but when I try it on a new HTC One Android device the FPS drops to 8 when I call detectMultiScale. It even runs better on my old iPad 3. I would have expected Android to run better.

I've tried tweaking the scale, minSize, maxSize, reducing the size of the webcam image but this only makes small improvements. Is there a specific setting that will kick the Android device into life?

 

Hello, i need to know, can this plugin detect hand movement?
if the answer is yes, how many hands?

 

Q2:I have solved it.

Now I have new problems.
Q3:[FaceTracker Sample 1.0.2]
I know you fix the direction of rotation in OpenCV for Unity1.1.2.But this not fix it at FaceTracker Sample.By the way,set Orientation "Portrait" on Play Setting,direction of rotation is not well.When run in FaceTrackerARSample scene,the "Landscape Left" is well,but "Portrait" is not.Maybe just work for Landscape Left.

Q4:Use face tracker running vuforia with OpenCv for Unity
I use vuforia QCAR develop AR app.The face tracker run well on Android device.But on ios device,the WebCamTexture is not well,the camera not work.Maybe the vuforia use WebCamTexture in its framework.I hope you can show me how solve it.Maybe you have a good solution.Of course,don't use WebCamTexture and use ARCamera of vuforia,change the code of FaceTrackerARSample is a solution.But it is big job.It is hard for me.

 

Can this OpenCV For Unity used on Web application?
Is there any demo for web?

 

detectMultiScale() in Android seems to slow processing speed compared to other platforms.
By processing the detection of the face in another thread, I think there is a possibility that the processing speed can be further improved.

 

Probably I think that it is possible.
http://simena86.github.io/blog/2013/08/12/hand-tracking-and-recognition-with-opencv/

 

Q3:
Thank you for a report.
Orientaion of ProjectSetting is overwritten "Landscape Left" when importing "FaceTrackerARSample" from AssetStore.

Q4:
Please refer to the following sample.
https://github.com/EnoxSoftware/VoforiaWithOpenCVForUnitySample

 

Since the Unity Web Player does not support the native plugin, "OpenCV for Unity" does not support "WebPlayer　Platform".

 

OpenCV for Unity
Released Version 1.1.4

Version changes
1.1.4
[Common] Add FaceRecognizer subclass.
[Common] Add FaceRecognizerSampleScene.
[Common] Fix SampleScene.

 

@EnoxSoftware: I've sent you a PM.

 

help me please~~~~

Utils.textureToMat could work on pc but could not work on android device.
my code:

Texture aaa = tmp.GetTexture(0);
Mat bb = new Mat (aaa.height,aaa.width, CvType.CV_8UC4);
Utils.textureToMat(aaa, bb);
Highgui.imwrite (path +"/9t.jpg",bb);
double[] de;
de = bb.get (100, 100);
foreach (double xx1 in de) {
Debug.Log (xx1+"======rgb======");
//here i can get RGB value on pc,but cant get it on android device,why???
}

 

When using the Utils.textureToMat(), "aaa" of Texture must be "4byte per 1pixel Format".
Because Utils.textureToMat() is using Low-level Native Plugin Interface,Internal format of Texture in Android must be the 4byte per 1pixel.