Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

WebCamTexture rotated and flipped on iPhone when applied to renderer.material.texture

Discussion in 'iOS and tvOS' started by saline13, Mar 7, 2012.

  1. saline13

    saline13

    Joined:
    Feb 26, 2012
    Posts:
    2
    Hi,

    I have created a WebCamTexture webCamTexture = new WebCamTexture( ... ) and then applied this texture to a plane.

    renderer.material.mainTexture = webCamTexture.
    This works in the editor as expected!

    On the iPhone the result texture is rotated 90 degrees and flipped instead of being like the screen in landscape. It's like it's storing the result texture in portrait mode.

    In the editor it works ok but on the iPhone or iPad the result texture is rotated and flipped.

    Anyone know why this could happen? Is it a bug on iOS? Am I missing something?

    Any help greatly appreciated.
     
  2. Aizee421

    Aizee421

    Joined:
    Nov 16, 2011
    Posts:
    5
    I'm having the same issue on the iPad. I've tried playing with rotating the plane and fixing the orientations but I haven't found a solution. BUMP
     
  3. J_P_

    J_P_

    Joined:
    Jan 9, 2010
    Posts:
    1,027
    That's just how the iOS camera's raw data comes in afaik. Easiest thing to do might be to create a plane and mirror the UVs there. It'd be faster than manipulating it as an image.
     
  4. pomedas

    pomedas

    Joined:
    Jun 15, 2011
    Posts:
    7
    You can set a negative scale factor to the object with the webcamtexture
     
    tedneffv likes this.
  5. reistiago

    reistiago

    Joined:
    Jun 11, 2012
    Posts:
    1
    Hi,

    I'm new to Unity so I'm not sure how to do that. Could you please provide a sample on how to do that?

    Still trying to get my head around Unity.

    Thanks
     
  6. Kragh

    Kragh

    Joined:
    Jan 22, 2008
    Posts:
    657
    I have the same problem, and as others in the Unity Answers, I don't understand we are so few talking about this. And I do not have the choise of just mirroring Uvs and what not, as I'm using the raw webcam pixel data for calculations. Is this really just how it is? Is anyone aware of a fix in the pipeline?

    I'm using an iPad 2 for testing.
     
  7. prime31

    prime31

    Joined:
    Oct 9, 2008
    Posts:
    6,426
    @kragh, there is nothing broken to fix. The camera returns the raw data exactly as it sees it. If you are using the raw data just take into account that the image is flipped in your calculations.
     
  8. Kragh

    Kragh

    Joined:
    Jan 22, 2008
    Posts:
    657
    But it is not just flipped, it is rotated as well, 90 degrees.
    So it is squeezed as well, because the webcam renders it in 4:3 ratio, and then display it rotated.
    Why would that not be considered a bug? Or at least problematic? Ofcourse, intended behaviors are not "bugs", but an intended behavior can still be wrong.
    The point is, my scripts are doing enough work as is (Augmented reality is heavy enough, without having to do such extra data manipulation), and it is just a pain in my mind having to take every pixel, and flip/rotate it before I can use it for anything. Seems foolish.
    I don't care what the raw data is, Unity has provided an API, which, to make the Unity product consistant, should offer the same result across platforms. Imagine everytime you coded something, you had to code special cases for different platforms (well, you already havce to do that, but that is more to do with optimization, not the basic math).
    IF the iphone/ipad does indeed deliver such data to Unity, the rotation/flipping should be done, before we get the data as users of the Webcam API.
    In my very humble opinion.
    Arguing: "Well, there's this workaround" is never a justification.

    If anybody have a very cheap way of rotating every pixel in the data (I won't display the actual image, I just need to know if I ask for pixel(x,y) I'll actually get the correct pixel). For now my solution is to simply throw every request I do for pixels through a simple rotation operation, before asking for the actual pixel.
     
    AngryCreeper and Cliwo like this.
  9. sonicviz

    sonicviz

    Joined:
    May 19, 2009
    Posts:
    1,051
    Has anyone found a good solution to this?
     
  10. Alexey

    Alexey

    Unity Technologies

    Joined:
    May 10, 2010
    Posts:
    1,623
  11. sonicviz

    sonicviz

    Joined:
    May 19, 2009
    Posts:
    1,051

    Hi Alexy,
    Thanks, I've been playing with the videoRotationAngle but still not quite understanding it.
    Unless I'm wrong the code in the docs seems to flip my texture 180 degrees!

    Great to hear there is more stuff coming in 4.2!
    Ideally it should work the same as the iOS/Droid cameras, as in when you rotate the device physically the image maintains its physically locked orientation, not rotating the image with the device. My 2 yen anyway.

    Edit: On iOS the Back Camera works as the native camera does ie: if you rotate the ipad the image maintains its orientation. However, the front camera doesn't it (at least on my iPad2) because if you rotate the ipad then the image rotates with it which is completely loony!

    I'm waiting!;-)
     
    Last edited: Jun 26, 2013
  12. Ravensburg3r

    Ravensburg3r

    Joined:
    May 23, 2013
    Posts:
    1
    I found next solution.

    1) Create new shader:

    Code (csharp):
    1. Shader "Custome/Web Camera Shader"
    2. {
    3.     Properties
    4.     {
    5.         _MainTex ( "Main Texture", 2D ) = "white" {}
    6.     }
    7.    
    8.     SubShader
    9.     {      
    10.         Pass
    11.         {
    12.             CGPROGRAM
    13.            
    14.             #pragma vertex vert
    15.             #pragma fragment frag
    16.            
    17.             uniform sampler2D _MainTex;
    18.             uniform float4x4 _Rotation;
    19.            
    20.             struct vertexInput
    21.             {
    22.                 float4 vertex : POSITION;
    23.                 float4 texcoord : TEXCOORD0;
    24.             };
    25.            
    26.             struct vertexOutput
    27.             {
    28.                 float4 pos : SV_POSITION;
    29.                 half2 uv : TEXCOORD0;
    30.             };
    31.            
    32.            
    33.             vertexOutput vert(vertexInput v)
    34.             {
    35.                 vertexOutput o;
    36.                
    37.                 float4 newPosition = mul( UNITY_MATRIX_MVP, mul(_Rotation, v.vertex) );
    38.                
    39.                 o.pos = newPosition;
    40.                 o.uv = v.texcoord;
    41.                
    42.                 return o;
    43.             }
    44.            
    45.             float4 frag(vertexOutput i) : COLOR
    46.             {
    47.                 return tex2D( _MainTex, i.uv );
    48.             }
    49.            
    50.             ENDCG
    51.         }
    52.     }
    53.     Fallback "Diffuse"
    54. }
    2) Apply shader to material of a plane you want to display WebCamTexture

    3) Set matrix for the material (in Start() method):

    Code (csharp):
    1. Quaternion rotation = Quaternion.Euler (0, 90, 0);
    2. Matrix4x4 rotationMatrix = Matrix4x4.TRS (Vector3.zero, rotation, new Vector3(1, 1, 1));
    3. gameObject.renderer.material.SetMatrix ("_Rotation", rotationMatrix);
    4) Set WebCamTexture to material texture like this (in Update() method):
    Code (csharp):
    1. cameraScreen.renderer.material.mainTexture = webCamera.texture;
     
  13. raja1250

    raja1250

    Joined:
    Mar 6, 2014
    Posts:
    6
    Just rotate the maincamera along the Z axis to the 90 degrees tthats all.
     
  14. Doodums

    Doodums

    Joined:
    Feb 19, 2011
    Posts:
    25
    I agree 100% with what @Kragh said above.

    "Just rotate it" is a terrible solution. This is not just an issue of displaying, but storing as well. If I want to take the picture, save it on the server and display it elsewhere, must I just rotate it there as well? I must not only write code to display the camera correctly within Unity, but also to take the raw pixels I get back and manually adjust them to get the correct pixels? This is not regarded as a bug?

    What is being said to me is, if I want to write code to take a picture for Android and iOS, instead of me just being able to call photo.SetPixels(wct.GetPixels()); as the method implies I can do, I have to do the following:

    Code (CSharp):
    1. if Android
    2.     DefaultCameraBehaviour();
    3. if iOS
    4.     If Camera is front facing
    5.         SpecialFrontFacingIOSCode();
    6.     if Camera is back facing
    7.         SpecialBackFacingIOSCode();
    You are already encapsulating the behavior in a 'GetPixels' method. Why can't that method handle this behavior for us, if you are already aware of it?

    I can't believe this issue is more than 3 years old, and is still in this state.
     
    Last edited: Dec 17, 2015
  15. povilas

    povilas

    Unity Technologies

    Joined:
    Jan 28, 2014
    Posts:
    427
    Why can't you do something like the following for all platforms?

    Code (CSharp):
    1. photo.SetPixels(wct.GetPixels());
    2.  
    3. if (wct.videoVerticallyMirrored)
    4.     photo = FlipTexture(photo);
    5. if (wct.videoRotationAngle != 0)
    6.     photo = RotateTexture(photo, wct.videoRotationAngle);
    I agree that this is not optimal, because you need to implement FlipTexture and RotateTexture yourself, but it's also not the end of the world as you imply.
     
  16. stevecox

    stevecox

    Joined:
    Oct 28, 2015
    Posts:
    1
    WebCamTexture.videoVerticallyMirrored doesn't appear to be working properly on the iPhone 6. It's returning false while the image is clearly flipped vertically. Any chance this could be fixed?

    Update: seems I need to read the documentation more carefully. In case others run into this problem, the docs say "Please note, that this will query platform-specific part, which might be not ready before actual video feed started; so it is not enough to call it once after play." Once I waited, the value was correct.

    http://docs.unity3d.com/ScriptReference/WebCamTexture-videoVerticallyMirrored.html
     
    Last edited: Dec 19, 2015
  17. Doodums

    Doodums

    Joined:
    Feb 19, 2011
    Posts:
    25
    Thanks for your reply povilas.

    I agree I made it worse than it actually is, but end of day frustration kicked in and "Just rotate it" answers rubbed me the wrong way.

    Thanks for your solution, which is better than what I threw together on Friday. I will update my code on Monday and link it here for people who run across this in the future.

    Any plans on changing this soon, or should we expect this behavior to remain for quite a while?
     
  18. Alexey

    Alexey

    Unity Technologies

    Joined:
    May 10, 2010
    Posts:
    1,623
    >>Any plans on changing this soon, or should we expect this behavior to remain for quite a while?
    well, not really (unless smth change deep inside ios)
    to put things into perspective so you understand WHY:
    in older unity version we were getting web cam data, read bytes from here, and upload them into texture. If you think this is crazy it is because it is - perf was abysmal
    now we switched to use CVTextureCache which more or less gives us gles/metal texture that points to data directly (well if you need more details go read docs for CVOpenGLESTextureCache and CVMetalTextureCache) so that means we get NO perf hit. That brought us to two caveats:
    1. at some point we were doing readable vs non-readable on webcamtexture, as extracting data becomes a bit more complicated in that case - thankfully we managed to get rid of it so now you can have best of two worlds
    2. we cannot do anything with texture provided to us and OS also do nothing (well that's supposed to be no-perf-hit fast-path to access hw) so you are hit by these rotate/mirror, but i would say that is small price to pay for having video capture + playback at essentially no cost
     
  19. 00christian00

    00christian00

    Joined:
    Jul 22, 2012
    Posts:
    1,035
    Alexey, can you tell me what's the proper way to query videoVerticallyMirrored?
    Since it is not guaranteed to be ready at video start, do we need to continuosly check for the value in a coroutine or update?

    Since now it is using the OS directly,why not add camera focus and exposure control?
     
  20. Alexey

    Alexey

    Unity Technologies

    Joined:
    May 10, 2010
    Posts:
    1,623
    >>do we need to continuosly check for the value in a coroutine or update?
    yeah, thats the plan - if you saw samples they all just query rotation/mirror and recreate transform. I agree would be easier to do smth like waitForVideoReady and tweak once, but for now i dont see how we would implement it cleanly

    >>why not add camera focus and exposure control
    simply because you are talking about ios-specific things - we dont like to add lots of platform specific things unless really needed. but on the other hand - you can do it in objc yourself ;-). Or create bug report (feature request) so i dont forget to look into possibility at some point (just please describe use cases)
     
    00christian00 likes this.
  21. subjectZero

    subjectZero

    Joined:
    Oct 19, 2014
    Posts:
    37
    rawImageARF.aspectRatio is giving me an issue. does not contain definition for "aspectRatio"

    this is driving me mad. I've tried all combination of getting the component and still does not pick it up. please assist.
     
  22. subjectZero

    subjectZero

    Joined:
    Oct 19, 2014
    Posts:
    37
    Thanks for getting back to me, i did this, like so and got it working.

    Code (CSharp):
    1. AspectRatioFitter rawImageARF = rawimage.GetComponent<AspectRatioFitter>();
    2.  
     
  23. georgehbr

    georgehbr

    Joined:
    Sep 19, 2015
    Posts:
    2
  24. Alexey

    Alexey

    Unity Technologies

    Joined:
    May 10, 2010
    Posts:
    1,623
    >>So I'm pretty sure at this point that "verticallyMirrored" means that a rotation about the horizontal axis is needed.
    well, yeah. If i will be totally precise this means that order of lines in image is opposite to what gl expects (as in lines go top-to-bottom as opposed to gl bottom-to-top, or vice versa i prefer to forget all gl and never get back ;-))
    so in some regard it is indeed equivalent to "rotate around horz axis by 180", though it is NOT z axis, but rather x
    so yes, mirror is totally independent on image rotation itself. If you want you may think about it like this:
    1. camera have "orientation" which is not same as device orientation, e.g. camera might be "portrait" only so if you hold phone on landscape image will be rotated by 90 degrees (because camera itself is "rotated")
    2. hw gives image rows/lines in order opposite of gles, so you need to flip it vertically.
    Now while in general case things would be complicated here we have two independent rotations around orthogonal axises so *i think* you can totally combine them together (or you can flip uv rect too)
     
  25. vinay_vidhani

    vinay_vidhani

    Joined:
    Oct 27, 2016
    Posts:
    10


    I am working with region capture, and the problem I am facing that is my application is working on some of the android phone not all example my application is not working on Samsung SIII, SIV and SVI I dont know about others. And In case of IOS devices, On some devices my application is working fine but for some devices I am reversing matrix. Can u please let me know why this? Please
     
  26. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,969
    From working extensively on NatCam and facing the perils of orientation (and now autorotation), I know a bit of how this works.
    Actually, the camera is always in the natural orientation which is landscape left. Because the app might be in a different orientation, the difference between these is the cause of any apparent rotation.
    From my experience, this only holds for the CVMetalTextureCache on iOS. Flipping here can be done row by row.
     
  27. Janoschii

    Janoschii

    Joined:
    Oct 15, 2014
    Posts:
    9
    Am I right, that I can ignore the videoVerticallyMirrored property, if I use GetPixels32() method to create a new texture from a WebCamTexture?
    I get the flipping and rotating of the WebCamTexture on display to run on iOS and Android.
    But if I use myWebCamTexture.GetPixels32() the resulting texture seems to have the mirroring corrected. So it seems that I don't need to check if videoVerticalMirrored is set true or false if I use myWebCamTexture.GetPixels32().

    Version: Unity 5.4.5p5
     
    Last edited: Jan 24, 2018
  28. J3-Gaming

    J3-Gaming

    Joined:
    Dec 23, 2010
    Posts:
    55
    Wow I found a time machine of almost 6 years, and this STILL happens?
    I am also running into this issue. I would like to keep the information stored as a WebCamTexture (Need to read QR codes)

    Anyone have a solution to this? If I get something working I'll post for everyone else, and those who find this in another 5 years.
     
    gcscreations and Saicopate like this.
  29. gcscreations

    gcscreations

    Joined:
    Mar 8, 2018
    Posts:
    2
    Here also same problem bro....
     
  30. User340

    User340

    Joined:
    Feb 28, 2007
    Posts:
    3,001
    I am interested in a solution here as well.
     
  31. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,969
    On iOS, the WebCamTexture might be flipped, but the data itself (using GetPixels32) is not. This is an important detail to keep in mind.
     
  32. Motts182

    Motts182

    Joined:
    Feb 27, 2017
    Posts:
    2
    its still more bad, the front camera right now is -90 save photo and back cam ´90 ... the better solution is rotate the texture when u use a different img (?) unity gg
     
  33. yiningliangpwc

    yiningliangpwc

    Joined:
    Jan 10, 2019
    Posts:
    1
    It's 9102 and we still need to rotate the webcam texture from an iPhone front camera in portrait mode :) From that respect, Unity is doing a great job by keeping the API consistent.
     
    AngryCreeper likes this.
  34. urfx

    urfx

    Joined:
    Dec 12, 2013
    Posts:
    12
    lol 9102
     
    Lanre likes this.
  35. andrew78041

    andrew78041

    Joined:
    May 10, 2020
    Posts:
    1
    You can use this.
    Screen.orientation = ScreenOrientation.LandscapeLeft;
     
  36. J3-Gaming

    J3-Gaming

    Joined:
    Dec 23, 2010
    Posts:
    55
    2020 calling. Still an issue
     
  37. Luiz_Thiago

    Luiz_Thiago

    Joined:
    Feb 27, 2013
    Posts:
    32
    Holy crap! Same here... any way to rotate it?
     
  38. moriarty83

    moriarty83

    Joined:
    Mar 29, 2019
    Posts:
    1
    The tutorial N3K EN on YouTube is very close to the solution:



    I'm still tinkering with the code to get the rawimage to fill the screen on my phone. So far

    Code (CSharp):
    1. rawimage.rectTransform.localEulerAngles = new Vector3(0, 0, -orient);
    I'm still having trouble getting the camera image to fill the screen but it's close. Hope this helps crack the case!
     
  39. HasithaCJ

    HasithaCJ

    Joined:
    May 20, 2017
    Posts:
    11
    I had the same issue when tried to get camera input for RawTexture in android potrait mode. So I dcided to move on with a quad.

    I just modified this code to work on orthograhic camera.

    Hope this will help you.

    Code (CSharp):
    1. using UnityEngine;
    2.  
    3. public class CameraInputManager : MonoBehaviour{
    4.  
    5. private WebCamTexture camTexture;
    6. [SerializeField] private GameObject BackgroundQuad;
    7. private Quaternion baseRotation;
    8.  
    9. void Start()
    10. {
    11.    WebCamDevice[] devices = WebCamTexture.devices;
    12.  
    13.    if (devices.Length ==0)
    14.    {
    15.        Debug.Log("No camera");
    16.    
    17.        return;
    18.    }
    19.        
    20.    for (int i = 0; i < devices.Length; i++)
    21.    {      
    22.        if (devices[i].isFrontFacing) {
    23.        
    24.            camTexture = new WebCamTexture(devices[i].name,  Screen.height, Screen.width );
    25.        
    26.         }
    27.    }
    28.  
    29.  
    30.    if (camTexture == null) {
    31.        Debug.Log("Unable to find cam");
    32.    
    33.        return;
    34.    }
    35.  
    36.    baseRotation = transform.rotation;
    37.  
    38.    BackgroundQuad.GetComponent<Renderer>().material.mainTexture = camTexture;
    39.  
    40.    camTexture.Play();    
    41.        
    42.    CalculateBackgroundQuad();
    43. }
    44. void CalculateBackgroundQuad() {
    45.     Camera cam = Camera.main;
    46.     float ScreenRatio = (float)Screen.width / (float)Screen.height;
    47.  
    48.     float videoRotationAngle = camTexture.videoRotationAngle;
    49.  
    50.     BackgroundQuad.transform.localRotation = baseRotation * Quaternion.AngleAxis(camTexture.videoRotationAngle, Vector3.forward);
    51.     float fixedHeight =  cam.orthographicSize*2;
    52.  
    53.     Vector3 QuadScale = new Vector3(1f, fixedHeight, 1f);
    54.     float TextureRatio;
    55.  
    56.     //adjust the scaling for portrait Mode & Landscape Mode
    57.     if (videoRotationAngle == 0 || videoRotationAngle == 180) {
    58.         //landscape mode
    59.         TextureRatio = (float)(camTexture.width) / (float)(camTexture.height);
    60.         if (ScreenRatio > TextureRatio) {
    61.             float SH = ScreenRatio / TextureRatio;
    62.             float TW = TextureRatio * fixedHeight * SH;
    63.             float TH = fixedHeight * (camTexture.videoVerticallyMirrored ? -1 : 1) * SH;
    64.             QuadScale = new Vector3(TW, TH, 1f);
    65.         } else {
    66.             float TW = TextureRatio * fixedHeight;
    67.             QuadScale = new Vector3(TW, fixedHeight * (camTexture.videoVerticallyMirrored ? -1 : 1), 1f);
    68.         }
    69.     } else {
    70.         //portrait mode
    71.         TextureRatio = (float)(camTexture.height) / (float)(camTexture.width);
    72.         if (ScreenRatio > TextureRatio) {
    73.             float SH = ScreenRatio / TextureRatio;
    74.             float TW = fixedHeight * -1f * SH;
    75.             float TH = TW * (camTexture.videoVerticallyMirrored ? 1 : -1) * SH;
    76.             QuadScale = new Vector3(TW, TH, 1f);
    77.         } else {
    78.             float TW = TextureRatio * fixedHeight;
    79.             QuadScale = new Vector3(fixedHeight * -1f, TW * (camTexture.videoVerticallyMirrored ? 1 : -1), 1f);
    80.         }
    81.     }
    82.     BackgroundQuad.transform.localScale = QuadScale;
    83.  
    84. }
    85.  
    86. }