Search Unity

Latency/Jitter Issue with Holographic Remoting

Discussion in 'VR' started by sstannus, Apr 18, 2017.

  1. sstannus

    sstannus

    Joined:
    Jan 7, 2017
    Posts:
    2
    I'm posting here because I haven't received any reply to the comment I added to another user's thread and the bug report I posted. It seems there is something fundamentally wrong with the Holographic Remoting support in Unity that is causing significant lag or jitter. In that previous post it was implied by the Unity team that this used to work fine but was broken by Microsoft, but this doesn't seem likely, since Microsoft's own example works fine with their slightly newer binary (replacing Unity's version with which doesn't remove the jitter). The amount of geometry and the positioning of the stabilisation plane also seem to be irrelevant. If I were to guess, I'd say it's more likely that there are a few unnecessary steps between rendering and sending the frame that are introducing extra latency.

    I hope I'm not coming across as critical; this feature is really cool and with a bit more polish it could be quite useful for more than just prototyping. On that note, is there any word on whether or not there are any plans to enable this feature in standalone builds?
     
    sstannus, Apr 18, 2017
    #1

  2. Unity_Wesley

    Unity_Wesley

    Unity Technologies

    Joined:
    Sep 17, 2015
    Posts:
    558
    The team is aware of the performance issue and it is on their radar. I believe a fix is in the works, but is unknown when it will be available for a patch
     
    Unity_Wesley, Apr 18, 2017
    #2

  3. MartijnIvory

    MartijnIvory

    Joined:
    Apr 14, 2014
    Posts:
    7
    We would also be very interested to hear about any fixes to this issue. Right now, the jittering problem is the only thing for us that stands in the way of using the HoloLens in a project where a lot of data has to be rendered.

    If it would be solved, we would be able to use the HoloLens without any modifications to our current codebase, thus saving a lot of development time. Without it, we would need to do a lot of work in terms of 3D model optimization, getting the HoloLens to interact properly with our network code, accessing asset bundles from a remote storage etc. etc.

    So please Unity, fix this issue and we would be very grateful ! :)
     
    MartijnIvory, May 1, 2017
    #3

  4. caiojahara

    caiojahara

    Joined:
    May 9, 2017
    Posts:
    15
    @Unity_Wesley Any news on this issue?

    @MartijnIvory and @sstannus Did you guys manage to find a solution?

    I came around this repository and I'm considering using it to replicate what RemotingHostSample does, what do you think?
     
    caiojahara, May 9, 2017
    #4

  5. Unity_Wesley

    Unity_Wesley

    Unity Technologies

    Joined:
    Sep 17, 2015
    Posts:
    558
    Hello,

    I believe the fix is incoming and should or has already landed, I'm still trying to track down were it landed. I will update as soon as I can, hopefully in a few hours.

    I not sure about the grahpics demo, someone else might need to give some advice on that.

    Thank you,
    Wesley
     
    Last edited: May 10, 2017
    Unity_Wesley, May 10, 2017
    #5

  6. caiojahara

    caiojahara

    Joined:
    May 9, 2017
    Posts:
    15
    Thanks for the prompt reply @Unity_Wesley , we are looking forward for the fix.

    Don't worry about the graphics demo, the fix should make it unnecessary.

    Thank you!
     
    caiojahara, May 10, 2017
    #6

  7. caiojahara

    caiojahara

    Joined:
    May 9, 2017
    Posts:
    15
    Quick update @Unity_Wesley @MartijnIvory @sstannus.

    Just downloaded Beta 4 (https://unity3d.com/pt/unity/beta/unity2017.1.0b4) from May 5th and the issue persists.
    • VR: Fix crash in Editor when attempting to do Holographic Simulation when graphics API forced to DX9 (885634)
    • VR: Fixed crash on HoloLens and Windows Mixed Reality caused by large amounts of input data overflowing queue
    It does mention a few Hololens related fixes, but nothing jitter related. I believe the patch is still to come.
     
    caiojahara, May 10, 2017
    #7

  8. Unity_Wesley

    Unity_Wesley

    Unity Technologies

    Joined:
    Sep 17, 2015
    Posts:
    558
    Correct, I was able to track it down today. The patch hasn't not landed in public builds as of yet.
     
    Unity_Wesley, May 11, 2017
    #8

  9. caiojahara

    caiojahara

    Joined:
    May 9, 2017
    Posts:
    15
    Thanks for the update @Unity_Wesley. Do you have a date on when the patch will be made public? We have a few threads regarding this issue at the Hololens Forum and it would be great to give developers this information. Precise Holographic Remoting is going to change the industry!
     
    caiojahara, May 12, 2017
    #9

  10. Unity_Wesley

    Unity_Wesley

    Unity Technologies

    Joined:
    Sep 17, 2015
    Posts:
    558
    I don't have an exact date yet of when it will be made public, I can update the thread once I have a date.
     
    Unity_Wesley, May 15, 2017
    #10

  11. MartijnIvory

    MartijnIvory

    Joined:
    Apr 14, 2014
    Posts:
    7
    Yes please, that would be awesome! We are doing a demo next week on a trade show using holographic remoting, and it will look so much better without the jittering!

    Lots of thanks in advance...
     
    MartijnIvory, May 16, 2017
    #11

  12. caiojahara

    caiojahara

    Joined:
    May 9, 2017
    Posts:
    15
    Latest beta was released yesterday but no Hololens nor Holographic Remoting fixes were on the list.
    Let's hope it comes next week.

    @Unity_Wesley Does Unity have a private beta program? From the Hololens forums it's clear that many developers would love to collaborate on bug reports or fixing.
     
    caiojahara, May 20, 2017
    #12

  13. Dustin-Horne

    Dustin-Horne

    Joined:
    Apr 4, 2013
    Posts:
    4,568
    If it makes you feel any better, on 5.6.1f1 I can't even use Remoting. It just says "Connecting.." forever and never connects to my device. I'm able to connect to the device portal, connect to it from Visual Studio and connect to it from the HoloLens app without any issues, just not the Unity Remoting.
     
    Dustin-Horne, May 24, 2017
    #13

  14. Unity_Wesley

    Unity_Wesley

    Unity Technologies

    Joined:
    Sep 17, 2015
    Posts:
    558
    @Dustin-Horne it sounds like you have blocked unity editor from being able access the network, you made need to go and allow network access to unity in order for this to work.

    I confirmed that remoting is working 5.6.1f1.
     
    Unity_Wesley, May 24, 2017
    #14

  15. Dustin-Horne

    Dustin-Horne

    Joined:
    Apr 4, 2013
    Posts:
    4,568
    Thanks I'll double check.
     
    Dustin-Horne, May 25, 2017
    #15

  16. caiojahara

    caiojahara

    Joined:
    May 9, 2017
    Posts:
    15
    Any news on the jitter patch release date?
     
    caiojahara, May 25, 2017
    #16

  17. Unity_Wesley

    Unity_Wesley

    Unity Technologies

    Joined:
    Sep 17, 2015
    Posts:
    558
    We are currently in a holding pattern until we can verify the fix with some other tools that we need. Sorry for the delay
     
    Unity_Wesley, May 25, 2017
    #17

  18. caiojahara

    caiojahara

    Joined:
    May 9, 2017
    Posts:
    15
    Thanks for the update @Unity_Wesley , can we expect the patch to arrive later this week?
    We are really counting on this. I understand Hololens still represents a tiny amount of developers, but this is a very serious issue that's been going on for a few months. Hololens loves Unity :)
     
    Last edited: May 30, 2017
    caiojahara, May 30, 2017
    #18

  19. Unity_Wesley

    Unity_Wesley

    Unity Technologies

    Joined:
    Sep 17, 2015
    Posts:
    558
    I don't have an ETA on when it can be expected
     
    Unity_Wesley, May 30, 2017
    #19

  20. MartijnIvory

    MartijnIvory

    Joined:
    Apr 14, 2014
    Posts:
    7
    MartijnIvory, Jun 13, 2017
    #20

  21. Unity_Wesley

    Unity_Wesley

    Unity Technologies

    Joined:
    Sep 17, 2015
    Posts:
    558
    I have news but it is not happy news, we are in a holding pattern with the fix for now. I can't give much detail on it at this time.
     
    Unity_Wesley, Jun 14, 2017
    #21

  22. wechang

    wechang

    Joined:
    May 25, 2017
    Posts:
    1
    Hi @Unity_Wesley, I'm also curious about the status of this bug. Has this been deployed to the newest Unity editor? Thanks!
     
    wechang, Jul 1, 2017
    #22

  23. caiojahara

    caiojahara

    Joined:
    May 9, 2017
    Posts:
    15
    Same here. Any news on this issue
     
    caiojahara, Jul 7, 2017
    #23

  24. caiojahara

    caiojahara

    Joined:
    May 9, 2017
    Posts:
    15
    Hi @Unity_Wesley , any news ?

    I've noticed a few fixes related to Hololens on the latests beta, but nothing Holographic Remoting related.
     
    caiojahara, Jul 18, 2017
    #24

  25. osamahchoudhry

    osamahchoudhry

    Joined:
    Jan 29, 2017
    Posts:
    3
    @Unity_Wesley any news on when this will be fixed? also the ability to build remoting directly into a deployed Unity project? Thanks
     
    osamahchoudhry, Aug 11, 2017
    #25

  26. Unity_Wesley

    Unity_Wesley

    Unity Technologies

    Joined:
    Sep 17, 2015
    Posts:
    558
    Sorry for the delay, there have been some fixes for remoting in build 2017.3 and 2017.2. I'm unsure if they are going to be back ported to 5.6
     
    Unity_Wesley, Aug 11, 2017
    #26

  27. Unity_Wesley

    Unity_Wesley

    Unity Technologies

    Joined:
    Sep 17, 2015
    Posts:
    558
    Unity_Wesley, Aug 11, 2017
    #27

  28. osamahchoudhry

    osamahchoudhry

    Joined:
    Jan 29, 2017
    Posts:
    3
    @Unity_Wesley how do you suggest we go about doing this in a Unity/C# application? The link you posted seems to apply only to C++/DirectX projects.

    Any ideas on how to re-use it for Unity/C# projects?
     
    osamahchoudhry, Aug 12, 2017
    #28

  29. Unity_Wesley

    Unity_Wesley

    Unity Technologies

    Joined:
    Sep 17, 2015
    Posts:
    558
    I'm sorry I don't have a lot of knowledge on this area, my best guess would be build a dll to use with unity
     
    Unity_Wesley, Aug 14, 2017
    #29

  30. caiojahara

    caiojahara

    Joined:
    May 9, 2017
    Posts:
    15
    Thanks for the updates.

    IMPORTANT @osamahchoudhry , @wechang
    @Unity_Wesley, downloaded every version of Unity, and found out that Holographic Remoting is working fine on Unity 5.5.0f3. Can't you have someone just roll back to the working dll?
    Don't you think it's way easier for Unity to fix it since it's a feature that is officialy offered and you already have working and a failing version, than having every developer come up with their own solution?

    I hope it helps! It was very time consuming to download and test every version :(
     
    caiojahara, Aug 17, 2017
    #30

  31. Unity_Wesley

    Unity_Wesley

    Unity Technologies

    Joined:
    Sep 17, 2015
    Posts:
    558
    @caiojahara what are your referring
    What are you referring to? I'm unsure of what you are asking for or what you are trying to say, there appears to be missing context.
     
    Unity_Wesley, Aug 17, 2017
    #31

  32. caiojahara

    caiojahara

    Joined:
    May 9, 2017
    Posts:
    15
    My mistake, everything is working great on 2017.2.0b8. Thanks a lot for everything @Unity_Wesley
     
    caiojahara, Aug 21, 2017
    #32

  33. irene888

    irene888

    Joined:
    Oct 25, 2017
    Posts:
    4
    Hi @Unity_Wesley , I have a issue when using Holographic Remoting in Unity 2017.2.0f3. When I run it, I can hear the sound, but cannot see anything...
     
    irene888, Oct 25, 2017
    #33

  34. unity_andrewc

    unity_andrewc

    Unity Technologies

    Joined:
    Dec 14, 2015
    Posts:
    223
    Do you have the "Enable Video" checkbox disabled in the Holographic Emulation window? I think that un-checking that toggle would result in that behavior, just want to double-check before digging deeper.
     
    unity_andrewc, Oct 26, 2017
    #34

  35. irene888

    irene888

    Joined:
    Oct 25, 2017
    Posts:
    4
    irene888, Oct 26, 2017
    #35

  36. unity_andrewc

    unity_andrewc

    Unity Technologies

    Joined:
    Dec 14, 2015
    Posts:
    223
    K, I figured that might be the case, just wanted to make sure. I haven't tested that directly myself recently, but I just talked to QA and they said they didn't see that issue when they tested that build.

    Would you mind filing a bug report for us? The system info it collects might point us in the right direction, I'm hoping this is just some dev environment issue on your machine. Regardless, we'll investigate.

    Either way, it might be worth trying out an MRTP build: https://forum.unity.com/threads/cus...ndows-mixed-reality-technical-preview.498253/ - let me know if that helps.
     
    unity_andrewc, Oct 26, 2017
    #36

  37. unity_andrewc

    unity_andrewc

    Unity Technologies

    Joined:
    Dec 14, 2015
    Posts:
    223
    Also, after I poked QA to confirm as a sanity test, the guy that re-tested it said that when the wi-fi here is congested, he was running into something similar - tracking works fine, but the video in the headset would (sometimes?) be black/blank.

    He tested by running it over USB, and said it worked much better that way. Would you mind testing with a USB cord connecting to your HoloLens? All you need to do to get remoting to work that way is to give it an IP address of 127.0.0.1. (Lame solution if the problem is your wi-fi, I know, but at least you could get a longer USB cord and unblock better iteration time that way.)
     
    unity_andrewc, Oct 26, 2017
    #37

  38. pfreese

    pfreese

    Unity Technologies

    Joined:
    Feb 23, 2015
    Posts:
    85
    You could also try lowering the Max Bitrate used for streaming. This will lower the video quality, but might resolve the issue with blank frames if the problem is in fact caused by limited WiFi bandwidth.
     
    pfreese, Oct 26, 2017
    #38

  39. irene888

    irene888

    Joined:
    Oct 25, 2017
    Posts:
    4
    @unity_andrewc , @pfreese made it work using Unity 5.6.2. I guess some updates may be missed on my machine, resulting in not working with Unity 2017.2.0f3. Could you think of any upgrades I need to install?
     
    irene888, Oct 30, 2017
    #39

  40. irene888

    irene888

    Joined:
    Oct 25, 2017
    Posts:
    4
    @unity_andrewc , @pfreese by the way, is there a way that I can check the latency when using the remoting mode? Thanks in advance!
     
    irene888, Oct 30, 2017
    #40

  41. pfreese

    pfreese

    Unity Technologies

    Joined:
    Feb 23, 2015
    Posts:
    85
    You might have better luck with the 2017.2p1 build which should be available next week. We've had quite a few fixes which could improve stability while remoting (although most are targetting towards Windows Mixed Reality on desktop). If you try it and run into any issues, be sure to let us know.

    AFAIK, there isn't a way to check latency.
     
    pfreese, Nov 4, 2017
    #41

  42. Octoplow

    Octoplow

    Joined:
    Jan 24, 2015
    Posts:
    6
    @irene888 @pfreese
    After you start the Holographic Remoting app on HoloLens, say "enable diagnostics" before connecting from Unity.
    https://developer.microsoft.com/en-us/windows/mixed-reality/holographic_remoting_player
     
    Octoplow, Dec 14, 2017
    #42