Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Random disconnections while simulating packet loss

Discussion in 'Multiplayer' started by srylain, Apr 24, 2016.

  1. srylain

    srylain

    Joined:
    Sep 5, 2013
    Posts:
    159
    I've been simulating small amounts of packet loss (around 2%) and the client has been getting disconnected every time. I have the Disconnect timeout set to 10 seconds, and the ping timeout set to 5 seconds.

    Data is still getting to both instances just fine as both can see each others' characters moving.
     
  2. srylain

    srylain

    Joined:
    Sep 5, 2013
    Posts:
    159
    Is there anything anyone knows about how to prevent these disconnections? I've narrowed it down to being packet loss related, even at simulated 1% loss rates it still happens. I've also tried connecting to an instance of my game running from a tethered hotspot and it still disconnects.
     
  3. srylain

    srylain

    Joined:
    Sep 5, 2013
    Posts:
    159
    I just tried setting the ping timeout to 10 seconds, and that pretty much immediately caused a disconnection after about 12 seconds or so after joining. Is it something to do with ping timeouts? It's currently set to 5 seconds, as that seems to keep them connected when there isn't any packet loss at all.

    After looking for this bug in the bug tracker, both of the ones I found were flagged as "By Design". I honestly can't see how Unity team can expect there to be absolutely no packet loss from the real world internet.

    Is there any help I can get here?
     
  4. srylain

    srylain

    Joined:
    Sep 5, 2013
    Posts:
    159
    So is UNET broken in this case? Or has nobody else ever had this happen? I'd hate to think it's broken because it's something I want to use, but after getting no help and not finding anything else on this subject it's the only I can say about it.
     
  5. Ashkan_gc

    Ashkan_gc

    Joined:
    Aug 12, 2009
    Posts:
    1,117
    It works much better in the real world, as I understand the simulation is not that good yet. I'm no expert on uNet yet, Only made 2-3 games with it but in the real world, it works well with none webGL clients specially. The webGL issues are being resolved as well but the simulator currently is not in the best state or I just misconfigured it like you.
     
  6. srylain

    srylain

    Joined:
    Sep 5, 2013
    Posts:
    159
    I've tried tethering a computer to a phone's hotspot, and it still disconnected roughly the same. Even with using an external network simulator (Clumsy) it still does the same. My game isn't the standard competitive FPS either (I'm working on a co-op campaign based game), so generally connections will probably last longer than 10 minutes or so and I'd hate to have people need to reconnect every so often, especially when it seems to happen randomly which can cause lost progress.

    Isn't there a way to notify a Unity dev or something? I'd really like something official about this issue.
     
  7. Ashkan_gc

    Ashkan_gc

    Joined:
    Aug 12, 2009
    Posts:
    1,117
    There is a uNet improvement thread on top of sticky threads of multiplayer forums. Go write your feedback there.
     
  8. Paradoks

    Paradoks

    Joined:
    Oct 13, 2009
    Posts:
    436
    Great thread, i have deconnection bugs on my webGL game.
    This could be the explanation for my problem too.
    I am in contact with the unity QA, i keep you informed.
     
    Tset_Tsyung likes this.
  9. srylain

    srylain

    Joined:
    Sep 5, 2013
    Posts:
    159
    I mainly just wanted to see if there was a potentially easy fix before I made mention of it there, because I'm sure there's plenty of things that could've escaped me that could've fixed this. I also just can't for the life of me understand how they expect there to be no packet loss (I know the internet nowadays generally doesn't lose that many packets anymore) because it's something that you can't control.
     
  10. Paradoks

    Paradoks

    Joined:
    Oct 13, 2009
    Posts:
    436
    Any news on this, i didnt get any from the QA guys btw.
     
  11. srylain

    srylain

    Joined:
    Sep 5, 2013
    Posts:
    159
    I don't have any. After even more testing I still just come to the same conclusions that any amount of packet loss will eventually lead to a disconnection. Which is fine and all for games where you're only connected for minutes at a time I guess, but when you are expected to be connected for hours at a time it's a massive problem.
     
  12. Ashkan_gc

    Ashkan_gc

    Joined:
    Aug 12, 2009
    Posts:
    1,117
    There is a bug related only to WebGL which I've submitted and it will be in a future patch release. It's fixed. The buffer for receiving messages is not sometimes big enough and they made it configurable but it's only for WebGL.
     
  13. srylain

    srylain

    Joined:
    Sep 5, 2013
    Posts:
    159
    How small was that buffer? And is there any way to see how many buffered messages there currently are? If I'm reading your post right, you're saying that although they let you change the buffer size it doesn't actually do anything to non WebGL games?
     
  14. Ashkan_gc

    Ashkan_gc

    Joined:
    Aug 12, 2009
    Posts:
    1,117
    It's only related to WebGL games which use webSockets, All other platforms use UDP and uNet's own protocol and transport layer.
    The buffer was 4k in size by default. I saw the bug when the buffer could not receive the incoming data and would issue a warning in the log and cause disconnects (it seemed there is a causal corolation between them).
    There are methods for getting Network stats but I don't know if they work in WebGL too or not.
     
  15. PrimeDerektive

    PrimeDerektive

    Joined:
    Dec 13, 2009
    Posts:
    3,090
    I'm seeing this too. I try simulating packet loss with either clumsy or the networkmanager simulator and I can visibly see the network slowly degrade (position updates start to come less and less frequently on my player prefabs) before I start getting "failed to send internal buffer" errors and disconnection. No WebGL here, just simple windows builds. I'm not syncing anything crazy, just position and rotation + timestamp updates 15x per second, and using only unreliable channels.
     
  16. Ashkan_gc

    Ashkan_gc

    Joined:
    Aug 12, 2009
    Posts:
    1,117
    I guess someone should post this thread in the improvements thred for uNet team to see.
     
    PrimeDerektive likes this.
  17. PrimeDerektive

    PrimeDerektive

    Joined:
    Dec 13, 2009
    Posts:
    3,090
    Hey Ashkan you were the uLink guy! I miss uLink :( Sometimes I think about still using it because I know it still works but with every unity patch I'm afraid its going to break.

    Someone should some rewrite the uLink statistics GUI to work with unet so I could actually calculate my traffic easily. I still use the ConsoleGUI since it really had nothing to do with uLink.
     
  18. Ashkan_gc

    Ashkan_gc

    Joined:
    Aug 12, 2009
    Posts:
    1,117
    Yeha I miss uLink as well and still use The console GUI. In uNet you should use the profiler for now and the report is more detailed than uLink there actually.
    The issue is not the GUI, The statistics functions don't work atm in uNet.
     
  19. aabramychev

    aabramychev

    Unity Technologies

    Joined:
    Jul 17, 2012
    Posts:
    574
    Hello guys. @Ashkan_gc thanks for pointing this thread.

    @crynryan : difficult to say, so far i don't know do u use udp or tcp (web sockets), what qos channel do you use, what is your bandwidth, how many messages and what is the message size? The easiest way is - report the bug and attach bug number here + ping me. I will take a look. But I suppose that in your simulation packet loss is more than 2%. Anyway, I guess you can do 3 things right now
    1. Reduce ping timeout back to 500 ms
    2. Increase NetworkDropThreshold to 40% (in connection config)
    3. Decrease packet size to 1400 bytes (in connection config)

    @PrimeDerektive : please point number 2 from the list above. Should help.

    @Paradoks : hey man, Web sockets is a different story, almost no parameters in connection config works with web sockets. I guess that your "disconnects" related to library itself. I do not believe that tcp connection can be disconnected easily without any internal problem What is the your reported bug number?

    Sorry, guys, my fault, a lot of work in last month cause that i read that forum quite rare... Ping me directly please if you think that you need my attention. Again unfortunately i cannot read everything :( I'm sorry about this
     
  20. Paradoks

    Paradoks

    Joined:
    Oct 13, 2009
    Posts:
    436
    Hi @aabramychev, i just sent you the case number in pm.

    Vaidas tryied to reproduce the case and found no bug, but as i told him, i think he used a virtual machine instead of a real distant server like my EC2 so the bug could not occur because there was no packet loss or connection problems.

    but i dont know if he get my latest message as i got no answers from him since april 26th.

    Could you keep me informed please ?

    thx.
     
  21. srylain

    srylain

    Joined:
    Sep 5, 2013
    Posts:
    159
    Thanks for the reply @aabramychev!

    I have channel 0 as reliable, channel 1 as unreliable. I use channel 0 whenever a player picks up an item or shoots an enemy, and channel 1 for movement syncing. I'm not using Network Animator either (from what I saw, there's no way to change it to use an unreliable channel?), as I'm sending inputs and it works just fine without it.

    As for packet sizes, I should only be sending ~36 bytes a packet at 20 times a second per player. I'm only planning for two players in my game, so I'm not sending a ton of packets either. My bandwidth is pretty high, 125Mb down and 12Mb up. It still happens in LAN though, where I'm not sure on the actual speeds of my wireless but I'd think it's above 600Mb.

    I'll try what you suggested a bit later and get back to you with the bug report if nothing changes. Thanks again for the reply!
     
  22. srylain

    srylain

    Joined:
    Sep 5, 2013
    Posts:
    159
    So I just tested with both Clumsy and two instances on my computer, and an instance on my computer and an instance on another computer that was connected through a phone's hotspot (both ways would've disconnected the same before). It looked like changing what you told me to worked, although I did still have a couple disconnections which I think might have come from me browsing the internet and having both instances running in the background. I have the "Run in background" checked as well, but I'm not sure if there's any problems from running two instances that are both not in focus.

    As for it working, I had Clumsy set to 5% packet loss (both incoming and outbound packets) and it stayed connected for at least 40 minutes. Before it would disconnect when Clumsy was set to 1%.

    A couple other questions I have:

    What is the default for NetworkDropThreshold? And do you keep track of all packets dropped overall, or just packets from the last x amount of seconds?

    I've been using NetworkTransport.GetNetworkLostPacketNum(); to get packet loss, and it appears that it doesn't include protocol packets and only RPC/Command packets. Is that correct? Is there any way to see how many lost packets there are overall?

    Again, thanks for the help @aabramychev!
     
  23. aabramychev

    aabramychev

    Unity Technologies

    Joined:
    Jul 17, 2012
    Posts:
    574
    NetworkDropThreshold - mean how many packet was dropped by network condition between two pings, and default value is 5% (which is relative small if you send few messages per ping period, for example if you send only one message per ping timeout == 500 ms by default and packet will lost you will receive 100% packet lost). What's happened next? Library is thinking that it sends a number of packets which is too much for your network and starts to decrease send rate (actually starts increasing send timeout, when this timeout will > then disconnect timeout you will receive disconnect). I would say I was wrong with my assumption 5% and really this value should be set about 95 in most cases and should be decreased in special conditions only when user understanding why he wont decrease this. I will change default value in the next version...

    About statistic, it is quite S***ty now :( I did make significant changes for 5.5 but when it will be accessible for user I don't know). About this value it is not easy to explain, as on that level I do not have message statistic. Let me explain, when you send message to system, it is not necessary that this message will send immediately and will occupy whole udp packet. Actually, the system will wait for amount of time (SendUpdateTimeout) and will try to combine all incoming messages per one packet (up to PacketSize size) and send this packet as payload of UDP packet. Each packet has number in the header and receiver defines the how many packet was lost using the following algorithm
    packetLost += newPacketId - lastReceivedPacketId;
    lastReceivedPacketId = newReceivedPacketId;

    Library keeps this numbers (packetLost and lastReceivedPacketId) during whole session per each connection (from connect to disconnect) and this function will report you absolute value (packet lost from connect) periodically asked this function allow you to setup relative picture, packet lost per sec... But again, this stat not necessary related to any messages which you sent.

    Hope my explanation was clear enough.
    Alex
     
  24. PrimeDerektive

    PrimeDerektive

    Joined:
    Dec 13, 2009
    Posts:
    3,090
    @crynryan , when you do get your disconnects do you get any errors? If so what are they? Just want to compare to mine.
     
  25. srylain

    srylain

    Joined:
    Sep 5, 2013
    Posts:
    159
    I don't get any errors, just console level messages saying that they were disconnected due to timeouts.

    Thanks @aabramychev for the detailed explanation.
     
  26. PrimeDerektive

    PrimeDerektive

    Joined:
    Dec 13, 2009
    Posts:
    3,090
    Hrm I was getting errors, something about buffer something bytes: 38. I can't get it to happen now though :(

    What I do notice though is with even 5% packet loss simulation, my client is only getting updates on objects that are sending updates 20x per second like once every 5-10 seconds, so it is unplayably choppy. @aabramychev, do you have any idea why that would be happening?

    Also, where should I change the connection config to alter the networkthreshhold?
     
  27. aabramychev

    aabramychev

    Unity Technologies

    Joined:
    Jul 17, 2012
    Posts:
    574
    NetworkManager has property ConnectionConfig, so i guess you can do
    NetworkManager.ConnectionConfig.NetworkDropThreshold = 40; Note, that you should to do this before you start server or client.

    About first problem, No I do not have any idea, and I cannot speculate what;s going on without seeing the code. If you think that there is a bug you can report this... But Note, that 5% is default value of NetworkDropThreshold, so if you did setup 5 in your packet drop simulation you will receive disconnect soon.
     
  28. srylain

    srylain

    Joined:
    Sep 5, 2013
    Posts:
    159
    After running a couple more builds, I found that with Clumsy set to 3% packet loss (both in an out) it stayed connected for a good 2-3 hours. With 5%, it still disconnected although it did stay connected for quite a bit longer than it used to. I assume though that setting it to both in and out packets effectively quadruples the packet loss.

    I remember the XNA guidelines for the Xbox 360 saying that you should design around 10% packet loss, although I assume 10% would be the worst case scenario. Does anyone know how much I should expect with the internet of today?
     
  29. aabramychev

    aabramychev

    Unity Technologies

    Joined:
    Jul 17, 2012
    Posts:
    574
    Itis very difficult to say, the problem is the last mile which novadays is usually wi fi... In S***ti environment you should expect up to 100% loss. My recomendation set this value to 90 percent your game most probably cannot overfull your router so, no conjection control is required here
     
  30. srylain

    srylain

    Joined:
    Sep 5, 2013
    Posts:
    159
    One more question for now, although I'm sure I might have more later. When you say that you figure the packets lost in between pings, does that also count pings as well? So say that I send no data in between pings, and the 2nd ping is dropped but the 3rd is received. Would that raise the send timeout?

    I ask because I'm still getting quite a few disconnections when my players are just standing still and not sending any data. The connection generally lasts longer now than it did before, but the highest amount of packet loss I've been able to set and have it stay connected for a while was 6%. And that's with NetworkDropThreshold set to 90.
     
  31. srylain

    srylain

    Joined:
    Sep 5, 2013
    Posts:
    159
    Another question. Does the send timeout ever reset or go back down if the connection clears up? Unless there's something that wouldn't let it work, would it be possible to add that if it isn't already there?
     
  32. aabramychev

    aabramychev

    Unity Technologies

    Joined:
    Jul 17, 2012
    Posts:
    574
    About first question. Yes they are calculated as well.
    >The connection generally lasts longer now than it did before, but the highest amount of packet loss I've been able to set and have it stay connected for a while was 6%. And that's with NetworkDropThreshold set to 90
    Hm, could you please open bug about this, I afraid I cannot add anything else without trying to reproduce this on my computer. If you will able to strip out your project you can directly ping me here and I will take a look on this issue without our bug system. Do not forget to mention what version of unity do you use.

    >Another question. Does the send timeout ever reset or go back down if the connection clears up?
    yes it should.

    Alex
     
  33. srylain

    srylain

    Joined:
    Sep 5, 2013
    Posts:
    159
    Last edited: May 17, 2016
  34. aabramychev

    aabramychev

    Unity Technologies

    Joined:
    Jul 17, 2012
    Posts:
    574
    thanks dude. I ve subscribed to this bug will look if our qa will be able to reproduce
     
  35. srylain

    srylain

    Joined:
    Sep 5, 2013
    Posts:
    159
    I got an email saying they were able to reproduce it a day or two ago.
     
  36. srylain

    srylain

    Joined:
    Sep 5, 2013
    Posts:
    159
    Hey @aabramychev

    Just got an update from one of the devs, which explained what I was doing wrong. Because I was used to using Lidgren, I created an instance of ConnectionConfig and then set the values in it and then used ConnectionConfig.Validate() while passing in the created ConnectionConfig assuming that would work. The documentation's a bit lacking in that area, so there wasn't anything telling me the right way to do things.

    The dev that responded said to use NetworkManager.singleton.connectionConfig.NetworkDropThreshold, which I did and the game stayed connected for a good 3-4 hours at 50% packet loss. I set the value to 90, and even at 85% packet loss it stayed connected for quite a bit, and only disconnected after I set the loss rate to 95%.

    Thanks for the help though, no idea if I would've stuck with UNET if I didn't get this worked out. Thanks again!
     
  37. aabramychev

    aabramychev

    Unity Technologies

    Joined:
    Jul 17, 2012
    Posts:
    574
    It was my response :) Thanks
     
  38. cubrman

    cubrman

    Joined:
    Jun 18, 2016
    Posts:
    409
    I am currently experiencing random disconnects by setting simulated latency at 400 and launching 2 instances of the game on the same machine. The client usually gets dropped around 5 to 10 seconds after they connect. I noticed that increasing "Ping Timeout" increases the connection interval, but there is clearly something wrong.
     
  39. aabramychev

    aabramychev

    Unity Technologies

    Joined:
    Jul 17, 2012
    Posts:
    574
    Did you set latency only or packet dropping too? If no packet dropping was set, it is probably another bug, report it please
     
  40. cubrman

    cubrman

    Joined:
    Jun 18, 2016
    Posts:
    409
    I indeed had NO packed dropping, how to report a bug?
     
  41. aabramychev

    aabramychev

    Unity Technologies

    Joined:
    Jul 17, 2012
    Posts:
    574
    I read your another post. Yes, you need report the bug, looks like something wrong with hlapi :(
     
  42. cubrman

    cubrman

    Joined:
    Jun 18, 2016
    Posts:
    409
    How do I report a bug?)
     
  43. cubrman

    cubrman

    Joined:
    Jun 18, 2016
    Posts:
    409
    Found it, sorry I was dumb :)
     
  44. aabramychev

    aabramychev

    Unity Technologies

    Joined:
    Jul 17, 2012
    Posts:
    574