Search Unity

Unity 5.6/2017 UNet - Stopping client via NetworkManager timeouts connection instead of disconnect

Discussion in 'UNet' started by xVergilx, Apr 13, 2017.

  1. xVergilx

    xVergilx

    Joined:
    Dec 22, 2014
    Posts:
    3,296
    I'm using custom network manager and trying to disconnect client from the server when user decides to leave the server like so:

    Code (CSharp):
    1.  
    2. void Leave(){
    3. ...
    4.            if (NetworkServer.active && NetworkClient.active) {
    5.                 CustomNetworkManager.singleton.StopHost();
    6.             } else if (!NetworkServer.active && NetworkClient.active) {
    7.                 CustomNetworkManager.singleton.StopClient();
    8.             } else {
    9.                 DisplayOfflinePanel();
    10.             }
    11. ...
    12. }
    13.  
    But it seems to be causing timeout for the connection on the server, instead of a proper disconnect:
    Is this intended, or it's a bug?
     
  2. gamevanilla

    gamevanilla

    Joined:
    Dec 28, 2015
    Posts:
    968
    I can confirm this has started happening with 5.6, with both the low-level API and the high-level API. When using the low-level API the output is a bit different:

    reactor filed to acomplish request with code {1234}, bytesReceived {0}
    reactor filed to acomplish request with code {1234}, bytesReceived {0}
    reactor filed to acomplish request with code {1234}, bytesReceived {0}
    Log: connection {1} has been disconnected by timeout; address {127.0.0.1:63855}

    I pinged @aabramychev about this a couple of days ago and he told me he is aware of the issue and that it should be fixed in an upcoming patch release. Which is, of course, very much appreciated!
     
  3. xVergilx

    xVergilx

    Joined:
    Dec 22, 2014
    Posts:
    3,296
    Thank you for confimation, that's good to know. Will be looking forward next patch release.
     
  4. aabramychev

    aabramychev

    Unity Technologies

    Joined:
    Jul 17, 2012
    Posts:
    574
    It has been fixed already. Should be available with p2 or p3. (Sorry for crap)
     
    Deleted User and TwoTen like this.
  5. xVergilx

    xVergilx

    Joined:
    Dec 22, 2014
    Posts:
    3,296
    Same behavior with p2. Will be waiting for p3.
     
  6. xVergilx

    xVergilx

    Joined:
    Dec 22, 2014
    Posts:
    3,296
    This is still an issue with p3.

    T_T
     
  7. xVergilx

    xVergilx

    Joined:
    Dec 22, 2014
    Posts:
    3,296
    Bump?
     
  8. shadiradio

    shadiradio

    Joined:
    Jun 22, 2013
    Posts:
    83
    I don't see any fixes related to networking in the release notes for p1, p2, or p3. :(
     
  9. shamsfk

    shamsfk

    Joined:
    Nov 21, 2014
    Posts:
    307
    Any news? It is critical
     
  10. aabramychev

    aabramychev

    Unity Technologies

    Joined:
    Jul 17, 2012
    Posts:
    574
    I'm a little bit confused :( I see two different problem, correct?
    first is from @VergilUa while the second from @Spelltwine-Games ?
    ok let's start from second: One fix was applied, and another fix on the way (it has been ready just waiting to be submitted)

    the first one: there are couple of reasons while you receive disconnect with timeout: when you call StopClient you actually call Transport.RemoveHost() whihc just kill the socket, after that there is no way to deliver disconnect to other side :( You can try NetworkClient.Disconnect (or on server side DisconnectAll) call, wait couple of frames and after that call StopClient or StopServer function.

    Some additional explanation, disconnect message has been implemented as "best attempt" event, it means that peer which want to disconnect sends only one "disconnect" packet and then closed connection. (So no warranty that this packet won't be dropped during delivery). Q: When it will actually sends? A: Approximately after 2~3 + GlobalConfig.threadawaketimeout period. When you call disconnect, real send happened when io thread will awake, it takes 1*GlobalConfig.threadawaketimeout period, if you have a lot of data to send, probably it will require more time. For example if you threadawaketimeout = 1 ms, you can expect that after 3 ms disconnect will send, and call Stop() function just in the next frame.

    Why it implemented like this? It is quite difficult to implement proper disconnect (see tcp disconnect as example), and in most cases it is not required. If you really need reliable disconnect, you can make it by yourself, add reliable channel, send "I'm disconnecting", ask server to respond "Disconnecting acknowledged" and then call disconnect.

    and the last thing: disconnect by timeout is not a disconnect with error, it is just generic disconnect which you need handle as usual. Q: why we add the log about this? A: there are few cases when disconnect by timeout can help to understand what's going on.

    We are going to add function which will put all logs only if user ask about this (like DebugLog = true) hope it will help.

    Was I clean?
    Any questions?
    Alex
     
  11. NongBenz

    NongBenz

    Joined:
    Sep 30, 2014
    Posts:
    26
    Hi aabramychev! My current issue is related to this (LLAPI)!

    In my server logs I get this for example...
    .................
    Log: connection {44} has been disconnected by timeout; address {::ffff:213.92.138.185:65316}
    Attempt to send to not connected connection {44}

    (Filename: Line: 359)

    Sending Error: 2
    .................................


    I assume the Timeout happened when I called NetworkTransport.Send()? (Send() and ReceiveFromHost() are the only 2 places where a timeout can occur, right?)

    But I need to get the connectionID to cleanup and destroy the Player object tied to that connection. But this timeout/Send() responds with Sending Error 2 (invalid connection) rather than Sending Error 4 (Timeout)....

    So it looks like my server cannot find then destroy this invalid connectionID (-> Player object) and gets stuck trying to send to it until all queues are full and the server crashes.

    Are my assumptions correct? How would I handle this timeout on Send().. when apparently I can't get the exact connection that's timed out?

    EDIT: My disconnect / timeout handling code looks like this... But doesn't seem to work with an invalid connection returned.

    if (!NetworkTransport.Send(
    host,
    connection,
    channel,
    buffer,
    writer.Position,
    out rError) || rError != (byte)NetworkError.Ok)
    {
    Debug.Log ("Sending Error: " + rError);
    disconnectQueue.Enqueue (connection);
    }

    EDIT2: Looking at my code again, I think the error is maybe another player connecting and getting the same connection ID before the disconnectQueue is flushed and cleaned up. I will try this fix.

    Please let me know
    Benz
     
    Last edited: May 10, 2017
  12. aabramychev

    aabramychev

    Unity Technologies

    Joined:
    Jul 17, 2012
    Posts:
    574
    @NongBenz
    >I assume the Timeout happened when I called NetworkTransport.Send()? (Send() and ReceiveFromHost() are the only 2 places where a timeout can occur, right?)

    No it i not right. One io thread there are a different timers, one of them periodically check when the last receiving was. If this time > disconnect timeout, io thread will set state of connection to Disconnecting and put in the log record about disconnecting by timeout (which you obtained in the log). With this state, if you try to send you will receive false and WrongConnection error (and record in the log that you did try to send via connection in wrong state). But state of connection will be Disconnecting before you pump the host (which this connection belonging too) and receive Disconnect event. After that connection will be consider as disconnected. So, nobody can occupy connection id before you will receive Disconnect event.

    According this.

    ```
    if (!NetworkTransport.Send(
    host,
    connection,
    channel,
    buffer,
    writer.Position,
    out rError) && rError == (byte)NetworkError.WrongConnection) //another way receive false is NoResources, in this case you just need to wait)
    disconnectQueue.Enqueue (connection);
    ...
    evnt = Receive( hostId, connectionId ... )
    if( evnt == Disconnect)
    disconnectQueue.Remove(connectionId); //here connection has bean cleaned up and will be reuse

    was I clean?
     
  13. NongBenz

    NongBenz

    Joined:
    Sep 30, 2014
    Posts:
    26
    Ah thank you! This finally explains the hidden crash bug and NoResources errors on my server! Will fix now.
     
  14. xVergilx

    xVergilx

    Joined:
    Dec 22, 2014
    Posts:
    3,296
    Ok, so there's no fix to this (Timeout instead of disconnect), right?
    I've tried coroutine wait approach with no luck. In that frame after disconnect but before stop there's actually messages that communicate between client and server, which results in even more error messages popping up.

    Any other ideas? Use LLAPI instead? I would REALLY appreciate an example of how it should be done.

    Why it matters? Because it adds additional unrelated to debugging information to the log, which is not usefull when there's multiple servers running on the same machine.
     
  15. aabramychev

    aabramychev

    Unity Technologies

    Joined:
    Jul 17, 2012
    Posts:
    574
    @VergilUa, hmm would you please cleanup your project (to emphasis the problem) and send me with explanation what do you obtain and what do you want? I will take a look.

    tnx
     
  16. xVergilx

    xVergilx

    Joined:
    Dec 22, 2014
    Posts:
    3,296
    That would take ages to do. I just want to have a proper player disconnect without receiving timeout message. In fact it doesn't even matter for our game if it is a timeout or an actual player-driven disconnect. It's the message that is the problem: "Failed to send buffer + timeout" error that pops every time when player disconnects.

    I bet there's a different way to implement disconnect than using Coroutines (since that didn't worked at all). I don't know, maybe a message pass between server and client?

    In what order and what should be called?

    Edit:
    This is what I tried so far - I'm sending message from client to the server asking for a disconnect, server does that and client is either disconnects on it's own, or does a proper disconnect procedure after a while anyway.

    Code (CSharp):
    1.         private Coroutine _disconnectCoroutine;
    2.         private readonly WaitForSeconds _clientShutdownDelay = new WaitForSeconds(0.5f);
    3.  
    4.         public override void OnServerReady(NetworkConnection conn) {
    5.             NetworkServer.SetClientReady(conn);
    6.  
    7.             RegisterServerHandlers();
    8.         }
    9.  
    10.         private void RegisterServerHandlers() {
    11.             NetworkServer.RegisterHandler((short) CustomOpCodes.UnetDisconnect, DisconnectClient);
    12.         }
    13.  
    14.         /// <summary>
    15.         /// Disconnects client connection from the server
    16.         /// </summary>
    17.         /// <param name="message">Empty message</param>
    18.         private static void DisconnectClient(NetworkMessage message) {
    19.             NetworkConnection networkConnection = message.conn;
    20.             networkConnection.Disconnect();
    21.             networkConnection.Dispose();
    22.         }
    23.  
    24.         public void StopClient(NetworkConnection clientConnection) {
    25.             clientConnection.Send((short) CustomOpCodes.UnetDisconnect, new EmptyMessage());
    26.  
    27.             if (_disconnectCoroutine != null) {
    28.                 StopCoroutine(_disconnectCoroutine);
    29.             }
    30.  
    31.             _disconnectCoroutine = StartCoroutine(StopClientAfterDelay());
    32.         }
    33.  
    34.         private IEnumerator StopClientAfterDelay() {
    35.             yield return _clientShutdownDelay;
    36.  
    37.             if (NetworkClient.active) {
    38.                 OnStopClient();
    39.                 if (LogFilter.logDebug)
    40.                     Debug.Log("NetworkManager StopClient");
    41.                 isNetworkActive = false;
    42.                 if (client != null) {
    43.                     client.Disconnect();
    44.                     client.Shutdown();
    45.                     client = null;
    46.                 }
    47.             }
    48.  
    49.             yield return null;
    50.         }
    Problem is - it still adds message to the internal buffer, spitting out errors. I've tried increasing\decreasing delay, WaitForEndOfFrame, still the same. Fun fact - proper disconnect occurs, if not counting that client is piling up messages before disposing. Critical backdraw - it utterly HANGS server. I don't know why.

    I guess I'll need to dig deeper into Unet NetworkServer/NetworkClient to implement everything properly.
     
    Last edited: Jul 20, 2017
  17. aabramychev

    aabramychev

    Unity Technologies

    Joined:
    Jul 17, 2012
    Posts:
    574
    the problem here (I guess), that hlapi doesn't have any idea do you disconnect or not, and still trying to send messages. Unfortunately I'm not expert in hlapi :(. Again, if you will able to prepare short project I can spend my personal time to see how I can help :)
     
  18. Andriod

    Andriod

    Joined:
    Dec 9, 2015
    Posts:
    2
    Complicated or not, the NetworkManager should supply a proper Leave/Quit/Something method. Having everyone make it themselves lowers the value of the NetworkManager as an API. Is there any arguments against adding it to the NetworkManager, except time required?

    Also, the error message is pretty bad. "ServerDisconnected due to error: Timeout" sounds like the server disconnected and not the client. I suggest it should say "ServerDisconnect due to error: Timeout" if changing error messages is not considered a serious backwards-incompatible change.
     
    Last edited: Aug 3, 2017
    xVergilx and thegreatzebadiah like this.