Difference between Ping / GetRTT and actual round trip time

I've been experimenting with networking settings trying get latency as low as possible. I've run into a wall though and I'm really not sure what's going on or how to figure out what's going on or if there's anything I can do about it!

I put together a little test project that just sends rpc's back and forth.
I output the time it takes messages to get from client to host and back to the client.

Running the two builds on the same computer I get something like 16-33ms.

I ping myself and I get 0ms of course. I output NetworkClient.GetRTT() and I get 0ms of course.

So where is that extra time being spent and is there anything at all I can do about it?

Things I've tried:
Upgrading from Unity 5.5 to 5.6 (makes it worse)
Setting Max Delay to 0
Setting Min Update Timeout to 1ms
AckDelay
SendDelay
NetworkConnection.FlushChannels()
Sending NetworkMessages
Sending Command / RPC
Sending NetworkWriter
SendBytes
Changing Reactor Model
Unreliable channel
sendInterval = 0
GetNetworkTimestamp()
Time.realTimeSinceStartup
StopWatch
Manually calling UNetStaticUpdate via reflection

Nothing seems to make any difference at all. I'm completely out of ideas. If anyone wants to recreate the issue I attached a sample project.

I'd really appreciate any insight into this. 30ms doesn't make much of a difference when there's no latency and only one hop but when you're dealing with owner->relay->host->relay->other client->relay->host->relay->owner for two non-host players to interact, every ms counts.

 

Hmm... I'm going to guess that considering the tests you've done, the cpu time could be generating your 16-33ms. For example, I don't use RPCs, I use lower-level stuff such as the NetworkClient/NetworkServerSimple.SendWriter methods to send data over the network because I know that the only overhead is the construction of a NetworkWriter object on the sending side, and a NetworkReader on the receiving end. I mentioned this overhead to @aabramychev as it could be reduced by re-using cached instances, hopefully this is something that gets improved in patch 1.

Personally, when I test locally, I see a latency of between 2-3ms, so that would suggest that the processing required to process PRCs is a little heavier. If you think about the logic of how RPCs work this makes sense, since at the very least they are going to need to do some form of reflection. Reflection isn't particularly cpu friendly...

I could be totally wrong here, but it would be good to know if you get the same results with NetworkClient.SendWriter etc?

 

Looking into that right now @donnysobonny, thanks for the suggestion

 

@donnysobonny
Yeah, no such luck. It seemed promising enough since there really are some reflections and dictionary lookups in there that can be avoided. Unfortunately it doesn't seem to actually gain me anything.

I just updated the attached project to include a NetworkWriter test. I use a custom network connection class so I can catch the bytes in TransportReceive(). I send the time from the host to client, then send it back from client to host as soon as it is received. Still takes 17-34ms.

Can you maybe take a look at my code and see if I'm doing anything wrong (it's really not much code). Or maybe just build it and run it locally and see if you get the same results? Maybe there's just some problem with my local network.

 

It is in fact so little code that I might as well post it here:

Code (CSharp):

1. public class NetworkManager : UnityEngine.Networking.NetworkManager
2. {
3.     static Stopwatch sw = new Stopwatch();
4.     static float elapsedTime;
5.     static byte error;
6.  
7.     public override void OnStartServer()
8.     {
9.         NetworkServer.SetNetworkConnectionClass<CustomConnection>();
10.     }
11.  
12.     public override void OnStartClient(NetworkClient client)
13.     {
14.         client.SetNetworkConnectionClass<CustomConnection>();
15.     }
16.  
17.     public override void OnServerConnect(NetworkConnection conn)
18.     {
19.         if (conn.hostId != -1) StartCoroutine(continuouslyPingClient(conn));
20.     }
21.  
22.     IEnumerator continuouslyPingClient(NetworkConnection conn)
23.     {
24.         while (gameObject.activeSelf)
25.         {
26.             sw.Reset(); sw.Start();
27.             NetworkTransport.Send(conn.hostId, conn.connectionId, 0, new byte[] { 1 }, 1, out error);
28.  
29.             yield return new WaitForSeconds(1.0f);
30.         }
31.     }
32.  
33.     private void OnGUI()
34.     {
35.         GUI.Label(new Rect(500, 10, 100, 100), elapsedTime.ToString());
36.     }
37.  
38.     public static void ReceiveBytes(byte[] bytes, int receivedSize)
39.     {
40.         if (!NetworkServer.active)
41.         {
42.             NetworkTransport.Send(singleton.client.connection.hostId, singleton.client.connection.connectionId, 0, bytes, 1, out error);
43.         }
44.         else
45.         {
46.             sw.Stop();
47.             elapsedTime = sw.ElapsedMilliseconds;
48.         }
49.     }
50. }

Code (CSharp):

1. class CustomConnection : NetworkConnection
2. {
3.     public override void TransportRecieve(byte[] bytes, int numBytes, int channelId)
4.     {
5.         NetworkManager.ReceiveBytes(bytes, numBytes);
6.     }
7. }

I'm even using NetworkTransport.Send to avoid the overhead of wrapping the bytes in a NetworkReader / Writer. No effect

 

I just tried testing with regular old sockets and I get right around 0ms pretty consistently. So it's not my network (not that it ever should have been for connecting two builds on the same PC), and it's not just standard overhead. It's something happening somewhere between the socket layer and LLAPI as far as I can tell.

 

Hmm, I haven't actually tested your code, but i've had a look into my solution, which is different from yours and does perform better from a lower-level, but I see the same issue. I have a funny feeling though that this might be an intentional change, since in real-time games you want latency, at least some, because this allows the internals of the framework to combine-up small messages into larger ones making for more efficient real-time networking (fewer large messages are better than more small messages, due to the packet overhead). Again though, I could be totally wrong and it could be an unintentional bug. Either way, we really need some input from @aabramychev on this one to clarify.

To give you a few tips on the route that you're taking, particularly since you are looking for the best performance:

• if you want websockets, you will need to use the NetworkClient and NetworkServerSimple classes at minimum (check this post: https://forum.unity3d.com/threads/client-side-websocket-support-with-the-llapi.457081)
• avoid using the NetworkManager. It has a lot of bloat that you can absolutely avoid to increase performance
• avoid using RPCs, Commands and Syncvars. They are convenience tools, and can be avoided
• instead, use messages. Check out the NetworkClient.RegisterHandler and NetworkServerSimple.RegisterHandler methods, these allow you to register a delegate to listen for certain message types. You can use your SendWriter methods on the sending-end to send very efficiently. Don't worry too much about the construction of the reader/writer overhead, it's currently only 32 bytes of garbage which I suspect will be fixed in a nearby patch
• if you don't need websockets, avoid everything HLAPI, and go all-out LLAPI. The HLAPI is built mainly for convenience, so even if you use it a low-ish level as described above, you'll still inherit performance reductions

Hopefully this helps you, let's wait for Alex's response on your findings.

 

@donnysobonny Thanks for the tips but I'm already doing all that. I'm using the NetworkManager in that example just for OnStartServer/Client but the actual sending is done via NetworkTransport.Send which is the lowest level you can get in UNet.

Yeah, you're not the first person to tell me this, and I'm sure you're right, I would just like a way to configure how long it waits or maybe have a high priority queue that doesn't wait at all. I _thought_ that's what some of the settings would let me do (such as minUpdateTimeout) but I've tried everything.

As an additional note, in 5.6 the RTT is worse unless I set the new sendDelay parameter to 0. The sendDelay seems to be exactly what you're describing, but apparently it is separate from the delay I'm seeing.

 

@aabramychev Sorry I'm not sure what you mean by that so the answer is probably no. Can you elaborate for a simpleton like me?

You can see the full code I'm using earlier in the thread. It's just a few lines.

It's as low level as I could get, using NetworkTransport.Send and a custom NetworkConnection to TransportReceive.

P.S. Thank you for seeing this post and responding!

 

How to I get / set the main thread frame rate? Is that Time.fixedDeltaTime or Time.deltaTime or something else?

I will say though, the time that I'm interested in isn't really RTT, it's the time it takes between sending the message and actually being able to do anything with it, which would include this frame time I guess.

I do understand much better now why my pure socket example was so much faster though.

 

Yeah, I think I understand the general idea at least, just trying to nail down the specifics. So the frame duration is Time.deltaTime?

Also, is there any possibility at all of getting the message the moment it is received, rather than waiting for the main thread to poll for it? I understand unity is not thread safe so this is dangerous, but as long as I only do thread safe things it should be fine right?

 

I have done exactly that and the remote delay each way seems to add up to the round trip time so that all makes perfect sense.

Thank you for helping me understand everything. Almost everyone I asked gave me misinformation about sendDelay and grouping messages and stuff that had nothing to do with the actual problem so it's great to hear from someone on the inside who actually knows.

I see exactly what you're saying about the threading too. It really wouldn't matter if I could receive the message immediately, nothing I could do with it would be able to effect anything until the next main thread update anyway.

Except for one thing! On the host there are a lot of messages that I receive from the owner and immediately echo back out to all clients. I think it might save a not-insignificant amount of time if I could receive those messages and send them back out without ever having to hit the main thread at all.

 

That is great to hear. Alternatively though, at least for my specific case, it may be safer / easier to implement some way for clients to say "I want this command / NetworkMessage to be echoed from the host to other clients without touching the main thread at all" if that makes any sense. The callback would be a more generic solution that could probably be used for other neat things that I haven't even thought of, but I think you're right about users abusing it. Even with a warning in the doc it's just inevitable.

 

Unfortunately I don't only need an echo server, some messages get echoed immediately back out but not all.

 

Hi again. Are there any solid plans for this to be added in the next couple of releases? I'd really love to be able to take advantage of this to shave off some critical milliseconds. We're getting close to locking everything down for release though, so if it isn't coming soon I won't be able to get it in and it will haunt me forever that our networking could have been potentially improved

Alternatively would it be possible to use the server.dll in parallel with the normal unet stuff on a host so that I can get the callback that way? Also..is the server.dll released yet and I'm just terrible at googling or is that still coming soon as well?

Thank you

 

Bump @aabramychev

 

Same here. A function to force "sends" and "receives" would be essential for me. The current RTT of UNET makes it hard to implement some networked games. Not to mention the RTT variance...

I plan on using LiteNetLib or Lidgren.

@aabramychev Any updates?

 

I have the same issue, would love to be able to send & receive messages without waiting for the next frame.

Currently I have a game server and client that both use Unity, and there is a significant extra lag in the round trip time due to both the server and the client having to wait for their next frames. Even if there is nothing for the client to do until the next frame, having the server respond more quickly could get the message to the client in time for an earlier frame.

 

Hello.
When 2018.1 will be available? We really need this feature.

 

@aabramychev Hi there, I'm trying to apply Lag Compensation for my project, but I can't if Client has VSync turned on, which means that messages are waiting for the next frame, so the difference between client rewind and server position rewind goes into 0.01f - 0.03f instead of 0.0001f when VSync is turned on.
We need "to be able to send & receive messages without waiting for the next frame."

 

Actually my bad. The problem was fixed by considering Time.smoothDeltaTime in the InputPacket, because of the FPS difference Server thinks that Interpolation goes quickly since interpolation is calculated in the Update() method. when VSync turned off delta is only 1ms, but when it's on delta is 13ms. 13 ms makes a lot of difference in fast placed game, that's why I had ~ 0.2f difference. Now I got values like that:

Code (CSharp):

1. Client shot time (1093.263): (-42.5613, 1.5001, 32.2621)
2. Server rewind (Rewinded in 1093.263): (-42.5604, 1.5001, 32.2614)