4x Latency usgin LLAPI (transport layer) in Unity 5.60f3

Hi, I have been working in some optimization comparison tests using Unity's LLAPI (Transport Layer) and I noticed that since I updated to Unity 5.60f3 today, my latency has gone up by 4 times compared to how it was in Unity's previous version.

I ran a very basic test just to gauge the performance of the new Network changes in Unity 5.6.
It is just an Echo server (that echoes back a message as soon as possible) and a Latency testing client. The whole purpose of these two projects is to merely test the response time (Ping, latency, ...).

I used to get around 15ms with both server and client in the same machine (127.0.0.1:8000). Now with the exact same test using 5.60f3 I'm getting around 65ms. I ran several tests with both Unity versions on the exact same machine today.

I mean using legacy (Raknet) RPCs within Unity gives me a lower latency at around 30ms. I'm sure there's some type of bug there that needs to be fixed. Please help us with that, my whole focus with Unity revolves around network optimizations while keeping everything simple and accessible to most users, hence why I use LLAPI and not HLAPI for simplicity or some custom Network library DLL for a lower level approach and performance.

Thank you.

 

Interesting find, and thank you for taking the time to do these tests!

I have to admit that overall, I feel like the networking in 5.6 does perform better but I do get spikes where it slows down significantly. I mentioned this to @aabramychev and he mentioned that there is one major performance improvement coming in patch 1, and a few minor things to fix.

It is a bit concerning if performance is diminishing... so it would be good to see your test results again after patch 1.

 

Thanks for the reply, looking forward to the new changes.

Maybe to complement a little on the tests environment information.

I'm running Windows 10, and before the update I was using VS2015 Community edition, and after the Unity update it installed VS2017. Maybe some updates in the VS suite may affect some things when compiling code for Unity.

I also ran a test on a completely new project, just importing the scripts hoping that maybe migrating the project to the new version may have had something to do with the performance issue but no luck, latency times are still around 65ms.

For the sake of checking things off, I also ran the test on Linux (server and client) and sadly it also shows the same latency.
$ uname -a
Linux arch 4.10.8-1-ARCH #1 SMP PREEMPT Fri Mar 31 16:50:19 CEST 2017 x86_64 GNU/Linux


Thanks again for the help, and if i can be do anything to further help please let me know, I will keep running tests and comparing different solutions for networking on my end.

 

Just to add to the pile, I was just running some tests trying to get latency as low as possible. With two builds on the same computer I get a RTT of 16ms to 36ms in Unity 5.5. Same project upgraded to 5.6.0f3 and I'm getting 30ms to 50ms (Sometimes even 80ms). It sucks to see things moving backwards when I was hoping to see better performance. A little more inspection and I notice that in 5.5 NetworkClient.GetRTT() is giving me 0ms while in 5.6 it will return up to 10ms or so.

 

@RZer0S @donnysobonny I've found setting the new SendDelay property to 0 alleviates all of the newly introduced latency. Now if only I could find a way to get rid of the old latency..

 

@thegreatzebadiah , Thanks a lot for the tip. Ill try it out right away.

Regarding the old Latency, I'm assuming your'e talking about the previous 15 ms latency.
Sadly as simple as I could get anything working on LLAPI was not good enough for me.

I really didn't like that so I Started working on my own Asynchronous UDP library and I'm also adding functionality for reliable UDP, pushing types into a message in a reliable and fast way (Not Serializing, way way faster and with a much smaller footprint than that) and pushing Async messages to Main Thread (Unity does NOT like things that do not run on main thread).
So far its going great, I'm getting an average latency of 0.5ms highest latency was 1.1ms and it does not spike under load so far XD.

Ill probably make some videos on that and also post on my Blog Game-Savvy eventually if you would want to check it out XD.

 

@RZer0S We're locked into UNet right now for our current release but you can bet your ass I will be researching every possible solution before starting another networked game.

It's looking like we're just going to have to rework our matchmaking so that relays are avoided almost always. Players may have to wait longer to get in a game but at least the latency won't be god awful. If we could just get rid around this extra bit of wait time at every hop it would all be gravy and we could allow relay connections no problem..but such is life.

 

I think i figured out why the constant 15 to 30 ms latency is, its not really a latency in the actual message arrival through the network, is more dependent on the time its takes the Main-Thread (MT) to push the Async messages through so Unity does not complain.

Basically Messages are received Async, pushed into a Queue/List or something and a MT reader pushes them back to MT. I don't really know why its so redundant in the sense of calling the Receive() function should just read these messages out of that Queue and the pace that we want to (OnUpdate, OnFixedUpdate, or OnCoroutine, NO Separate Thread). I think its the time it takes to validate if it needs to send back an Ack as RUDP, weird thing is that it also happens when Unreliable is selected.

@thegreatzebadiah : I finally finished my Library and usage is very simple and similar to LLAPI, this allowed me to learn why LLAPI was lagging so much, maybe there will be future updates that make this latency lower for LLAPI. As far as my library concerns the average latency now is 3ms and spikes up to 11ms every now and then including RUDP and pushing messages to MT. Although my RDUP solution is limited to received messages, it does not care about the order though.

Looking forward to see the new changes on the LLAPI, hopefully everything gets fixed soon.

 

Ok, so the networking is all on a separate thread, and there is some time to get any info back to the main thread, which is necessary because unity is not thread safe so not being on the main thread could cause crashes. So it seems like what I want is a way to tie in to the actual networking thread so I can avoid the waiting when I can be sure whatever I'm doing is thread safe. I'm wondering if that would actually gain me anything though, I don't understand threading well enough to know if I'm just going to see that overhead anyway when I try and read / write some variable that's also used on the main thread.

 

Hi @aabramychev , thanks for the concern, Tes is as follows:

Using specifically Unity 5.60f3
The Receive function is called every 3ms in a coroutine, the coroutine is on main thread (obviously XP), VSync is off, Network sendrate = 60 (even when i bump it up to 200 or down to 5 does not change much).
SendDelay = 0, running on fastest quality settings (should not matter at all).
I understand that when pushing things to main thread, it might take more time to push it than to actually receive the message from the network, but from a custom solution im getting ~ 3ms (rUDP) while using LLAPI is giving me ~ 40ms.
Even back in the day when i was using Legacy Network (rakNet) it used to give me ~ 15ms which is till better.

Can you tell us what config You are using, because none of ours seems to be good enough. Maybe we're missing something.

Thank you!

 

@aabramychev
Just to clarify, the 60, 200 and 5 are not frames, they are ConnectionConfig -> sendrate .
The Receive function is being called in a Coroutine every 3ms and I ran some test calling it every 1 ms, still made no real or measurable difference in the round-trip time at ~ 40 ms and very volatile too. Now this is all for LLAPI.

Running in a very similar setup a custom .NET sockets solution, the Async time is under 1ms, but to be able to push it to MT (in both the Server and Client) it takes the round trip a total average time of 5ms including reliable messages.

Are you sure you're using the same Unity version 5.60f3? Because this was introduced in this version, previous versions had a very stable latency at around 15 ms, which is not bad.

 

Hi thanks for taking a look at the project.

As fast as i could get it, still remains around 30 most of the time then drops to around 15 and rises to 45 at every now and then. A little volatile for such a simple program.

Here's the Project:
https://drive.google.com/open?id=0B-KlQOyy1slhak1nTG92N3R4T00

 

Hi @aabramychev , thanks for the detailed explanation, sounds good to me to have almost 0 real latency.
And going around 7ms to 20ms spikes to actually being able to read a message is pretty good.
Thanks for your help, really helps to know these things X)