GetIncomingMessageQueueSize Constantly Rising

I'm having a weird issue, originally I thought I was simply trying to put too many messages through the pipe during heavy load and that was causing my "no free events for message" errors however I wrote aggregators/flow controllers for nearly everything in my game so my messages per second are less than a hundred in most heavy load cases.

So with a bandwidth of less than 5kB/s and so few messages being sent I was confused as to why I'd always get a no free events for message error and clients booted after about forty minutes of play.

I added a little counter to my UI showing the current value of NetworkTransport.GetIncomingMessageQueueSize and I noticed that it steadily climbs throughout the game and the client's queue is getting inundated with messages that it never appears to be processing.

This isn't an issue of packets being duplicated or some for loop running away with packet transfers as the buffered/unbuffered messages in values are constant throughout regular gameplay.

My question is why would my message queue be getting plugged up when gameplay appears fine for the client? I do end up with "Did not find target for sync message for x" errors in some cases, though the number of these warnings are not equivalent to the number of messages waiting in the queue.

I'm quite stumped by this problem as it's basically the networking version of a memory leak and once the client hits a seemingly arbitrary value of queued messages, they get booted. I have ReactorMaximumReceivedMessages set to 2048 and the last playtest the client was disconnected at 2345 in the queue. Maybe the incoming message queue size is not the right direction either and I actually am just trying to send too much stuff through the pipe though even with the most synthetic load possible I was below 300 messages per second.

My current network settings are all ridiculous as I set everything to massive numbers hoping it would work:

Code (csharp):

1.  
2.         globalConfig.MaxPacketSize = 1470;
3.         globalConfig.ReactorMaximumReceivedMessages = 2048;
4.         globalConfig.ReactorMaximumSentMessages = 2048;
5.  
6.         connectionConfig.AcksType = ConnectionAcksType.Acks128;
7.         connectionConfig.MaxSentMessageQueueSize = 2048;
8.         connectionConfig.FragmentSize = 1000;
9.         connectionConfig.MaxCombinedReliableMessageCount = 14;
10.         connectionConfig.MaxCombinedReliableMessageSize = 100;
11.         connectionConfig.PacketSize = 1470;
12.         connectionConfig.DisconnectTimeout = 2000;
13.         connectionConfig.PingTimeout = 500;
14.         connectionConfig.ConnectTimeout = 1000;
15.         connectionConfig.MinUpdateTimeout = 10;

and in the individual channels, their max pending buffers are all set to 510.

Any help would be a godsend as I am truly stuck on this issue and have been chasing it for months. Even an idea on how to make a bandaid solution like a networking version of the garbage collector emptying the queue would be great.

 

Those aren't related most likely. Those warnings are from messages that try to apply network state to the object that were destroyed or not instantiated yet on a client.

And for the other actual problem, I think bandwidth limit is 4kb/s per player, and I think this is a Unet issue.

 

The game isn't running through any Unity multiplayer service, it's directly p2p by IPs with their ports forwarded at the moment so I can't see 4kB/s being a hard limit when that would pretty make it impossible to create a shooter in Unity which is clearly not the case.

The game will actually run under a synthetic load of about 300 messages/s and 20kB/s for over an hour, it's just each client's message queue slowly fills up until they're finally disconnected. Now this is a correlation, I can't say that the queue constantly rising in size and hitting some maximum is the reason for the disconnects, it could be something else that suddenly runs out of space after that time frame.

So after writing the first bit of this post I did a lot of testing and found that most likely it actually was related to those warnings. I was trying to figure out what was causing the queue to fill up and it wasn't players, defenses, attacks, or any of my flow controllers/aggregators but specifically when enemies died and their death Rpc couldn't come through before their despawn event which was causing a race condition somehow even though the despawn was on a 4.5 second timer after the onDeath rpc event. Increasing the tickrate of enemies from 5Hz to 20Hz fixed the problem completely as I stopped getting those warnings and my GetIncomingMessageQueueSize never went above 10 throughout the entire playtest in normal conditions.

Putting synthetic load on I can still eek out a sync warning or two but it's much less meaning I might end up with 50 "leaked" messages permanently in the client's queue over the course of an entire match running unrealistic loads which would allow for at least a few hours of nonstop play on the same server. These tests were done with an average RTT of around 200ms which I think is a fair latency to test at so I think I might be able to call this issue fixed to some degree.

Overall I don't know if this counts as a bug or not for the Unity team, but I can say with a bit of confidence that "Did not find target for sync message" errors will slowly fill up a client's message queue and never leave causing an eventual "no free events for message" error on the server followed by a disconnect of the client.

 

I think it is a bug. You should submit your bug report.

 

Which version of Unity are you using? I'm not sure if it relates to this problem but there is a fix for network transport memory leak in the latest version 2017.1.0

 

I'm on 2017.1.0f3 and it has been persistent for many many versions of Unity.

I am now further confused as I have done some playtests while clients testing with me have their GetIncomingMessageQueueSize as a UI element while my host has the outgoing queue size shown. I'm getting the same disconnects as before after the same period of time played but my clients' queue sizes are not "leaking" and constantly work themselves back to 0. My outgoing queue also works itself back to 0 but once the client experiences a disconnect quickly fills up to 1024 and I get no free events for message errors.

What's puzzling to me is that the game is a procedurally generated wave sort of game, so nothing major is changing, my outgoing queue size in regular play never rises to any alarming level. I just cannot for the life of me figure out why after ~40 waves and close to 50 minutes of gameplay people get dropped from the server. If they wait until their connection times out they can reconnect and play for another ~50 minutes before being dropped again.

It really doesn't appear to have anything to do with network stress and I can run levels specifically designed to stress test the networking, such as 8x enemies, impossibly fast firing rates on defenses, massive amounts of defenses that players could never reasonably afford, and I don't get the disconnects until around 30 minutes of gameplay even in those conditions. In fact if I paused the game when the player disconnected and they rejoined, even though the exact same conditions were present upon unpausing, they got another ~30 minutes of gameplay before another disconnect.

Checking the client's output.log I get this error for the disconnect:

Code (csharp):

1. Log: connection {1} has been disconnected by timeout; address {<REMOVED>} time {1574431}, last rec time {1572365} rtt {48} timeout {2000}
2. Unloading 5 Unused Serialized files (Serialized files now loaded: 0)
3. ClientDisconnected due to error: Timeout
4.  

My timeouts are as shown:

Code (csharp):

1.         connectionConfig.DisconnectTimeout = 2000;
2.         connectionConfig.PingTimeout = 500;
3.         connectionConfig.ConnectTimeout = 1000;
4.         connectionConfig.MinUpdateTimeout = 5;

 

@aabramychev I upgraded to 2017.1.0p1 and it does appear to be fixed.

I playtested with the same friend for over an hour on the stress test map and didn't experience a disconnect.

Kind of crazy that a bug I've been experiencing for months and months is finally gone and I no longer have to worry about people reconnecting every now and again. Thanks for the help!

 

Having the same problem while in version 2017.1.0p1...
GetIncomingMessageQueueSize already shows 13k and it's still increasing (10 players online).
Server only gets on avarage 70 incoming packets a second...