Network problem, buffer overflow (in 5.1.3)

Hello!

We're running the online multiplayer shooter (webplayer clients, stack of authoritative Linux Unet servers), and now players started to report "jittery" behaviour during gameplay.

We use reliable sequenced QoS channel (with index 0) for sending game events and unreliable sequenced QoS channel (with index 4) for syncing vars.

We had similar issues before, they were cured with upping the values of NetworkDropThreshold and OverflowDropThreshold, but it seems that the new factor is introduced now.

We now tend to think that the problem is caused on (our) server side - the packet buffer gets flooded, probably due to poor connection with one or several clients, and then server start spamming the log with "no free events for message" entries so intensively that is causes lags on client side.

Less frequently, we also get loads of following log messages: "Warning: received system packet belongs to wrong session"

What would you suggest to handle this buffer overflow?

------
Please check below the log examples:

===
...
Log: connection {8} has been disconnected by timeout; address {178.134.72.3:60920}
aissp: time {1098566} lastRecTime {1078263} DisTme {20000}, rtt {93}

(Filename: Line: 1845)

Player [IP: 178.134.72.3 disconnect from 95.213.162.94 GUID: 8 ]

no free events for message

(Filename: Line: 358)

no free events for message

(Filename: Line: 358)

no free events for message

(Filename: Line: 358)

no free events for message

(Filename: Line: 358)

no free events for message

(Filename: Line: 358)

no free events for message

(Filename: Line: 358)

no free events for message

(Filename: Line: 358)

no free events for message

(Filename: Line: 358)

no free events for message

(Filename: Line: 358)

no free events for message

(Filename: Line: 358)

no free events for message

(Filename: Line: 358)

no free events for message

(Filename: Line: 358)

no free events for message

(Filename: Line: 358)

no free events for message

(Filename: Line: 358)

no free events for message

(Filename: Line: 358)

Failed to send internal buffer conn:3 channel:4 bytesToSend:1130

Send Error: 4 conn:3 channel:4 bytesToSend:1130

no free events for message

(Filename: Line: 358)

 

no free events for message could indicate that your MessageQueue is running full before being able to send. The default limit on this is 128.

Try putting connectionConfig.MaxSentMessageQueueSize = 256; (or a higher number) in the Awake() of your Networkmanager to see if it makes a difference.

 

Thank you for you answer, Larus.

Please find below some additional information about our connection parameterss.

Our connection configuration:
AckDelay = 33
AllCostTimeout = 20
ChannelCount = 0
Connection timeout = 10000
Disconnect timeout = 20000
Fragment size = 500
Is Ack long = True
Max combined reliable message count = 10
Max combined reliable message size = 100
Max connection attempt = 32
Max sent message queue size = 2048
Min update timeout = 20
Network drop threshold = 40 // we had to set these high to avoid UNet disconnects during lag spikes
Overflow drop threshold = 40 //
Packet size = 1500
Ping timeout = 500
Reduced ping timeout = 100
Resend timeout = 500

The problem usually happens when there are 8 or more players connected. When this number is lower, everything works fine. During the battle server normally sends around 20-80 custom messages per second (4-30 bytes each) via reliable sequenced channel, but number of messages jumps to 120-150 per sec when there's intense gunfight. Besides that there is somewhat like 500-700 (5-7 per client, 10 times per second) NetworkBehaviour objects serializations per second (over unreliable sequenced channel).

As for the disconnect - yes, there was no "No free events" errors before that. Moreover, this typically happens after server receives "Client is ready" message (in OnServerReady callback). As far as we could guess after this client starts receving spawn data and, supposedly, connection get stuck which causes disconnect.

But we also sometimes see disconnects during battles even if initialization was successful.

 

Thanks Scorr, will try that!

 

We're investigating "no free events" as well, and we believe that when setting the ConnectionConfig.MaxSentMessageQueueSize to a non-default value, it may actually make things worse in some cases, since Networking.HostTopology.MessagePoolSizeGrowthFactor is ignored (Case 773411), which leads to the pools used to process the events in the LLAPI to be too small relative to the number received, and so the client will discard many of the events, and cause a lot of resends if the server sends a lot at once. We are testing setting our HostTopology pool sizes to match our ConnectionConfig.MaxSentMessageQueueSize

Any particular reason these can't be larger? I'm considering using our ConnectionConfig.PacketSize for MaxCombinedReliableMessageSize and MaxCombinedReliableMessageCount to be orders of magnitude larger.