NetworkTransport NoResources

Hello everyone.
I'm currently working with unity's LLAPI to try and get a basic mmo architecture working.
I'm having the client-server system working using 2 different channels for reliable and unreliable communications and all works fine.
All works fine UNTIL I get a certain amount of data to send (multiple npcs at same place, etc) in which case my server is spamming me with "NoResources" error on NetworkTransport.Send()
(i send one packet per npc per player every 20th of a sec for position update)
I'm having multiple question I would love to get some answer to if anyone could help me :
1 :how can I know WHICH resource is lacking? memory ? network ? hardware ? configuration ?
2 : how can I find proper documentation about unity's LLAPI, the doc on the help is dreadfully hollow and I couldn't find any good tutorial or video about it.
3 : is there any help somewhere about good code practice on networking for mmo's ? (like optimal packet size, grouping packets before sending ? variable tcp packet sizes? etc.)

Any help is welcome. I'm still not experienced in networking and find it hard to get good docs about it

Thanks.

Yoirgl.

 

Okay so, skimming your post i'm seeing "20 messages per second", "MMO" (which usually means thousands of players), "NoResources". The issue here is:

• 20 messages per second is WAY too many. This might be viable in an arena-based game (think battlerite/DOTA/League of legengs etc) where there will only ever be a few players and the synchronization of objects needs to be super fast and accurate. MMO's however, which are normally designed to handle hundreds/thousands of players through various strategies, unfortunately have to be able to deal with much lower send rates in order to account for larger numbers of players. Even 10 messages per second would be too high, but that would be the absolute maximum that I would recommend (getting it down to 2-5 maximum messages per second, per player, would be ideal)
• In multiplayer games, particularly MMOs, the client-side's job is to try and make the world look alive and seamless as much as possible. For example, if you were to only send 5 messages per second to update the position of an object, the client-side is going to have to account for this, to try and make the movement look nice. The way to do this is the interpolate the movement so that is smoothly moves the object to the currently set position (based on the most recent update from the server) over time. How fast the "smoothing" is is the tricky part. Make it too fast and the movement will look jerky. Make it too slow and you'll end up with elastic looking movement.
  
• MMOs are serious business. Be prepared to do alot of the above to be able to account for large numbers of players.

The "NoResource" error is a little misleading. In my experience it doesn't actually relate to system resources (such as CPU or RAM). It seems to relate to internal limits, such as how many buffers/messages each connection can have (based on the ConnectionConfig settings for example). So in some cases raising the config setings can help avoid the NoResource issue, however I absolutely recommend against this. The default settings of Unet are actually really good, so the chances are if you're getting the NoResource error, you need to optimize your networking solution considerably.

You are very right about this. The LLAPI documentation is pretty terrible right now. I spent a long time with trial-and-error before I finally managed to have a robust solution and understand how it all works, and currently that seems to be the only way to get to grips with it. With that being said, i'm on here fairly actively so if there's anything you're stuck with, stick up a question here and i'll hopefully be able to help.

Other than what i've suggested above, there's no one-size-fits-all type thing for MMOs. MMOs push current technology to their absolute limits because the goal is to be able to support as many players as possible, in a game where everything feels as seamless and "real-time" as possible. Ultimately, in order to build an efficient MMO, keeping the costs as low as possible while being able to support large numbers of players will require you to spend the majority of your time optimizing your networking/code to ensure that message counts are as low and as small as possible.

Some tips:

• read up on interest management. It is the most essential part of any MMO's network. It ultimately attempts to ensure that updates about networked objects are only sent to players that are within a certain range
• ensure the data being sent in each message is lean as possible. Only put information in each message that absolutely needs to go in there. For example, when updating information about an object, only send information that has changed (delta compression)
• send byte arrays. Because every piece of run-time data can essentially be broken down into bytes, it's possible for you to convert any piece of information that you want to send over the network into an array of bytes, and then convert it back on the other end.
• the efficiency of your byte arrays is important. For example, converting an object to an XML string, then that string to a byte array would be very easy to do and use, however this would cause your byte array to be bloated as hell. This topic is fairly complex, but the idea is that you should use some sort of stream writer/reader, where you can easily write/read certain basic data types to/from byte arrays. This will keep your resulting byte arrays as low as possible
• every message that you send has it's own overhead of bytes (8 bytes for UDP), and it takes cpu time to both send and receive the message. So, keeping your individual message counts as low as possible is really important. Unity handles some of this for you by trying to combine messages, but you are much better off handling this yourself by just ensuring that message counts are low, or by combining the data of what would have been many small messages into one larger one
• if you're not clued up on it already, you'll need to get clued up on multi-threading. At some point, you're going to need to move some of your network processing off of unity's main thread (and out of the Update loops), otherwise you'll struggle when things get into the thousands. Unity is mainly designed to be used to build single-threaded games, with limited support for multi-threading (enough though to anything related to multi-threading, it's just awkward sometimes), so it can be tricky to do, but you'll need to use it for heavy processes that could otherwise block the update loop

There are potentially millions of other things to consider, but that should be enough to get you started. Let me know if you have more questions.

 

First and foremost, thanks for your long and detailed reply

I'll follow your advice and try to play more with the configs on my game until i get it right both client and server.

I do already apply some of the elements you mention, like sending only data for elements in range, using delta compression, sending byte arrays, efficient byte arrays (Binarywrite all transport classes and optimisation of data types to use smallest space possible) and also do some client prediction and lerping for most effects.
I didn't work on multithreading yet (so i'll give it a go and read about it - if you have any good advice for blogs/articles, i'll take all the help i can get-) nor on stacking messages together because simply i don't really see what i could stack together when sending position updates.

However there are some school-case scenarios that do not apply to my situation as i'm attempting to do this with oculus avatars (VR) and i do need quite a bit of refresh for the position of the avatars themselves so that players can see one another's hand movements and positions.
I could narrow player world position + avatar pose to as small a 50 bytes per update and was hoping I could send this at a strict minimum of 7-10 updates a second.
if we consider a server of 5000 people in worst case scenario, all stacked together,
If i do the maths right, considering a player can see a maximum of 50 other players, the max amount of data (in the worst case scenario) would be for the server 5000players*50otherplayers*10persecond*50bytes = 119MB/s on the server's bandwidth + 10% additional data (spells, custom actions - very small packets) let's say around 150MB/s on the server.
My main issue is the amount of packets as this means 2.5 million packets a second.
Can unity handle that kind of load or do i need to come up with a completely custom one and create my own socket api on C# ?

Thanks !

Yoirgl.