Handling UDP Bytes

I'm reading from a UDP stream where I know data should be coming in a certain order (e.g. there will be one double from bits 32-95 and another from bits 96-159). How can I go about parsing this? I know how to convert everything into one big string, but I was wondering if there was any existing helper methods to pulling out ints, doubles from a byte steam.

 

https://msdn.microsoft.com/en-us/library/system.bitconverter.todouble(v=vs.110).aspx

 

Thank you very much, sir!

If I want to grab individual doubles, do I just have to create multiple different arrays and convert them? I don't see a version of this method with length as a parameter. Otherwise, how would it be able to differentiate a 2 3 from a 23 in a stream?

 

If you have a stream. You can read a set amount straight into that.

 

However, you might want to look into something like Protobuf to send data and serialize and deserialize it. If you have many different message types.

 

Nobody should be writing their own data formats in this day and age. Easy noob mistake to make, but yes use protocol buffers. It's the best all around serialization to use for games for a whole bunch of reasons.

 

Can you elaborate on this? Advantages/disadvantages to using protocol buffers? why is it a noob mistake to not use protocol buffers?

 

It's a noob mistake to re invent the wheel, rolling your own. But then the OP sounds like he was doing that with his networking also, so....

Protocol buffers is kind of the defacto standard for serialization in high performance server frameworks and networking libraries. Not necessarily in the game industry but just generally.

Protocol buffers is space efficient, which is a big deal in games. But the real power is the varint/msb encoding it uses for integers. Which it uses for bools/enums also. Msb encoding lets you encode integers using less then the number of bytes they normally take. It's optimized for the kinds of small numbers games send a lot of over the wire. So combine that with the fact you can convert floats to integers over the wire to take advantage of this (and at the same time sending just the precision you need), and you have pretty much the optimal design for multiplayer games.

 

While your points are not bad. I personally don't recommend Protobuf. It generates too much garbage IMO. (That is if you can't send a stream object directly).

Unless you have found a sweet way to use Protobuf.NET without generating too much garbage. If so, I would be very happy if you shared it. It has caused us alot of issues so far.

 

Thanks for elaborating. I haven't really used things like protobuf. I'm glad to see it has options for sending integers that use less bytes, but I do kind of have similar concerns as @TwoTen here - those libs add a bit of bloat.

In my own project, I wrote my own version of NetWriter that lets me pack ints up to a specific number of bits, not bytes. This is pretty much the most condensed way you can send messages over the wire. I'll admit it's probably more error prone than something like protobuff because you need to make sure you read from the stream in the exact way you wrote it. But overall it's pretty lightweight. One class, and I have tons of control over when I allocate buffers.

 

Yeh, the biggest flaw IMO with Protobuf.NET is that we have to use Streams. So it's a pain for the GC.
But we are currently using Protobuf until we find something better. But we have done our implementation easy to replace.
We us the RecyclableMemoryStream project that a was released by MS instead of using the MemoryStream. And we just have a helper class liket his:

Code (CSharp):

1. using Microsoft.IO;
2. using ProtoBuf;
3. using System.IO;
4.  
5. public class Serialization
6. {
7.     private static RecyclableMemoryStreamManager streamManager = new RecyclableMemoryStreamManager();
8.  
9.     public static T Deserialize<T>(byte[] data) where T : struct
10.     {
11.         try
12.         {
13.             using (MemoryStream stream = streamManager.GetStream())
14.             {
15.                 stream.Write(data, 0, data.Length);
16.                 return Serializer.Deserialize<T>(stream);
17.             }
18.         }
19.         catch
20.         {
21.             throw;
22.         }
23.     }
24.  
25.     public static byte[] Serialize<T>(T record) where T : struct
26.     {
27.         try
28.         {
29.             using (MemoryStream stream = streamManager.GetStream())
30.             {
31.                 Serializer.Serialize(stream, record);
32.                 return stream.ToArray();
33.             }
34.         }
35.         catch
36.         {
37.             throw;
38.         }
39.     }
40. }
41.  

So if we ever need to, we can just replace Protobuf.

 

GC has never been an issue for us. But the whole reason we use protocol buffers really is so we can leverage varints, so we just don't send a lot of data even for hundreds of things moving around. So the GC of serialization is almost white noise compared to other stuff in this case. I would have to believe you are sending a lot of data you really don't need to, to be seeing GC pressure that matters.

 

Outside of fairly uncommon scenarios, you should never be sending whole floats or vectors. We optimize most of our space out by just using movement techniques that either require inputs or just a 2d vector and a heading. Which works for 90% of what most games need to do.

For stuff that is updated often, like incoming movement updates which generally account for around 98% of what you are dealing with, there are straight forward ways to optimize that more.

First one is track on the server which players have seen what, and don't send data if it hasn't changed.

Second one is a big saver, send delta's. So just send the amount moved not a complete position. It's fairly simple to calculate what the maximum movement could be in this case. Which you need to do so the client can detect if a packet was missed. In this we we send a complete position once so the client can resync.

All of that combined and the bandwidth it takes to track 100 moving characters comes in at around 40kbps. Which is why our GC is white noise.

 

Also one more thing that can have a huge impact. Don't send updates to clients as they happen to the clients. Let the update from the client be the trigger to grab everything in range from your spatial grid, and send that as a response. So at every network tick, the client sends and receives one message in each direction. This also leverages protocol buffer arrays better, you get more space savings.

 

Well, optimally, you want to avoid the Heap all together. If we allocate a few KB a frame, that adds up.

As for your optimization suggestions. We send our rotations as integers.
And we don't do delta compression. Cause we send our movement on the Unreliable channel. But what we do is we mark our fields as required = false. Then we pass them to 0. And they are not included and the server knows to just use the same as last time

 

Forgot to be explicit about what I mean't by whole floats/vectors. This is what we use for converting floats/ints and visa versa. It's rare to need more then 2 points of precision. We have our own classes for 2/3/4 dimensional vectors that use integers, and those are what we send over the wire.

Code (csharp):

1.  
2. public static float ToFloat(this int i)
3.         {
4.             return i / 100f;
5.         }
6.  
7.         public static int ToInt(this float i)
8.         {
9.             return (int)(Math.Round(i * 100, 2));
10.         }
11. [/CODE/

 

Compared to? You always have to ask that question in context. Otherwise it's mostly meaningless. The GC created by the unity character controller for example, completely dwarfs serialization.

A few more bytes actually might not matter AT ALL. Because it's what is pushing the collections to take place that matter. This idea that any extra bytes are bad needs to die in a fire. It's extremely dependent on your specific game as to whether it matters and how much.

 

Ye, you're right man.
Also, do you use structs or classes for your objects?

 

Lolz you would ask that. I'm using classes because on the server we have some database stuff that can't serialize structs. Ran into that recently so I just changed them to classes for now. Didn't want to hassle with solving that at this point on a new game But normally it would be structs.

 

40kbps seems a bit high, even for that many characters. how often are you sending updates?

i did a quick calculation here for my game...i send actual 3-D (compressed) position & rotation data at 14 bytes per character snapshot (that's including 4 wasteful bytes for a unet net id). with interpolation on the receiving end, it's pretty easy to bring this down to 15 updates per second without a huge degrade in quality. so that'd be 21kbps for 100 characters. figure a little extra data sent with each message too (message id, character count)

not that i've actually pushed my system this far. i'm doing a 6 player vr game. so 3 transforms per player + maybe 5 other transforms that are always around, and then whatever else the players spawn during a match. so realistically 23-30 things at a time.

i really like talking about bandwidth optimization lol

 

I am also doing 15 a second. Found that to be a sweet balance between quality and quantity.
But I think valve games use 20.

Also, Thanks alot for the float compression tips @snacktime . 2 decimals is plenty.
But is it really worth sending your movement updates on the reliable sequenced channel just to get delta compression?

 

Oh I don't use unity networking. I use DotNetty and raw udp. So delta compression I was referring to is application level.

 

It's been a while but pretty sure rate was 20 per second. You certainly can get better then protobuf if you optimize for specific cases. Although I would argue that it's still the best solution for your general serialization. But outside of certain areas like mobile for example, going that last mile just never showed a huge return. I have done some of that in the past and I just ended up reverting it, because in the larger context it didn't really give me much.

Plus, serialization if you choose something not horrible, is still just one part of the equation. I'd rather see new developers learn space optimization as a more wholistic thing then concentrate just on one aspect of it. And most people new to this, tend to focus on the low level stuff, which is not really where you want to start. Probably the best place to start is asking do I really need to send this, and is there some other way of doing this that could result in sending less.

Like when I realized that for characters, you don't need to send a rotation, just a heading. That's a whole 3d vector you don't have to send at all. And that kind of optimization doesn't care what networking library you are using.

 

i got a little obsessive about bandwidth optimization when i was considering using Unet's relay server. Going the extra mile in that realm literally equates to long-term reduced business expenses. In that context, it was totally worth the day or so I spent writing & debugging my compression class.

 

speaking of cost. Ya in some games bandwidth can be a big cost. And it's an area where it pays to NOT use big cloud providers, because they make a good chunk of profit on bandwidth costs. They do it because cloud hosting is generally a web thing where bandwidth is not a major cost factor.

If you buy bandwidth at colo prices, it's generally around one tenth the cost of a major cloud provider. But you can get some pretty good deals on bandwidth with good cloud providers, you just have to use second tier providers. Which are still generally pretty good, just not well known.

 

Well, If you use the UNET Relay (Which I strongley suggest against), I can honestly see how every byte matters. Every byte will cost you more. I mean too be honest, if you have some basic experiance. You can write something VERY similar for your game and put it on AWS servers. Would probably be fairly easy to do even.

 

Hello, I had tried to use RecyclableMemoryStream, but it seems that implement required .Net 6, How can you use this in Unity?

 

There is no ".Net 6". If mean C# 6, try switching your project from the legacy backend to the .Net 4.x backend.