Demystifying manual NetworkBehaviour serializing

I'm getting a bit bothered by my lack of understanding of what is really going on with NetworkBehaviour serialization. To confirm: I am overriding OnSerialize/Deserialize of a NetworkBehaviour with manual encoding, sync vars are not being used. So far everything does seem to work, but I feel a gap in knowledge about what I am doing and that == future bugs. C#.

1. In OnSerialize, what happens when I write to the buffer and then return false? Should I be returning false before any writes occur?
2. Should I always return false when OnSerialize[initialState] is true, since this seems to imply a full state update for a client and not an actual change in dirty data?
3. Does NetworkSettingsAttribute[sendInterval] actually control the rate of overridden OnSerialize, or is it only for SyncVars?
   
4. Do both the return of OnSerialize and sendInterval simply provide different ways of accomplishing the same task (deferring updates for a later time)?

Thanks for any clarification. This project requires a lot of serialization that isn't appropriate for SyncVars and is preferable to RPCs, and I want to avoid creating a mess of confusion later if I am doing things incorrectly.

 

If your project is preferable to RPC's, why don't you just use RPCs? If you are doing your own serialization, I don't see why it would be necessary to take what you have serialized and then use UNET serialization. Then you would be just sending an array of bytes through UNET?

 

Data is already being successfully serialized/deserialized to/from clients, but I am trying to get clarification on certain behaviours and best practices. I am using NetworkBehaviour serialization because it is suited exactly for synchronizing states, visibility, and when full states vs. partial states need to be sent. It doesn't make sense to try to re-invent these things using RPCs when they already exist to be used.

 

It sounds like you are doing exactly what syncvars already do. What is the exact reason that you want to implement your own serialization? I have a funny feeling that you may be making things alot harder for yourself for no real gain...

To answer your questions though:

As you have already noted via the documentation, return true if there are updates to be sent, and false if nothing is to be sent. So if you notice that a variable has changed since the last OnSerialize call, you would write the value of that variable to the buffer and return true at the end. If no variables have changed, don't write anything to the buffer and simply return false.

I'm not sure what you mean by "should I return false before any writes occur?" because if you are returning false, you would be doing so because there is nothing to update on the object, and therefore you wouldn't be writing anything to the buffer anyway? Let me know if I have misunderstood you here.

The initial state is designed to tell you whether this is the first OnSerialize call on the object or not. If you think about it logically, when an object if first serialized it must send ALL of the networked properties over the network as they may not be the default values of the object itself. For example, an object may start at a position that is not Vector3.zero, and therefore must synchronize it's position at the start.

Maybe you don't need to use this if you are able to detect that it is the first OnSerialize call without needing to use the initial state. Either way though, the same rules apply: if there are values to send, write them to the buffer and return true. Otherwise return false.

That's a tricky one, because there's nothing to state whether this is correct or not. However, knowing that the ConnectionConfig.updateInterval is used to basically compile updates and send them on a semi-fixed interval, I would assume that the same can be said here: that the updates are sent at the maximum frequency of sendInterval. Nothing that you send over the network will ever be sent instantly. It will always be pooled/queued up within the internals of unet. The closest you can get to sending something instantly would be to send it on an AllCostDelivery channel, but even then there will be some intentional delay.

Let me know if you need further clarification on this.

Well no... OnSerialize exposes a stream to you and expects you to return true/false based on whether you wrote anything to the stream. So OnSerialize is a method that you use to compile updates while sendInterval is a property.

The send interval dictates the minimum amount of time that has to pass before another message is sent to a connection over the network. So for example if sendInterval is 10ms, and you write to the buffer in OnSerialize more than once in 10ms, there is still only one update sent over the network to the connection over that 10ms period.


Hopefully this helps. Let me know if you have any further questions.

 

Thanks for your time, donny

It is part a matter of information complexity being more than reasonable to try to work out with SyncVars, and part because I am comfortable with writing/reading network buffers already. SyncVars seem best used when dealing with sparsely-related individual components of information (location, rotation, state of a particle effect, etc.), but for an example I solved with manual serialization, it is a lobby manager object that stores/tracks/updates all of the 'slots' that players can join and are shown in a list. This is tightly coupled groups of data about each slot (name, team, player color, avatar, ready state, etc.). I could have also used RPCs with custom network message objects, but I like the cohesiveness of the lobby object serializing its state via a buffer rather than sending out sequences of RPCs and having to keep track of who needs a full state update (which initialState already tells me).

This is perhaps an area I'm misunderstanding. As I (thought I) understood, setting the behaviour's dirty bits is what would flag the network system to realize my behaviour wants to be serialized again, during the next update interval. But now I also can return false in OnSerialize to apparently not send a network update? For a real-world example, this seems like I am whistling for a taxi, but then saying "Nevermind" when a taxi shows up. You can see why this seems redundant to me, and one reason I feel that I don't really understand it fully. Am I using dirtybits wrong?

Right, again this ties into my assumption I just mentioned: if I didn't want a network update, wouldn't I have not set dirty bits? This fits into the logic of seanr's comment I quoted previously: "if any networkbehaviours on a NetworkIdentity are dirty, then an UpdateVars packet is created for that object"

Maybe it will help reveal the flaw in my logic to illustrate how I am handling my lobby object (again, it works just fine, but I feel maybe it's not "the right way"):

My lobby object has server-only methods to add/remove players, set details like what teams are open, which players are ready, etc. as well as set some meta data (like the room title and game session information). What is contained in the serialization is determined with a 1 byte bitmask that is always serialized first so I know how much data to deserialize on the client, read and loop through as much as necessary, which parts of the UI to update, etc. If it's an initialState, then it all goes through.

When I call any method that changes the lobby's state, I SetDirtyBit(1). I actually always use the value of 1, just so it isn't zero, which sounds sketchy/hacky, but it does seem to do its job making sure that my object's state is serialized next send interval. In OnSerialize, I always return true to clear my dirtybits after the update sends except when someone needs a full state update (initialState == true). As per the docs, returning true clears the behaviour's dirty bits meaning I no longer need a network update, false leaves them alone, presumably meaning I still need a network update? Only when I am sending a client's initial state does it make sense to me why I would return false and not clear my dirtybits.

Hopefully this isn't too headache-inducing. I guess the whole post is strange since I'm writing about something that's actually already working; it just feels like I don't entirely know why it is working. Again, thanks for your time.

 

Hmm okay, so basically you have objects (class instances with a set of properties) that are not game objects, but are objects that you want to synchronize over the network?

If this is the case, yeah you're absolutely making life harder for yourself. You are right in your alternative thinking to use a custom message, and to send your data using an implementation of the MessageBase class (take a look at that here: https://docs.unity3d.com/ScriptReference/Networking.MessageBase.html) to serialize/deserialize your custom object. In the interest using things the way that they were designed, this is exactly how you should be sending your collections of data.

An example of what I mean. The below is an example of how you could implement a class, containing the values that you are talking about, as a class that can very easily be sent over the network (and very efficiently I might add):

Code (CSharp):

1. using UnityEngine;
2. using UnityEngine.Networking;
3.  
4. public class PlayerData : MessageBase {
5.     public string name;
6.     public int team;
7.     public Color color;
8.     //etc...
9.  
10.     public override void Deserialize(NetworkReader reader) {
11.         this.name = reader.ReadString();
12.         this.team = reader.ReadInt32();
13.         this.color = reader.ReadColor();
14.     }
15.  
16.     public override void Serialize(NetworkWriter writer) {
17.         writer.Write(this.name);
18.         writer.Write(this.team);
19.         writer.Write(this.color);
20.     }
21. }

The difficulty here is that sending and receiving MessageBase instances is something that you have to do directly on a NetworkConnection, which ultimately isn't something the HLAPI can do for you. You have to implement this yourself, but again in the interest of doing things the right way, this is exactly how you should be doing it.

So for further reading on how you can make use of the above class:

• https://docs.unity3d.com/ScriptReference/Networking.NetworkConnection.Send.html is the base method that the HLAPI uses to send anything over the network. Note that the second argument takes a MessageBase (or class that inherits from it) instance.
• from the NetworkClient: https://docs.unity3d.com/ScriptReference/Networking.NetworkClient.Send.html, this is just an alias for NetworkClient.connection.Send
• From the NetworkConnection, NetworkClient and NetworkServer/NetworkServerSimple, you register handlers in order to be able to handle incoming messages. All of the above end up calling NetworkConnection.RegisterHandler, as shown here: https://docs.unity3d.com/ScriptReference/Networking.NetworkConnection.UnregisterHandler.html
• The idea is, you would pick a unique short number (a value between 0 and roughly 65k) to identify the sending of your "PlayerData" MessageBase. You would send this via the various Send methods above, and then handle it on the other side within a registered handler. Your registered handler could look like this:

Code (CSharp):

1.     public void MyCustomHandler(NetworkMessage msg) {
2.         PlayerData playerData = msg.ReadMessage<PlayerData>();
3.     }

OnSerialize is probably called every frame, regardless of what is happening within the NetworkBehaviour itself. I don't know exactly what part of Unet initiates the OnSerialize method but I would see no reason not to call it every frame...

So ultimately, if you still want to use your custom implementation of OnSerialize (which by now, i'm hoping you are seeing that you don't need to...) then there is no "dirty bits". In the documentation, "dirty bits" applies to the use of SyncVars. Unity's default implementation of OnSerialize and SyncVars will set any properties flagged as SyncVars to "dirty" when they change, knowing to write them to the buffer in the next OnSerialize call. If you're using a custom OnSerialize implementation, you shouldn't be using syncvars, and therefore the "dirty bits" don't apply to you.

So, to recap: in your OnSerailize custom implementation, you need to check whether the variables that you are sending have changed since the last OnSerialize call (this is what unity does for you if you use SyncVars) and only write those values to the buffer if they change (and therefore return true). Otherwise, you wont write anything to the buffer, and you will return false.

You could also just write to the buffer every OnSerialize call, and not worry about delta compression (only sending variables that have changed). However that would be bad practice.


I feel like your last point should be fairly well explained in my points above. If not, feel free to let me know if you're still suck.

 

Thanks donny, I will try refactoring to use messages instead of behaviour serialization.

Some more digging (since it has really been bugging me and making my obsessiveness go off), which also helped to resolve some of my confusion with how I was interpreting different bits of information on the same subject. The following appear to be true:

• If dirtybits != 0, OnSerialize is called every frame (not every send interval) until they are reset. This means dirtybits do have implications for serializing outside of SyncVars, actually are fundamentally required or you won't call OnSerialize except for initial states.
• However, despite being called every frame, nothing is sent over the network until you return true. This part is truthful and in-line with the docs (apologies to Unity for doubting them, but my interpretations from multiple sources suggested they might not be correct - it's happened before!) I suspect that the actual network send only occurs on the next send interval, not immediately.
• This means that dirtybits are only used to flag the NetBehaviour as potentially ready to send an update over the network. It essentially tells Unity to begin polling the behaviour every frame until you return true, sending the data and flushing the dirtybits.
  
• When initialState == true, it appears that the return value is ignored. This would be logical, as you would not want to postpone the first full-state update to a client.

Ultimately I think most of my confusion was part assumptions and part from other posts via Googling which may have been outdated and/or just not correct to begin with.

You're right that I'm probably trying to do things 'too manually', I am probably focusing too much on the raw data being sent as a stream and deserialized back into per-object data, and not considering that with Unity I can keep the data associated as individual objects without needing me to manually keep track of this. This is probably out of habit from doing strict manual serialization in non-Unity game projects before, where the data is read as monolithic streams and you must encode and decode/route it to the proper objects on your own. I'll probably still find myself treating objects as individual streams since it's what I'm comfortable with, but I should at least be separating them more instead of creating large router objects that are distributing data and creating unnecessary complexity, and making use of messages when it just doesn't make sense to view sparse updates as streams.

Thanks again for your patience!

 

I can only assume that this is the default behaviour, so that if you don't use syncvars, you can expect OnSerialize to be called every frame (since, in theory, dirtybits != 0 will always be the case if you aren't using syncvars and are using custom OnSerialize). So that makes sense.

Yeah that would be true. It doesn't actually make sense to send anything immediately, when considering how sockets work and how inefficient it would be to send a message every time you call a "send" method. Unet's internals do a lot of work to compile small messages into bigger ones, and send all messages in bulk every send interval.

You're 100% correct on that, but again this is in relation to syncvars. The default implementation of unet will set dirtybits != 0 when a syncvar changes, which as you've stated then tells us that there might be something that needs to be sent in the next OnSerialize. It might be worth noting that "dirty" is a fairly common term used in programming, it's not just a term used in unet. A "dirty" object/value is generally something that has changed, and needs attention before it is set back to being "clean". When considering this common concept, it only really makes sense in syncvars, maybe this will help in your understanding.

Hmm I'm not sure what you mean by this. You mean if you write to the buffer and return false, it still sends the data that you wrote to the buffer? If this is what you've discovered i'd suggest testing this when initialState == false, because I assume the same would happen there too (since the buffer is just a stream shared between all objects that get OnSerialize called on them, there would be no way for unet to "un-write" what you've written to the buffer even if you return false... so the chances are it would still send it). If the above is the case, then it strengthens the need to make sure that you are correctly returning true/false, based on whether you actually wrote anythign to the buffer.

In the interest of learning and experience, I implore you to keep discovering and trying things out. Doing that will without a doubt make you stand out from the crowd, but yeah at the same time it's important to be adaptable. At least look at what unity offers and try out what it offers before considering that those options wont suit you and that you'd rather implement your own way of doing things, simply because going with the opposite approach first will always make things more difficult for you.

Hopefully this helps, good luck with the game!