NetworkBehaviour isReady & OnSerialize Needs Robustness Changes

The NetworkBehaviour OnSerialize which controls SyncVars or any custom networked serialization needs some robustness changes. It seems to OK at first, but after using it it appears unrobust with a few bugs. Here's the details of why it breaks.

SyncVar's use dirty bits to keep track if they are dirty, but they can be de-synced 2 ways
1: By late-connecting players or
2: By scene changes on persistent DontDestroyOnLoad objects

Example1:
-Server sets syncVarInt value to 20
-Client1 connects: During OnSerialize/Deserialize Client1 gets the correct latest value of 20 during initialization
-Server sets value to 30. dirtyBit = true
-Client 2 connects: During OnSerialize/Deserialize Client2 gets the correct latest value of 30 during initialization. However the dirtyBits are cleared and the latest data is never sent to Client1. So at this point in time
Server: int = 30
Client1: int = 20 DESYNCED!!!
Client2: int = 30
-Client 1 will remain desynced until next time the value is changed.

Example2: Persistent DontDestroyOnLoad object carried from scene to scene (i.e. player)
-Server sets syncVarInt value to 20
-Client1 connects: During OnSerialize/Deserialize Client1 gets the correct latest value of 20 during initialization
-Server sets value to 30. dirtyBit = true
-Server commands change to scene. Client1 networkConnection.isReady = false
-Any OnSerialize/Deserialize are not sent to Client1 because isReady = false
-After Client1 loads the new scene networkConnection.isReady = true.
-NetworkManager.OnServerReady is called forcing OnSerialize (initial state == true) essentially "spawning" the object a 2nd time.
-Client 1 receives the new value of 30. But what happened is Client1's SyncVar value was changed without the hook being called! (hooks are not called when OnDeserialize initialState == true)

Having the isReady flag set to false during scene change causes other problems for RPC's. If a ClientRpc is called it will only send it to observers where isReady == true. Momentarily during scene change isReady is set to false. So if you call a ClientRpc it is not guaranteed that clients will receive it. For example if you move an item from inventory slot from 10 to 15... and then change scenes... the Rpc will never received and the server/client can be desynced. The host (server + client) can even fail to send messages to itself!. Even if the host RPC works there is a delay (i.e. if you move an item to slot 15 and then access item 15 it will still be in slot 10 for a while).

For things like transform position control it's not a big deal to intermittently miss RPC's, hooks, or have de-synced syncVar values briefly. However, if you try doing anything deterministic (inventory control, dictionaries, lists, etc.) then missing a single message can break it and these robustness issues become a big problem.

Solution ideas:
To fix these issues I think each NetworkBehaviour needs to keep track of each NetworkConnection and only "spawn" on those networkConnections once. Trying to spawn an object multiple times on the same client is likely unintended. Calling OnSerialize (initial state == true) and spawning should not clear dirty bits. Persistent DontDestroyOnLoad objects should not have their observers removed during scene change otherwise the player (or any other persistent object) can miss SyncVar changes and RPC's.

 

Tring to re-structure things so that when an object is spawned for the first time on a client then I do a few things.
1: Send ALL data to the new client as part of the OnSerialize initialState == true spawn message
2: Send incremental data to all other existing clients (but not the new client) using message
3: Clear dirtybits
Then at the end of this all clients should have the exact same state.

All future incremental updates are sent using messages.

This accomplishes...
-There is no double spawning on the same client
-There is no de-syncing of synced data due to scene changes or late-connecting clients.
-Hooks are always called
-More efficient network bandwidth usage

Since this requires custom serialization UNET's SyncVar's & auto generated code cannot be used. But seems to work so far. Hopefully this allows for robust control of deterministic systems (inventory control, lists, dictionaries, etc.) without intermittent failure to send data/hooks.