New multiplayer transport release

Are you decide to bug when the client reconnects to another servers more than 16 times? Or is there any advice how to reconnect more than 16 times without getting: maximum hosts cannot exceed {16}.

Code (CSharp):

1. using UnityEngine;
2. using UnityEngine.Networking;
3.  
4. public class NetworkClientMy : NetworkClient
5. {
6.     public NetworkClientMy(string _host, int _port) {
7.         Connect(_host, _port);
8.     }
9.  
10. // In 16th ReConnect error occurs: maximum hosts cannot exceed {16}
11.     public void ReConnect(string _host, int _port)
12.     {
13.         if(isConnected)
14.         {
15.             Disconnect();
16. //                   NetworkTransport.RemoveHost(connection.hostId);
17.         }
18.  
19.                Connect(_host, _port);
20. //                  ReconnectToNewHost(_host, _port);
21.     }
22. }

 

Yes, that's what I mean. Something like calling, under the hood:

- For each server's Add[Multicast]Group()/AddHost() with multicast address call:

Code (csharp):

1.  
2.     int multicastTTL = 255;
3.     // For IPv6
4.     setsockopt( socketFD, IPPROTO_IPV6, IPV6_MULTICAST_HOPS, (const char*) &multicastTTL, sizeof(multicastTTL));
5.     unsigned int interfaceIndex = 0;
6.     setsockopt( socketFD, IPPROTO_IPV6, IPV6_MULTICAST_IF, (const char*) &interfaceIndex, sizeof(interfaceIndex));
7.  
8.     // For IPv4
9.     setsockopt( socketFD, IPPROTO_IP, IP_MULTICAST_TTL, (const char*) &multicastTTL, sizeof(multicastTTL));
10.     in_addr_t interface = htonl( INADDR_ANY );
11.     setsockopt( socketFD, IPPROTO_IP, IP_MULTICAST_IF, (const char*) &interface, sizeof(interface));
12.  

- And for each client's Join[Multicast]Group()/Connect() with multicast address call:

Code (csharp):

1.  
2.     // For IPv6 socketFD and struct sockaddr_in6* address
3.     struct ipv6_mreq multicastRequest;
4.     memcpy( &multicastRequest.ipv6mr_multiaddr, &(address->sin6_addr), sizeof(multicastRequest.ipv6mr_multiaddr) );
5.  
6.     multicastRequest.ipv6mr_interface = 0;
7.     setsockopt( socketFD, IPPROTO_IPV6, IPV6_ADD_MEMBERSHIP, (char*) &multicastRequest, sizeof(multicastRequest) );
8.  
9.     // For IPv4 socketFD and struct sockaddr_in* address
10.    struct ip_mreq multicastRequest;
11.    memcpy( &(multicastRequest.imr_multiaddr), &(address->sin_addr), sizeof(multicastRequest.imr_multiaddr) );
12.  
13.     multicastRequest.imr_interface.s_addr = htonl( INADDR_ANY );
14.     setsockopt( socketFD, IPPROTO_IP, IP_ADD_MEMBERSHIP, (char*) &multicastRequest, sizeof(multicastRequest));
15.  

So that a single Send/sendto call to the chosen multicast address would send the message to all subscribed clients.

 

That's really great to know. Thanks so much for all the feedback.

 

We're using the latest build on a real project and so far everything seems to be working quite nicely - our local multiplayer over Wifi is now synchronizing at 60 fps on an average latency of ~1 frame. I can only comment on the LLAPI side of things though since it's a low level implementation and relies on just a very narrow set of API functions. Both clients run the game at 60 fps (all physics and gameplay internally run on a fixed 60 fps time step, which is then interpolated for rendering if necessary) and send limited state updates over on every frame. This is obviously impossible to achieve on the trunk/release version of Unity.

I should be able to share some video or maybe even a build for internal testing later, but here's a screenshot from a video recorded from running the game on two computers on the same wireless network. Basically, they're synchronized within the average 1 frame of latency and object positions are compensated by using the latency information and the motion vectors. Messages are also custom but constructed using the network API MessageBase class and simple types. Local network discovery and matchmaking is handled with broadcasts. Internet-matchmaking is not yet implemented, going to wait for NAT punch through etc. before I want to go there.

By the way, is there any way for me to find out the time when packets arrive on the low level API? Because the information comes in on the update cycle, that's a granularity of 1/60th of a second for ping times which is not very useful on local network. Obviously the old network API was able to show the average latency in milliseconds, but since I'm using the new LLAPI it would seem very redundant to open a secondary connection just to find out the latency of the primary connection.

 

Are that improvements in the Unity5.5.0b10 ?

Because i have a serious bug with QOS channel (and so Transport layer ?). My first QOS channel is reliableSequenced, so it's definetly reliable, but not sequenced ...

I call a CMD function for a client, the CMD will call the same function but in RPC "mode" in the reliableSequencedChannel.

And finally, for 2 differents functions, all in this channel, the others clients will call them in a bad order when recept the final RPC functions.

Is this by concept ?

 

@aabramychev Thank your for your effort ^^. I can't switch the version because you renamed a lot of thing with 5.4->5.5, so i don't want to rename all i have.

i have my own NetworkManager (because your own had a lot of bug in the past (sorry ^^) and didn't convince me at this time).


The 5.5 version gives me the possibility of locking match (and it's ABSOLUTLY necessary for my game) and a lot of work on particuleSystem (light/noise essentially ) so.... I can't switch .

I don't really need this at this moment (take your time, but do your work well ) but in 1-2 months.... maybe ^^".

AND : Yes this is a bug, and it's really anoying because my character works with State/subState, subState are available for somes states (my design ^^") and i receive state and substate activation in the wrong order !

For explaning the bug (but you are working on a new and better version .. maybe there is no more bugs) :

The first channel i add in NetworkServer is QOS.ReliableSequenced

For CMD/RPC i use channel=0 (corresponding to QOS.ReliableSequenced), i tried to specify the channel or to use the default behaviour : it's the same behaviour.

On Client : -CMDActiveSubState(string "nameOfTheSubState");
-CMDChangeState(string "nameOftheState")

The server receive the CMD and so call the method and do (i don't know the call order) :
-RPCActiveSubState(string "nameOfTheSubState");
-RPCChangeState(string "nameOftheState")

On Client that receives RPC : State is changed and subState want to be actived : but the subState in the next state is not available so my character Debug.LogWarning("Substate not available") : it was call in a bad order !


What is interesting in this bug is that :

1- it's totally reproductible until you have the bug (it was not here in previous version of my prototype and i changed nothing ! )
2- I added a lot of RPC/CMD call until the bug appears, maybe it's that ?
3- it seems to not appear with NetworkServer.Send or client.Send or NetworkServer.Spawn because i use this to clearly synchronise some variables when a client connect (create a gameObject and assign it to the corresponding player on all clients).


It's seems to be very difficult to synchronise things with network, so good luck ^^.





(sry for my english )

 

Thanks, I'll see if this works for me - I just want to know the latency in finer granularity than number of frame when the packet was received. I'll try this and let you know.

By the way, should the Android platform work exactly the same as Windows Standalone? I'm getting the kind of network issues when connected to an Android device that make it seem like the SendDelay parameter is being ignored, however I didn't debug this for a long time yet. Our game supports VR so I'd love to run it head to head with two GearVR devices, but network performance is currently the primary roadblock.

 

Well, I didn't get an error that would tell me exactly what was going on - only a very heavily fluctuating latency which to me suggests that maybe somehow the buffers were getting congested the same way they did before the SendDelay fix. I'm building with the Unity 5.4.2f2 network code branch and the same code and app that runs smoothly on Windows-to-Windows seems to now have problems on Windows-to-Android. I can compare results on the 5.4.1 version tomorrow to see if anything has changed on my end, I had previously experienced smooth gameplay on that build between both Windows and Android.

I'm setting SendDelay to 0 and currently sending and processing packets with every Update() at 60fps - this might seem a bit overkill but works well for our local multiplayer since the packets aren't that big.

 

Guys I told this several times I guess but for cross-play only server needs to have all connections in one game

 

With a bit of added testing, SendDelay etc. seems to work fine. I'm experiencing a bit of slowdown on my GearVR version, which was probably causing my latency issues when my client lagged behind my server, so I'll have to profile code on my end. I'm sending multiple packets each frame at 60fps and it's still holding together just fine, most of the time.

Are there any special performance considerations with regards to work being done in the single thread, or specific things to look out for?

 

It seems my Android-Win issue was just something unrelated to the network library itself, and I've not had it replicate since. However, I'm trying to optimize the game performance on Android so any tips on how to maintain high performance with the network library are welcome.

I'm sending 60 messages per second and seem to be hitting 1-3 frames of latency with Update() also running at 60 fps, which is pretty much as good as I'd expect right now. My parameters are set so that SendDelay = 0, IsAcksLong = true, MaxSentMessageQueueSize = 1024 and FragmentSize = 1024. Only using LLAPI with NetworkTransport.Send() with raw messages extended from Networking.MessageBase with automatically generated serialization.

In summary - no problems for me in Windows-Android connection right now, but performance tips welcome.

 

By the way, if there's a 2ms minimum delay before packets are being sent in multi-threaded environment (delay with thread yield), when are they sent on the single threaded (mobile) implementation? Right away (in which case are packets ever combined)? Up to 2ms after frame is finished? First/last thing on every frame?

Basically I'm sometimes sending more than one packet per frame if there are rare state changes and I wonder if they get combined or introduce some delay. I could always combine everything internally to a single payload, but I don't know if that would make any difference.

 

Sorry, I thought the initial post in this thread said 2ms but checking now it says 1ms - I don't mean you have an extra delay, I just wondered when the delays happen - i.e. do they always happen at end of frame, or any context change like between Update() and LateUpdate()?

Assuming I call Send() twice, on Android mobile device, during one frame of rendering
- is the delay before sending the message(s) only 1 ms total?
- will there be only one data packet sent out (if the combined payload is small enough)?
- if I call Send() once during Update() and once during LateUpdate() will there be 1ms or 1ms+1ms delay, and are the packets still combined?

Thanks.

 

Thanks, this makes sense and should be rather trivial for me to implement. Actions in our game are already logically limited so that there can be only one packet of its kind on every frame (i.e. you can either catch the ball if it's free, or you can kick the ball if you're holding it, but not both actions at the same time). Batch sending will simply be a question of holding on to the packets until I have resolved the whole frame.

I'd tell you the performance differential but unfortunately it's quite impossible for me to profile this.

 

@aabramychev

It was my fault for the error on bad order for RPC/Cmd message. I tried with a clean Unity project and that works great !
So sorry ^^.

I wait a new Unity version for testing your new Transport layer

 

@aabramychev So Sorry for the delay. Didn't read your answer till now. We've moved to another solution. Thak you for your interest!

 

Are the new changes in 5.4.3p1 or 5.5. already, or is your 5.4.2p2 version still the latest Unity version that we can test it with? If it's not in the latest version, do you already know when it will be? I am really looking forward to that multithreading part.

 

Sounds good, thanks

 

Hi !

Not for the 5.5 and not before christmas is ok !

I Finally built my game : https://ragequit-studio.itch.io/dolls?password=F9k4>\~u .

But if you test it, sometime (not every time, it's more at 20 hours to 22 hours of the night) Unet doesn't work very well.
We get some disconnections (no timeout, just disconnected for no reason) and the service isn't very stable. Can you check it ?

Can you also fix and be more explicit about hostId ? I fought again this for the LAN mode and the switch betwen Online(Unet)/LAN.

 

Hummm sry, so i try again to be more explicit @,@.

"it is for this experiential build?" Yes i work with Unet for this game, but we have still some issues with your server and the system (disconnection... but why ?). It's a personnal project, i'm student ^^.


The hostId is not very clear. I didn't understand it's purpose. The majority of time i wanted to NetworkTransport.Shutdown() or close the listening of LAN match BUT : hostId caused some errors. (hostId out of bound {0} max {1}, or sometime (because that changes) bound{1} max {0} ... )

Why do you not handle this error automatically ?

the typical example of what i was doing :
-Searching online match
-Switching to LAN match
-Switching to online match by stopping the discovery of LAN match ----> hostId error.
-Create a match (and so close the discovery of LAN match) -> hostId error.
-Close a match and enable listing of LAN match -> hostId error.

So, what i did : i forget NetworkTransport.Shutdown(); and that... works, but why ? i don't know.

Maybe i'm just bad because i don't have so much network knowledge ^^ .

 

But you can test my project, you will see very soon the problem. You are connected, all things are ok and ..... dumb disconnected. And that's not true for every party.

You have more tools on the server side that i have ^^ .

 

Just to see the disconnection occured for no reason.
I can send you my sources, but it's a ton of lines ^^.
Mp if you really want.

 

@aabramychev I am testing the new version with a server and client running on the same computer. I want to send large amounts of packets to the client, so that I can see how much CPU power the server utilizies with ThreadPoolSize=8.

I am running into all kinds of UNET errors, crashes and possibly memory leaks when getting anywhere near 200KB/s already (that's what OSX Network Info shows). Can you advise on which config settings to use to send large amounts of packets to a client running on the same computer? Or is 200KB/s already the limit that can be sent to one client? Or maybe did I find a UNET bug? Or maybe the HLAPI can't keep up with your LLAPI?

Here is my config:


I send messages with a simple script attached to the player:

Code (CSharp):

1. using UnityEngine;
2. using UnityEngine.Networking;
3.  
4. public class Spam : MessageBase {
5.     public static short MsgId = 5000;
6.     public string text;
7. }
8.  
9. public class PacketSpam : NetworkBehaviour {
10.     [TextArea] public string message = "xxxxxxxxxxxxxx";
11.     public int amount = 200;
12.  
13.     public override void OnStartLocalPlayer() {
14.         NetworkManager.singleton.client.RegisterHandler(Spam.MsgId, OnSpam);
15.     }
16.  
17.     [ServerCallback]
18.     void FixedUpdate () {
19.         for (int i = 0; i < amount; ++i) {
20.             var msg = new Spam();
21.             msg.text = message + i;
22.             connectionToClient.Send(Spam.MsgId, msg);
23.         }
24.     }
25.  
26.     void OnSpam(NetworkMessage netMsg) {
27.         var msg = netMsg.ReadMessage<Spam>();
28.     }
29. }

Sending 100 messages per FixedUpdate works.
Sending 200 messages per FixedUpdate works for 10-20 seconds, then UNET errors appear.
Sending 500 messages per FixedUpdate causes errors almost immediately.

Server Errors:

Client Errors:

(I reported this one a long time ago on the forums and on the bug tracker: https://issuetracker.unity3d.com/is...lash-readbytes-out-of-range-errors-in-clients . It's HLAPI so probably not your problem)

I can also send you the project if you want, it's just a NetworkManager, a Player and two scripts.

 

Personally, what I want the most at the moment is not a feature, but a proof of performance and reliability of the basic UNET functions. I'd love to see a publicly-available UNET test game project that we can all join and stress-test. As long as it is pretty demanding network-wise, it wouldn't really matter what the game is. I think a networking system like this is just too vast and complex for simple closed-off tests to mean anything. It has to be tested in a real-life scenario (aka a complete game)

I think lots of people in here would feel more comfortable using UNET if we had something like that. People don't trust it enough at the moment, and many are still looking for alternatives

 

Thanks, I will try your suggestions and send you the project when I am back home later today.

I have some suggestions based on my experience with uMMORPG, uMOBA and VOXL.

Features:
I was able to implement just about any feature that I needed already, thanks to the HLAPI source being available on Bitbucket (which could be updated every now and then by the way). The last feature that I would like to see is an option for SyncVars and SyncLists to only sync to the owner player instead of all players around him. This would save so much bandwidth. Think about an MMORPG inventory. SyncLists make it very easy to implement, but sending it to all players instead of just the owner player is a huge waste of bandwidth.

Stability:
(this is pretty much all HLAPI since I can't see the LLAPI source code)
But what's most important really is the stability. This is the obvious answer and what everyone on the forums is constantly complaining about and while it's gotten better lately, it's still not enough.

Take a look at the 'curl' library. The developers have to be ultra careful because every tiny bug would be a disaster. This is what we want. We want you to treat the code with care so that we can have a 100% perfect UNET library. This includes the practice of always fixing all existing UNET bugs before adding any new feature. Good engineering practices like splitting everything into small, reusable functions, white box tests where possible, using modern C# features to keep the code short (less code == less bugs), etc.

Reality Check
We would all love to see a small UNET project that uses all the features, and testing it under heavy load 24/7. If you don't have the time to develop test projects, talk to us. I would be more than happy to test uMMORPG on your servers with some heavy load, or develop a simple UNET stress test project.

Disclaimer
I saw your Unite talk alex, and you mentioned that the HLAPI was meant as an example and that it's really all about the LLAPI. This explains everything and kinda sucks for you guys since everyone here, including myself, is giving you such a hard time for something that was only meant as an example. However it's clear that we all here really need a great HLAPI too, hence all the whining.

 

There is also positive feedback of course. UNET makes it possible for a single developer to work on multiplayer games without going crazy, that's a big deal.

Your features sound good. I didn't mean that you specifically shouldn't add any new features, especially since some bugs aren't even your work.

About the ReadString bug: yes it usually happens when we have a player with for example 10 NetworkBehaviours attached to it. Now if we decide that our 3rd NetworkBehaviour needs a custom OnSerialize/OnDeserialize function and accidentally make a mistake there, then the serialization of NetworkBehaviour 4-10 will be completely messed up, reading invalid values.

There are two problems here exactly:

• First of all, the 10 NetworkBehaviours should be completely safe from each other. If NetworkBehaviour 3 has a serialization error, then NetworkBehaviour 1,2, ,4,5,6,7,8,9,10 should still work fine, even if NetworkBehaviour 3 fails. For example:
  • Right now we have one big packet like: [NetworkBehaviour1.health, NetworkBehaviour1.mana, NetworkBehaviour2.position, ...] . If NetworkBehaviour1 never reads the mana properly, then NetworkBehaviour2 will read the mana, even though it expected the position, hence everything fails horribly.
  • Instead we should have one big packet like: [NetworkBehaviour1.data, NetworkBehaviour2.data], which is then used like: NetworkBehaviour1.OnDeserialize(NetworkBehaviour1.data); NetworkBehaviour2.OnDeserialize(NetworkBehaviour2.data). If NetworkBehaviour1 messes up and doesn't read the whole data, or reads too much data, it doesn't matter. NetworkBehaviour2 will still have it's own data that wasn't touched by anyone else.
  • In other words: before calling OnDeserialize, split the data into 10 independent datas. If one messes up, it doesn't matter.
• The second problem: if we get an error in OnDeserialize, then we don't see it in the console, we never know that there was an error. For example, if we have an error in NetworkBehaviour3, we don't see "error in NetworkBehaviour 3.OnDeserialize". It fails silently, then the data for NetworkBehaviour4,5,6,7,8,9,10 get corrupt, and then we see an error like "NetworkBehaviour7: can't cast int to Vector3", even though the code in NetworkBehaviour7 is totally correct, it only got corrupted data. So any error in OnDeserialize or OnSerialize should immediately be shown in the Unity engine, so that we know that we made a mistake somewhere.

About the 200KB/s problem from above:
I tried it with only the LLAPI. I didn't get any errors, only the "no free events" message. I adjusted the bandwidth settings like you said, but that didn't make any difference. Then I adjusted the SendDelay to from 10 to 0, which worked perfectly:


The "no free events" message does appear around the 25MB/s rate again though. But that's pretty good already.

I was also able to reproduce the Unity crash. It happens when I set MinUpdateTimeout to 0 and then start the server. But I guess it doesn't matter if you want to remove that parameter anyway.

In the end I used the same configs that allowed me to send 20MB/s for the HLAPI test. It still shows all kinds of errors when reaching the 200KB/s mark. So it works in the LLAPI, but not when sending over the HLAPI with the same configs. I sent you the project via PM.

 

@aabramychev
Is there any estimate on when will the server DLL be released? Is it Unity 5.6? Will it also be opensource? As far as I understand, the server DLL is just a LLAPI implementation that could be used outside of Unity, is that correct? Will it be available for everyone, or only on a per-request basis?
Sorry for a lot of questions, but server DLL is just something I was waiting for since UNet release

Thanks beforehand.