[Open Source] UdpKit, a networking library for .NET/Mono/Unity

No problem!

Networking for games in general has very little public information available (there are almost no books, very few resources on the web, etc.), most of us that know this well have just learned it by a lot of trial and error.

I think the Lidgren networking library can do UDP discovery by default, you can find a basic setup example of it right here: https://github.com/fholm/unityassets/tree/master/LidgrenExample

You could try asking on the gamedev.net multiplayer forum on someone to help you with this specifically.

 

Hello, I've been using uLink for my project mostly but realized I'd need my own solutions for lots of stuff down the line

I'd like to ask, for someone with no experience on working with low-level libraries, would it be a hard task (in complexity, not duration) to get things like server-side physics implemented with UdpKit.

Also, is there somewhere I can read up on developing from low-level + Unity integration?

Apologizes if any of this questions seems trivial, Unity is my first proper attempt at programming and everything else outside of its "scope" gets a bit complicated for me.

Thanks in advance.

 

If you give examples on the things you feel like you need your own/custom solutions for maybe I could advice on specifics.

Well, UdpKit works on a much lower level then uLink for example. You would most likely run UdpKit inside a Unity instance as the server also. This means you would get "server physics" the same way you get them in uLink, through Unity itself.

If you want to run a server outside of Unity, then yes it is a huge task to get proper physics up and running no matter which networking library you choose, but I'm not sure what game would require this to be honest.

Sadly, not really. The only texts available online which explains *some* of the lower level stuff are these articles by Gaffer: http://gafferongames.com/networking-for-game-programmers/ and Gambetta: http://www.gabrielgambetta.com/fpm1.html

I would advice you to stay with a prepackaged solution if this is your first foray into networking

 

Version 0.1.7.0 out, as always you can find it here: https://github.com/fholm/udpkit/releases/tag/v0170

 

Just wanted to inform that I got UDP Punchthrough working and implemented, so expect an update in a few days with this feature added.

 

Many thanks fholm!!

 

In my situation I'm developing an FPS but need authoritative servers for it. While uLink would be the go to, I'd rather learn to create a sickle and move on to create a plow than just pay someone to do it for me. I have to take the tough road to Networking. Thank you for making it open source fholm

 

Well, I can't argue with this method - that's how I did it I only started using Photon/uLink/etc. when I already knew all of the basics and core concepts of networking. I don't like being in the dark

No problem, as I myself have built a multiplayer FPS and a few other fast paced games, if you have any questions about anything (udpkit or not) don't hesitate to hit me up and ask away

 

Will do thanks! Future of FPS on Unity right here folks

 

@fholm: Since you've implemented reliable. Could you please help me to show how to use reliable send/receive message in you example chat.

Many thank...

 

I'll whip up a reliable + ordered example, the current chat is already reliable as in that it re-sends a message until it's delivered (but they are not guaranteed to be ordered).

 

Thank for your quick answer. udpkit very helpful for me to start with networking programming.

 

Just a minor update here, 0.1.7.1: https://github.com/fholm/udpkit/releases/tag/v0171

 

Just started playing around with UdpKit. Thank you, so far it's working great. My game is currently using Unity built-in networking, right now I'm not planning to change that but perhaps after I get a grasp on UdpKit I will consider changing the game logic to use UdpKit. I was however wondering if using UdpKit as a lobby/matchmaking server would be ideal? I was planning on using a simple php mysql implementation but I want to make sure whichever solution I use is scalable and robust enough to handle a potential large amount of players. I started to look at Node.js but after looking at numerous examples I couldn't get the client and server communicating reliably. I might revisit that again.

Edit: Just wondering if anyone knows how one might keep a server built with UdpKit running on linux. Can't seem to get mono to keep running the program in the background.

 

This looks really awesome, and I kind of feel like transitioning the project I'm working on immediately. Have you already checked in the UDP punch-through stuff, fholm? My primary gripe with the Unity networking is the serialization/deserialization, and that wretched BitStream object (why can't you read bytes from it?). The offer of backgrounding some net workload just sweetens the pot.

For prototyping purposes, what do you think of using the unity API for matchmaking, and possibly clock synchronization? I don't know if I want to re-implement those on top of raw udp just to play around.

e: to contribute to the discussion on this page of texts on multiplayer game development, IMO the best one anywhere online is this paper by Valve developer Yahn Bernier: http://web.cs.wpi.edu/~claypool/courses/4513-B03/papers/games/bernier.pdf

 

Trying to use UDPKit with Unity Networking for LAN Multiplayer, I want to udp broadcast the network server ip and port to clients which then connect to this ip using Network.Connect... I am looking for a solution that works even if you are behind a router without port forwarding.
Is this possible with udpkit and if so how can i achieve this? Any help would be much appreciated, My current code is:

public class UDPLan : MonoBehaviour {

UdpSocket socket;

[HideInInspector]
internal bool isServer;

[HideInInspector]
internal string serverAddress = "127.0.0.1:14000";

void Awake () {
UdpLog.SetWriter((s, lvl) => Debug.Log(s));
}

void Start () {
Network.isMessageQueueRunning = true;

socket = UdpKitUnityUtils.CreatePlatformSpecificSocket<demoSerializer>();

if (isServer) {
//Start Server

socket.Start(UdpEndPoint.Parse(serverAddress));


} else {
socket.Start(UdpEndPoint.Any);
socket.Connect(UdpEndPoint.Parse(serverAddress));
}
}

void StartServer(){
//Attach self to persistent group to survive the level load
this.gameObject.transform.parent = persistentParent.transform;

//Set this as server and start it again to send the ip and port to clients
isServer = true;
Start ();

//Load multiplayer level
Application.LoadLevel(3);

//Unatach self as made it through to the next level
this.gameObject.transform.parent = null;




}

void OnLevelWasLoaded(){
MGM = GameObject.FindWithTag("GameManager");
if (isServer){
Network.InitializeServer (32,14000,true);
Network.isMessageQueueRunning = true;
}
else
{

}

}

void Update () {
UdpEvent ev;

while (socket.Poll(out ev)) {
switch (ev.EventType) {
case UdpEventType.Connected:
UdpLog.User("Client connect from {0}", ev.Connection.RemoteEndPoint);
break;
}
}
}
}

 

Hey guys, sorry for the lack of replies - been busy with some other things for a week or so, getting back on UdpKit now I'll answer the three previous posts here:

I mean in theory you could, but most likely you would be better of with something based on TCP/IP, which is what most matchmaking services use.

I would recommend you to write the matchmaking service in the same language as the rest of your game (C# that is, I assume) so that you can re-use as much code as possible.

In the Main() method just do a while(true) { } loop which waits for event-event in udpkit.

I have not, I've been fiddling with it trying it out on several different router/NAT combos to make sure it works properly on as many configs as possible. It will get checked in over the next week or two. Sorry for the delay, I'm a bit strapped on time right now.

Just a note is that if you write a Char with unitys bitstream it will only write the first byte from it (its noted in the manual somewhere).

Matchmaking and lobbying I prefer to do over TCP/IP with some separate type of service. Clock synchronization is not that hard, it depends a bit on what you need, but basically to synchronize a clients clock to a server clock you would do something like this:

1. In the first 4 bytes of every package going from the server to the client you write the servers local time when the packet was sent
2. On the client, you read out the time and then update your local clock like this:

Code (csharp):

1.  
2. public class TimeSynchronizer : MonoBehaviour {
3.   static float offset = 0f;
4.   static float minServerTime = -1f;
5.  
6.   public static float GameTime { get; private set; }
7.   public static float LocalTime { get; private set; }
8.  
9.   public static void UpdateOffset (float serverTime) {
10.     if (serverTime >= minServerTime) {
11.       minServerTime = serverTime;
12.  
13.       float newOffset = serverTime - Time.time;
14.  
15.       if (offset == 0.0f) {
16.         offset = newOffset;
17.       } else {
18.         offset = (offset * 0.95f) + (newOffset * 0.05f);
19.       }
20.     }
21.   }
22.  
23.   public void Update () {
24.     LocalTime = Time.time;
25.     GameTime = LocalTime + offset;
26.   }
27. }

So what you are looking for is really two separate things:

1) Local LAN game discovery, this is done using udp broadcast. I would recommend you to just use plain .net/udp sockets for this as it's easier.

2) NAT Punchthrough, which is for getting past routers so two people without any open ports can connect to each other. UdpKit will be abel to do this in a few days/a week or so. The code is done it's just not tested enough yet.

 

Hi, Thanks for the speedy reply and information. I have looked into doing a LAN game discovery system using the in built unity UDP broadcast system but it seems to use the system net sockets class which is blocked for Android/iOS basic. I can't find a way around it. What would you suggest?

 

There is no direct support in UdpKit atm for doing broadcasting as I have not implemented that functionality in the native libraries for ios/android. I would recommend you to look at how I am using my native plugins in UdpKit and then look into extending them to allow for lan broadcast.

 

0.1.7.2 is out, you can find it here: https://github.com/fholm/udpkit/releases/tag/v0172
it contains a fix for a rather serious bug which would cause some send objects to be reported as both lost and delivered during situations of extreme packet loss for prolonged periods of time, > 50% .

I have been doing some *serious* testing of UdpKit and my Bolt engine built on top of it, and I can gladly report that this is the only major bug I have found in UdpKit during even the absolutely worst conditions (packet loss of 75%+ with 128 clients connected, it still manages to deliver everything properly)

 

amazing work fholm, thank you!

 

Seriously. Thanks a heap, Fredrik.

 

Thanks for UdpKit and the reply. I will definitely be making use of it on my next project.

 

Another big thanks here. I am currently using Ulink but like the control a thinner layer will give me so will start migrating my current project to this over the next few weeks.

 

Thanks to all of your for the kind words

 

On the latest version, in the unity sample, it sets the writer callback in the Awake. I think the values are round the wrong way:

It is currently
UdpLog.SetWriter((s, lvl) => Debug.Log(s));

but I think it should be

UdpLog.SetWriter((lvl, s) => Debug.Log(s));

So the string is passed to the Logger, not the level.

I could, of course, be completely misunderstanding the purpose of that callback

 

msmsm: You'd be correct, I pushed a fix for this to the repo just now. Thank you In the future if you have time and energy I appreciate the bug reports on GitHub if possible since it lets me keep track of them.

 

Yes of course. Sorry for not thinking about that!

I am liking it a lot so far. Still just creating helper methods on top of some test code, and a long way to go for me, but it finally feels like I have some control rather than just going along for the ride!

 

Thanks for the udpkit.
It seems like if you do socket.Close(), socket.Start(...) does nothing.
In the code for the pastebin, I create a server, disconnect, then create another server
http://pastebin.com/x7KBWBKg

 

You need to create a new socket after calling socket.Close(), you can't "restart" it.

 

Version 0.1.7.3 is out, still no NAT punch-through. I'm not happy with how the arbiter is working and it's to convoluted to use. Gotta rework the API/hosting solution for it.

However, the changes in this update is as follows:

* UdpSocket now exposes a property called StreamPool which is allows you to allocate stream objects on the user thread. This object is unique per socket and is not shared.
* UdpStream now implements the IDisposable property, call Dispose to return the stream to its pool.
* There is now a default serializer called UdpStreamSerializer which can be used to send UdpStream objects you have written data in on the user end. This makes UdpKit a lot easier to use for people who don't want to or need to deal with threading.

Here's a quick example on how to use the new features:

Code (csharp):

1.  
2.  
3.  
4.       // create a socket using the stream serializer
5.       socket = UdpSocket.Create<UdpPlatformManaged, UdpStreamSerializer>();
6.       socket.Start(new UdpEndPoint(UdpIPv4Address.Localhost, 14000));
7.  
8.       // UdpKit poll loop to wait for a connection
9.       UdpEvent ev;
10.  
11.       while (socket.Poll(out ev)) {
12.         switch (ev.EventType) {
13.           case UdpEventType.Connected:
14.             // grab a new stream
15.             UdpStream stream = socket.StreamPool.Acquire();
16.  
17.             // write a string into it
18.             stream.WriteString("Hey");
19.  
20.             // send this stream to the other side
21.             ev.Connection.Send(stream);
22.             break;
23.  
24.           case UdpEventType.ObjectSent:
25.             // once the stream has been sent, dispose it (returns it to the pool)
26.             // this is NOT done automatically for you by the UdpStreamSerializer
27.             ((UdpStream)ev.Object).Dispose();
28.             break;
29.         }
30.       }
31.  
32.  
33.       // On the other end, it would look like this:
34.       UdpEvent ev;
35.  
36.       while (socket.Poll(out ev)) {
37.         switch (ev.EventType) {
38.           case UdpEventType.ObjectReceived:
39.             using (UdpStream stream = (UdpStream)ev.Object) {
40.               Debug.Log(stream.ReadString()); // prints "Hey"
41.             }
42.             break;
43.         }
44.       }
45.  

 

Hey,
Thanks for another handy update, kind of related to something I was just looking at.

I was just wondering about MTU length. You mentioned earlier that

I wish to package up a number of waiting messages I have and send them at the end of the frame to increase bandwidth efficiency. I have the structures in place for doing this now and was wondering if I can determine the size of a Udpstream that will be allowed to be sent to a particular connection so I can split my messages accordingly prior to sending them? Or is it a case of working out a reasonable size and sticking to it?

 

Hey got a few questions about UDPkit

1. Does UDKkit come with some sort of generic server* that I can use?
2. How many players can UDPkit support?
3. Will UDPkit be updated for Unity5's 64 Bit / Multi threading?

*I would prefer a server that is redistributeable.

 

An interesting question, in 99% of all cases I would advice you to set the MTU default/min/max size to the same, something like 1kb. Now you know how large each packet will be, so if you have a queue of some message type you want to send and pack as much as possible into each packet, how would you do this?

Something like this is what I do in my Bolt engine:

Code (csharp):

1.  
2.  
3.     // some queue that exists somewhere of messages you want to send
4.     System.Collections.Generic.Queue<MyMessage> queue = new System.Collections.Generic.Queue<MyMessage>();
5.  
6.     // stream object we are writing into
7.     UdpKit.UdpStream stream = socket.StreamPool.Acquire();
8.  
9.     while (queue.Count > 0) {
10.  
11.       // store position in stream (how much we have written so far)
12.       var position = stream.Position;
13.  
14.       // peek the next message
15.       var message = queue.Peek();
16.  
17.       // write the message into the stream
18.       message.Write(stream);
19.  
20.       // if .Overflowing is true, means we wrote more then the stream can hold
21.       if (stream.Overflowing) {
22.  
23.         // reset position
24.         stream.Position = position;
25.  
26.         // send the stream with as much data as we have written (except the last message which failed)
27.         connection.Send(stream);
28.  
29.         // allocate a new stream
30.         stream = socket.StreamPool.Acquire();
31.  
32.       } else {
33.  
34.         // we wrote message successfully, remove it from queue
35.         queue.Dequeue();
36.       }
37.     }
38.  

 

UdpKit is just a low level udp socket library, writing generic server and clients, etc. is something you can do ontop yourself

Depends bandwidth per client, but since you can multiplex several UdpKit sockets I would not say there is a practical limit on amount of connections.

Yes.

 

Once again thanks for the quick, and incredibly helpful, reply!

 

First of all, thanks a lot, Fredrik!

I have been experimenting with Unity networking, and while it is nice to get things going quickly, I find it disturbingly "automagical" and surprisingly unintuitive for someone used to more traditional game networking libraries. I just want to make connections, and send arrays of bytes, is that so hard?

This udpkit library of yours is awesome. I downloaded it and got it working with not too much effort.*

The code looks clean and well-written, and I take a great deal of comfort in being able to see the source instead of having it hermetically sealed away. Unity's habit of locking things down has bitten me in the past with, for example, PhysX.

Keep up the great work. I'm very interested in the NAT punchthrough feature, as that is a pain to implement.

When you get something up on the asset store that you are selling, let me know by PM and I will happily buy one to support you, even if I don't need it. I'm making a turn-based strategy game, so I doubt I'll need much more than what you have provided in UdpKit, but I'm grateful for your good work.

Cheers!


* Side note: I was not able to get the latest version of your library building under Mono Develop on Mac OS X. I got strange errors about mis-matched ToolsVersions, which I tried to fix to no avail by manually editing the .csproj files. Eventually, I gave up on the assembly and just copied the source code into my project, which was very easy to get up and working.

 

No problem at all

I understand why unity designed their library the way they did, as it becomes easy for novices to use.

I'm inclined to agree

Yes, I agree with this fully, the same reason my upcoming network engine (which is built on top of UdpKit) called 'Bolt' will also ship with full source code.

Yeah NAT is a pain to deal with, the current implementation is working but it's very hard to use so I need to re-factor the API, etc.

I will make sure to give you a nudge once Bolt is live, but there's no need to buy anything

There should be an udpkit_mono.sln file which will open in MonoDevelop/XamarinStudio on OSX, but I may have forgotten to update some of the .csproj files it to include all the correct .cs files.

 

Yeah, I tried both the udpkit.sln and the udpkit_mono.sln but for some reason neither worked (similar errors in both).

I'm not a C# expert (my games industry experience was with C++), and I have more experience with VS than Monodevelop even though I am using Mono on my mac at home, so I wasn't really sure how to proceed.

Anyway, just using the source works fine, though.

Thanks for the reply!

 

I will double check why its not working properly and submit a fix for it to the repo asap.

 

Is there a way, when reading the stream, to determine if I am at the end of it or not? I have pushed several message onto the stream and want to read them off at the other end. I haven't recorded the number I have put on; I could do this if required.

Currently I am just reading until I get an overflow and then stopping. This seems to work fine and I just wanted to check that this the accepted way to do this?

Thanks,

Matt

 

There's two main ways to do this, you could record the count of the messages you write - but this requires you to skip 2 bytes (or something like that), write messages, then go back to the start, write the 2 bytes, and then skip to the end. Kind of messy. You could also do the same thing but use the size (in bits) instead of the message count.

The other way to do it, is that you write a bool(true) (one bit) before each message, and when you are done you write a bool(false), something like this on the writing end (expanding on my previous example, check the lines marked with "NEW"):

Code (csharp):

1. // some queue that exists somewhere of messages you want to send
2.     System.Collections.Generic.Queue<MyMessage> queue = new System.Collections.Generic.Queue<MyMessage>();
3.  
4.     // stream object we are writing into
5.     UdpKit.UdpStream stream = socket.StreamPool.Acquire();
6.  
7.     while (queue.Count > 0) {
8.  
9.       // store position in stream (how much we have written so far)
10.       var position = stream.Position;
11.  
12.       // peek the next message
13.       var message = queue.Peek();
14.  
15.       // NEW - write a bool true here to say "yes we have another message to read"
16.       stream.WriteBool(true);
17.  
18.       // write the message into the stream
19.       message.Write(stream);
20.  
21.       // if .Overflowing is true, means we wrote more then the stream can hold
22.       if (stream.Overflowing) {
23.         // reset position
24.         stream.Position = position;
25.  
26.         // send the stream with as much data as we have written (except the last message which failed)
27.         connection.Send(stream);
28.  
29.         // allocate a new stream
30.         stream = socket.StreamPool.Acquire();
31.  
32.       } else {
33.         // we wrote message successfully, remove it from queue
34.         queue.Dequeue();
35.       }
36.     }
37.  
38.     // NEW: Check if we can write any more (we need one more bit)
39.     // it is valid that the buffer is 100% full as we will check CanRead() on
40.     // the receiving end.
41.     if (stream.CanWrite()) {
42.       // Write what I call a "stop marker" (bool false) which says "no more messages to read"
43.       stream.WriteBool(false);
44.     }

And then on the reading end, something like this:

Code (csharp):

1.  
2.  
3.     // as long as we can read more
4.     while (stream.CanRead()) {
5.  
6.       // we reached the stop marker
7.       if (stream.ReadBool() == false) {
8.         break;
9.       }
10.  
11.       // .. read a message out
12.     }

I use this pattern all over the place in my Bolt engine.

 

What a neat idea, thanks. Good luck with the engine, sounds like it is shaping up really nicely.

I have the bare bones of a messaging system that can instantiate, destroy and route messages now, including buffering them for future clients, so once I have tested it in my test environment for a while I can start to rip out the uLink stuff in my game and replace it with this. I still have a long way to go to get it to a place I will be totally happy with, but I am having great fun getting there!

Thanks again!

 

v 0.1.7.4 is out: https://github.com/fholm/udpkit/releases/tag/v0174

contains a fix for a big in the new UdpStream pool functionality. Thanks to msmsm for submitting a bug report on github

 

I'm not sure I will ever need to do it but if I had a message that took a bitstream and wanted to pack this up into a larger stream of messages what is the best way of signalling the end of the stream within a stream? Is it possible to do without a short detailing how many bits in the stream before-hand?

I don't currently need it, but it would be interesting to know in case I do down the line!

EDIT: After sleeping on this it is obvious that a stream in a stream is just like, in principle, a string in a stream; that is the number of bytes in the stream must be recorded before the stream so we know how much to read. Note that this is considering a stream as a argument in a RPC call (i.e the contents of the stream are unknown to the serializer at the point of sending the message as it has already been constructed prior to the RPC call). Obviously if the stream is constructed from known variables you can just pack and unpack it using the variables.

 

Hey guys!

I just wanted to report success using the UdpKit in Unity for myself. I took Fredrik's example Unity project and merged it with his example console chat server, and got a serviceable chat server running in Unity in no time.

I love the library, it's easy to work with and reason about.

One extremely minor thing I noticed from reading the source code is that a couple of files misspell "incoming" as "incomming". I don't mention it to be a jerk, just to help out.

One question I have, Fredrik, is how you would suggest that I handle multiple kinds of objects being sent as messages.

The example I have is the turn-based strategy game I will be making. All in-game actions will be reduced to a single Command class, which is the only thing allowed to mutate the permanent state of the game (anything else is purely visual). However, in addition to that, players are likely to want to chat with each other. Therefore, I will be sending two things over the network, strings and Commands. The problem is, that by the time the data reaches the Serializer, it no longer knows what type was sent to it. I can think of at least two ways to handle this.

1. I could create two sets of sockets between the server and the client. One for the game, which uses a CommandSerializer, and one for the chat channel, which uses a StringSerializer.

2. I could package the string and/or Command and use it on the same socket. When sending a string or command, I first send a byte indicating the type, and then send the object. On the other side, I serialize it into the appropriate class based on the type byte.

1 has the advantage of being less "branchy", separating critical and non-critical messages, and consuming less bandwith (not a major worry for me).

2 has the advantage of being more easily extensible, and limiting the messiness of handling different types into a smaller part of the codebase.

Is there one of those two approaches you would recommend over the other? Or perhaps a third way I haven't thought of?

I am leaning towards the second choice, but for some reason I think a lot of games use the first 1 (I can remember opening multiple UDP ports for video games back in the day, and some were explicitly mentioned as being "chat" channels).

 

For the last week or so I have been writing a layer on top of this library to replace ulink. I had no problems with uLink, but financially it was not a viable option for me going forward as I doubt my game will ever sell!

I have only been using unity and c# since the start of this year so I didn't take this undertaking lightly, but the opportunity was too good to miss

Today I managed to finish ripping out all the uLink stuff and putting in my own networking. There is still a long, long way to go before I have everything working how I ideally want, but the basics are all now in and I have my player running around his world like nothing has changed!

So this is just to say thanks to fholm for providing such a neat little library, and to say to others thinking about doing this to give it a go. I have learnt a lot in the process, and feel now that I have control over the networking code at last. Given the way this library has been coded, and seeing some of the other assets fholm has produced, I really think his BOLT network engine is going to be something special!

 

Sorry to ask this again, but how about ordered example?

Thank for your support any help would be appreciated

 

Here are my thoughts:

For a reliable ordered system ensure that you keep sending the message until you know it has arrived, just like the chat example:

case UdpEventType.ObjectLost:
ev.Connection.Send(ev.Object);
break;

At the other end, use whatever gives your messages order, such as tick or time, to ensure that you do not send a message off for processing until you have all the messages preceding it. The hard work has been done by fhlom in implementing the ObjectLost even on the sender side, but if you want more details about how this is achieved then http://gafferongames.com/networking-for-game-programmers/reliability-and-flow-control/ is a godd place to start.

If you want ordered, unreliable, it is even easier; just throw away any packet with a timestamp/tick less than the last one you received and processed.

EDIT:

looking through the source it appears fholm has implemented a UdpReliableBuffer, which implements a send and receive queue. Take a look at the source code and have a go with it.

 

Hey guys! I know I'm a bit slow with answering questions in this thread ATM, trying to juggle both UdpKit and my own Bolt engine at the same time, plus working a full time job and taking care of my family

Here's a new 0.1.7.5 version of UdpKit, contains a few bugfixes and enhancements. There is one breaking API change in it, the DefaultMtu and MtuMin and MtuMax settings have been clumped into one config called "PacketSize" now.

https://github.com/fholm/udpkit/releases/tag/v0175