[Open Source] UdpKit, a networking library for .NET/Mono/Unity

this is a really useful utility fholm, nice job!
Although if you could make a brief user manual/documentation, it would be really helpful.

 

A user manual/guide is in the works. Including a free asset store package, with everything setup and ready to go inside of unity.

I pushed a bunch of updates, nothing major functionality wise just implemented some standard interfaces, cleaned up code, added comments, etc. But ended up being ~15 commits, bumped the version to 0.1.6.0.

Complementary tag-link: https://github.com/fholm/udpkit/tree/v0160

 

fholm, thanks for your contribution! I recently read up on lidgren, and am curious how you would stack (no pun intended) your library against it. I'm not as concerned about android or other platforms and its part in them, I'm just wondering what features you may have extra (or less).
Can you expand on this?
Thanks!
JC

 

jc_lvngstn: Of course, obviously any comparison written by me will be baiased towards UdpKit, but I have used Lidgren for a long time before writing UdpKit (see my SlimNet networking server, which uses Lidgren) so I know it very well and I'll try to be fair in my comparison.

At their core, both UK (UdpKit) and L (Lidgren) solve the exact same issue: To give you an abstraction of connections and a more controlled delivery mechanism ontop of simple UDP sockets. But they solve this in very different ways. The main features UK has, which are not in L are the following:

• Notification of message delivery/loss, UK will tell the user if a message was lost or delivered on its way to the
  other end. This is a very important feature as UK does not implement default methods for reliable communication, as I have always found that the way reliable communication is needed ends up being very game specific, but instead allows you to easily build this to suit your game on top of its delivery/loss notifaction mechanism.
• Packing/Unpacking of data happens (optionally) on a separate thread, saving precious time from the main thread if you have a lot of players (for example with 64 players, just serializing a full 1kb packet to them if done on the main thread takes 8-10ms, easily).
• The ack/nack algorithm of UK is a lot faster and uses a lot less bandwidth then Lidgren. This is actually one of the major reasons I wrote UK, the ack/nack algo in Lidgren has very bad behaviour when you are under a lot of packet loss or ping spikes (which is very common on mobile networks).

There are currently a few features missing from UK that can be found in L:

• UDP-punchthrough/NAT-traversal, I am working on this and it should be done soon (tm), but if you need this *now* you should use Lidgren
• Encryption, I *might* implement this in UK in the future but there are no concrete plans currently.
• Path MTU discovery, I am not sure this is needed or even wanted nowadays, as it has the possiblity to cause issues early on in the connection.

Like I have said before, UK and L solve the same problem in reality. They are very similar, L focuses more on giving the user a super easy experience and comprimises a few things internally (imho). UK gives you a lot more control, you get to decide exactly when one packet is sent, etc. but it also lets you screw things up more if you dont know what you are doing.

 

Awesome, I'm glad I looked in your profile after you've answered my question and found your framework.

Definitely going to try it out as soon as I can.

You don't plan on implementing reliable UDP? From what I understand it's pretty common practice to send reliable UDP delta changes for state data, RPCs, etc - imho it seems that this feature is too important to be left out.

Btw, I'm personally very interested to see the actual game you're working on.

 

Well, you can already send data reliably using UdpKit, but to do so you need to sequence and buffer the elements by hand. Due to the way UdpKit works, in that it allows you to say exactly when a packet is to be sent, and how big it's going to be, etc. Implementing a generic "reliable" mode by default in UdpKit just doesn't work out. I am however going to include some utility classes for implementing reliable messages yourself, which fit perfectly with your game. This is just a trade-off I have chosen, in my experience reliable communication is pretty rare in games, at least in fast paced action games (which is all you should use UDP for). If you are building something slower like an RTS or turn based game, just use TCP instead.

Well, not really. Reliable data is rare in actual real world action/real-time games. Like I said if you are building a slower game (RTS, Turn Based, etc.) then just use TCP. If you are building a fast paced/real-time game, use UDP. In these type of fast paced games reliable data is very rare.

Whenever it's done, and it's not really a game. It's the FPS Networking Kit which will be available for purchase as some future date

 

Nice,
it is always nice to see other networking alternatives.

got a few questions regarding UdpKit:

1- does the server runs inside unity instance like uLink? or is it a standalone server app?

2- if it runs in a unity instance, which it means it only supports as much ccu as a unity networking due of its single threaded cap.
so what makes UdpKit different from unity networking? less bandwith? more ccus?

3- can players jump from a unity UdpKit server instance to another like uLink?

4- how can it escalate for massive ccu? or is it only single server hold as many ccu as your single threaded server can hold?, is it a multiroom based? or peer to peer?

5- well, i guess by now you can supposed what is my main overall question or concern:
can UdpKit be used for a mmo game? server escalation, handling massive number or concurrent users, server load balancing, Area of Interest etc....


thanks,
just trying to see what benefits over unity networking and or uLink.

 

UdpKit is a (pretty) low-level library that you can use to build servers (and clients, proxies, etc...) with.

But I have a feeling that some much higher level tooling based on it will soon begin to manifest...

 

I see,
so i guess this means that for those of us who are not pros in low level networking may we better off with even unity networking than try to mess with UdpKit for now?

well i am not saying i am a noob in multiplayer game creation but into the actual networking core engineering which is a different song.

 

Yes, but hang tight. UdpKit is really the most promising thing to emerge in this space.

As fholm has mentioned, he is planning to roll out a high-level FPS framework (soon?).

And I've begun designing a (more general) framework and PaaS on top of UdpKit and CoreOS / Docker.
( A 'Heroku for game servers' kind of thing. )

I would not be surprised to see others pop up.

 

Even though Lucian_Games pretty much answered your main question, I will try to answer each point specifically.

UdpKit is (like Lidgren, RakNet, ENet, etc.) simply a thin abstraction on top of UDP which provides things like connections, ping, delivery notification, etc.

UdpKit is not tied to any type of server, single- or multi-threaded, async or sync, etc. You can use it in any way you want.

This would be up to you as the user of UdpKit to implement, if you want it.

This is pretty much answered in the three previous questions, but I want to mention a specific thing about supporting a lot of connections. UdpKit is like I've said a very simple socket layer, but it does a few things "extra" to allow it to handle a large/huge amount of connections:

• It can run the serialization (bitpacking) code in a separate thread, saving a lot of precious simulation thread performance.
• It lets you open serveral sockets, and them combine them into "one" socket using the UdpSocketMultiplexer, each socket runs in it's own thread, giving you very good scalability while still only having "one" socket to deal with from your own code

It's on the way, the first high level piece is starting to finish up. It's called Bolt and is a high level networking solution, which is specific for Unity and specific to fast paced action games (fps, tps, etc.).

Now you are just flattering me

If you need any specific help, support or assistance with UdpKit in this project you are free to contact me over mail (see my signature), and I will help out as you need and I have time (early adopters always get treated with silk gloves ;p)

 

Will do. Thanks, Fredrik.

That sounds luxurious.

 

I've been using UDPkit for the past few weeks to create a framework to replace uLink in my project. Already the performance is much improved.

I'm saving 30% in bandwidth over uLink with identical test scene (over 60% when using halfs for position sync) and have no garbage generation when syncing position (in my implementation, sending RPCs generates a tiny bit when boxing valuetypes for the params object array). And of course, simulation thread performance is practically unaffected by number of connected clients.

I'm also able to rid the use of string identifiers for RPCs (server auth generates a numeric ID for the method name attained through reflection) and replace everything with a 4-byte uint header containing view ID, RPC id, and 2 option flags. Already have a attribute based RPCs, NetViews/network instantiation, buffered messages, network groups, variable update rate, sync state message batching, scope, and some other goodies up and running.

It's a nice change from uLink and Lidgren, for sure! The no-frills, straightforward implementation of the kit makes it very easy to reason about the functionality and create your own higher level functionality.

I'm focusing more on the "MMO" side of things, if curious.

 

Jakevn,

you pretty much have read my mind, i am currently using uLink and would like an alternative more optimized option.
i like the rpc streams replacement for 4-byte units.

basically, i like uLink due of the server side collision, player handover from server to server and authoritative server deal out of the box,
so if you are cooking something else that can be used instead ill be the first one to try it out .

btw i can help you testing if you like, offcourse that is if you going to release your framework.

eaitherway pretty much what i am looking for.
dont get me wrong, i still like uLink over all the known networking solutions,

but i would like an alternative more felxible, in other words not a blackbox where one have to beg for updates and support, seriously, uLink staff really leave a lot to desire about their product for the lack of support.

 

I'm considering putting it on the asset store if there is considerable desire for an MMO solution built atop UDPKit. Ideally, it would be inexpensive ($50~). The source is well commented and I've purposefully been building it in a friendly way to facilitate this.

That being said, that is still at least a month out. I need to finish porting my "Lobby" code over as it facilitates auth, persistence, server config, and peering.

 

I completely forgot, I pushed an update to the master branch, it contains some utility classes for implementing reliable communication in UdpKit. I am also working on the Asset Store package, which I will submit over the weekend most likely. More specifically, this commit: https://github.com/fholm/udpkit/commit/9040f65f40efb6545616c2f1577141a105fc5804 it's what will be tagged as 0.1.6.1 later.

The asset store package will contain a "full" setup of udpkit inside Unity, which provides a simple implementation of reliable rpcs and unreliable state sync. And yes it will be free.

 

Oh wow, my apologies for not looking deeper into the license. I could of swore I saw somewhere, where you said "I don't think people should make money off of something I worked on" or something along those lines, maybe it was for SlimNet?

Anyways, that's awesome because I actually did want to start a project with this, so now I'm pretty happy I can do that!

 

Awesome,
Any early adopter deal?
pm me if you do.

Also will it support player handovers like ulink?
where a player can port to another server instance for a zoned mmo?

 

Yes, that would be the SlimNet license which is very restrictive, basically: "you can read the source code and learn from it, but you cant use it for anything". I spent *a lot* more time on SlimNet then I have on UdpKit, so I am not willing to let it go for free UdpKit however is and always will be 100% Open Source and Free.

Well that is great, if you run into any trouble or need advice on how to set up things dont hesitate to poke me. You can usually find me in #unity3d and #unity3d-chat on freenode irc. Or in here of course.

 

I'm not focusing on distribution or monetization at the moment, so I don't have much to say about this. I'll post something when I have time for working on the details.

If you're familiar with Pikko server, that is the ultimate goal. There are technical differences and my focus isn't on "roaming" server responsibilities.

I am not happy with the "zone" approach, as it's quite outdated. The peering model I'm working on allows a server to treat another server (peer) as a client with authoritative permissions. This allows clients to see network objects being simulated/controlled by other servers. The lobby provides the peering endpoints and allocates NetView ID ranges to prevent ID collision.

I haven't decided how I want to handle server<->client connection handoffs when client is outside of current server's jurisdiction for a significant time frame. I'm currently working on a proof-of-concept as I feel this is one of the more important parts of a modern MMO networking solution.

 

Sounds great. I'm using uLink, but have gotten closer and closer to the metal over time and now only really use it as very basic transport manager. uLink is far less efficient with bandwidth than I would be with lower level access, absolutely crushes the GC compared to my zero allocation code, and also uses the horrible and horribly misnamed Unity "callback" scheme which is just... yeah. Also, like Unity, it's nearly impossible to get actual information / documentation about how things actually work beyond a "for artists looking for a make game button" level. TBH it seems like their core market are devs with low technical requirements / programming skill who live in the casual "free to play" cash grab MMOs space... which is admittedly a pretty good market, but I need more efficiency and control, not more hand-holding and fluff.

I was thinking about switching to Lidgren at some point, but I may have to give this a shot.

 

Tell me about it. I told them over a year ago that their key structs didn't implement IComparable<T> and thus were causing needless allocations on every comparison, and it they finally fixed it in 1.5.7, released earlier this week.

When you're waiting over a year for something you could fix yourself in two minutes... well, that's a situation you don't want to be in.

 

Published a new version, 0.1.6.3, including pre-compiled binaries: https://github.com/fholm/udpkit/releases/tag/v0163

 

I've been trying to publish a new binary release to github for the past two days, it doesn't seem to be working (for some reason unknown to me). The newest version is 0.1.6.4 and contains .sln and .csproj files for building using Mono Develop/Xamarin Studio. The tag for github is here: https://github.com/fholm/udpkit/tree/v0164

And those of you who had problems building on OS X before (or if you were building using Unitys Mono Develop on windows), should be able to open the udpkit_mono.sln file in either MD or XS and it should build without a hitch.

An asset store package is on the way, it's taking a bit of time because I want to provide a very thorough example of how to setup UdpKit inside of Unity.

 

I am sorry that I am very stupid and can't seem to get this to work. can you please provide me a step by step to get this to work? i redownloaded it because I am interested in this but I can't seem to get this to work. my guess is for each folder you need to copy certain files into it to make it work. and that it was set up like that to save file size. But what files need to be copied?

 

Drop 'udpkit.dll' and 'udpkit.platform.managed.dll' into your project (I just use a folder named 'udpkit').

That's all you need to get going in the editor and standalone desktop builds. (For webplayer, you'll also need a policy server.)

For Android and iOS, you'll need 'libudpkit_android.so' and 'libudpkit_ios.a', respectively, in a folder named 'Plugins'.

 

Where do I find these .dll you speak of? i tried to build the sln with monodevolop like the read me states but i keep getting errors

 

https://github.com/fholm/udpkit/releases

 

I am using 0.1.6.4 should i use .3?

 

The DLLs can either be built using the udpkit.sln file for visual studio, or the udpkit_mono.sln file for mono develop/xamarin studio. If you dont want to build them most releases can be found in binary format on github under "releases".

But since i feel nice, I attached the latest v.0.1.6.4 dlls here: View attachment $v0164.zip

 

well thank you very much nice sir. i have been very eager to use this and I am subscribed to this forum so I can get all updates.

 

No problem.

There's been some issues with the releases pages on github recently, not letting me upload binary files (which is why 0.1.6.4 is missing in binary format). Im trying to get a page setup for udpkit outside of github that i have more control over where i can publish my releases.

 

Published version 0.1.6.5 to github, including binaries, it can be found here: https://github.com/fholm/udpkit/releases/tag/v0165

Only one change in this version, there is a new instance property on the UdpSocket object called AvailableEvent which is an AutoResetEvent which another thread can wait on instead of constantly polling UdpSocket.Poll for new events.

This event can be disabled with the new UdpConfig.UsePendingEventEvent setting.

 

I haven't had a chance to play with this yet, but just wanted to say it sounds great. Should cut down on CPU where possible.

Thanks, fholm!

 

This feature is mostly for cases where you are running a secondary thread to the main thread which deals with your games networking specific code. Just calling UdpSocket.Poll once every frame in Unity is no big problem, and you can't really wait for the event as that will stall the entire main thread. Basically this is a feature you need if you are building multi-threaded room-based servers for example, where maybe each room runs in a dedicated thread, etc.

 

Published version 0.1.6.6, including binaries, can as usual be found here: https://github.com/fholm/udpkit/releases
Changes in the version:

• Added UdpSocket.Peek method to allow for peeking of the next event without removing it
• Removed a race condition that could happen when calling UdpSocket.CancelConnect which would cause it to silently do nothing, now it will properly disconnect the connection you are cancelling even if it's been connected.
• Removed the bug where the socket created/socket started messages would be spammed in the log when an exception was thrown

 

Right on.

It mitigates some of the concern that motivated me to inquire about using epoll / kqueue / IOCP, where you might have frontend servers that just handle IO for a bunch of different backend services. The busy wait in that case was kind of bothersome.

Which reminds me: is it necessary to use non-blocking mode when you're handling the IO in a background thread anyway?

 

Ah, yes of course! Sorry I just assume everyone talks in the context of "unity" on the forums here

Yes, if you don't use non-blocking mode the timeout parameter doesn't function properly on all platforms in my experience.

 

Haha. No worries.

Okay. I'll take your word for it. Anyway, efficiency is probably much better now that the main thread has something to block on. I'm looking forward to trying it out.

 

I pushed another update to the UdpSocketMultiplexer class which exposes a Wait method on it which uses the underlying AvailableEvent properties of all the sockets to wait for an event on any socket, three overloads are available.

Commit: https://github.com/fholm/udpkit/commit/9b592e07056242fd5d26fe826edd39683a5ad86f

 

Very nice. Thanks, man.

 

i want to see UdpSocketMultiplexer example. Please.

 

I added an example of the multiplexer in action, it's basically a version of the simple example but it uses the multiplexer instead: https://github.com/fholm/udpkit/tree/master/src/managed/udpkit.example.multiplexer

You will see when you connect to the server with a client, that it will pick one socket on random and connect to that (you can see it on the port number the client connects to).

 

Wow, Thank u very much.

 

Heres the 0.1.6.7 release also, in binary/source: https://github.com/fholm/udpkit/releases/tag/v0167

 

Thank again for good support, This is my game [https://play.google.com/store/apps/details?id=com.arcadeplus.ninekeonline]. Now i use superwebsocket and try to change to other network engine

 

Published version 0.1.6.8, can per usual be found here: https://github.com/fholm/udpkit/releases/tag/v0168

Only one change in this, but a pretty major one. You can now supply your own noise function for use with network simulation by setting the UdpConfig.NoiseFunction delegate. Before UdpKit always used System.Random() for doing ping-jitter-simulation and loss-simulation, the problem being that System.Random gives way to even distribution leading to very predictable and stable jitter and loss periods, which is the exact opposite of what you'd want.

Now if you supply for example a 1D perlin noise function here, you will get "jitter" spikes and "loss" spikes instead of an even % loss spread nicely. For example my upcoming Bolt networking solution uses it like this:

Code (csharp):

1.  
2. #if DEBUG
3.     // base X axis of noise on realtime
4.     float noiseBase = Time.realtimeSinceStartup;
5.     Stopwatch noiseClock = Stopwatch.StartNew();
6. #endif
7.  
8.     // setup configuration
9.     UdpConfig udpConfig = new UdpConfig {
10.       ConnectionLimit = mode == BoltNetworkModes.Server ? config.serverConnectionLimit : 1,
11.       ConnectRequestAttempts = 5,
12.       ConnectRequestTimeout = 1000,
13.       AllowImplicitAccept = mode == BoltNetworkModes.Client,
14.       AllowIncommingConnections = mode == BoltNetworkModes.Server,
15.       AutoAcceptIncommingConnections = mode == BoltNetworkModes.Server,
16.       PingTimeout = (uint) (localSendRate * 1.5f * frameDeltaTime * 1000f),
17.       MtuMin = 1024,
18.       MtuMax = 1024,
19.       DefaultMtu = 1024,
20.       SimulatedLoss = config.simulatedLoss,
21.       SimulatedPingMin = config.simulatedPingMin,
22.       SimulatedPingMax = config.simulatedPingMax,
23.  
24. #if DEBUG
25.       NoiseFunction = () => Mathf.PerlinNoise(noiseBase, ((float) noiseClock.ElapsedMilliseconds) / 1000f)
26. #endif
27.     };
28.  

 

Hi fholm,
I currently have a turn based game that runs on a local network. It all works fine but I currently need to have the server at a fixed IP address. I really want the game to operate a system of server discovery so the first player to start a game acts as server. But all the threads relating to server discovery seem to be very out of date. I have two questions:
1) Would I be able to do server discovery using your UdpKit?
2) Have you got a link to 'crib sheet' for networking noobs like me on how to install and use your system - where do I put the dlls and how do I use them?
Thank you!

 

The best way to do local LAN discovery is with UDP broadcasting, this can be implemented outside of UdpKit easily using .net sockets, I have been thinking about implementing it by default in UdpKit but i have not yet.

No, not really, sorry. There is a free package on the way for the asset store, but I am not sure I would recommend UdpKit for a networking beginners, it gives you a lot of power and flexibility but if you dont know how to use it you will just shoot yourself in the foot.

 

Many thanks for the quick response - I'm sure you right about not overcomplicating things.

My main problem is that, unlike most aspects of Unity, there seems to be a real lack of clear information in this area. I haven't been able to find any simple examples of server discovery with UDP (or, at least, none that seem to work with the current version of Unity free version).