[Open Source] UdpKit, a networking library for .NET/Mono/Unity

UdpKit is my attempt at perfecting a networking library. UdpKit does its work on the same level as ENet and Lidgren by exposing a packet serialization mechanism on-top of UDP with notification of delivery and loss of packets.

Features

• Runs on all major Unity platforms: Standalone (Windows, OS X), Web Player (Windows, OS X), Android (Free and Pro), iOS (Free and Pro)
• Does packet serialization on a background thread, saving precious performance on mobile platforms
• During normal operations it generates zero garbage (currently the only operation that generates garbage during runtime is when you connect somewhere, which only happens once).
• Extremely simple poll/event driven interface

Examples of how to use the library can be found in the src/udpkit.examples.* directories. Project files for building are found in /proj.

It is currently the only networking library other then unitys built in networking which works on Android Free and iOS Free, this means you can use it with your games even if you only have the free version of Unity.

Source Code

Can be found at https://github.com/fholm/udpkit, under a very permissive MIT license.

 

Sounds great.

Thanks, fholm!

 

Hey fholm,

Besides working on android free and ios free what separates this from something like lidgren for example?

 

Can you please give a quick rundown on how to get started?


Edit:

I managed to get the demo running; I just had to build it first and add the assembly to the Unity project.

The path to UnityEngine.dll has to be changed on Mac. I'm pointing to the one inside Unity.app (/Applications/Unity/Unity.app/Contents/Frameworks/Managed/UnityEngine.dll).

It builds in Debug mode. Release mode gives errors, and I'm not bothering for now.

So far, so good...

 

Wait a minute : you have added an UDP library to Unity Free for Android/iOS??

 

Yes.

 

Im getting errors:

Assets/Plugins/udpUnityUtils.cs(1,7): error CS0246: The type or namespace name `UdpKit' could not be found. Are you missing a using directive or an assembly reference?

 

So, fholm, are you planning to ship a prebuilt library on the Asset Store?

 

I pushed a new version to github, version 0.1.1.0, as a tag: https://github.com/fholm/udpkit/tree/v0110
It contains a few tiny bug-fixes, and also the C# dlls pre-built can be found in bin/managed/debug and bin/managed/release

 

This is because you need to build the project and copy the files to the unity project by hand, they are not included there by default.

You can find a new version, v 0.1.1.0, here: https://github.com/fholm/udpkit/tree/v0110 which has pre-built binaries in the bin/ folder.

 

Thanks a lot.

 

Any way to get a tutorial to make a group chat client with this?

 

yes thanks a lot

nice/interesting...

 

Is this a personal learning project you've decided to make public or did you seek out to improve other frameworks (enet/lidgren) in some way?

 

one of the examples is already a chat sample.

 

This library was born out of me building an FPS networking kit, and not being satisfied with any of the currently available libraries. While ENet/RakNet are good they will not run in the webplayer or unity free.

I have used Lidgren extensively, but I have several issues with it. Note that this is just my personal feelings about it:

* Uses a lot of bandwidth for acking
* Is almost thread safe, as in it will work 99.999% of the time, but during heavy load i have experienced crashes related to threads trampling each other.
* Has a rather naive ack/resend mechanism which makes it behave incredibly bad on networks with a lot of packet loss (mobiles, 3G, etc.)
* Doesn't work on unity free mobile platforms.
* Very gnarly interface to work with (everything that is passed to the library user has to be serialized into a NetOutgoingMessage)

UdpKit does not try to hide as much of the lower level udp details as say Lidgren and RakNet, for good and bad I suppose. For example it only exposes an unreliable and sequenced send mechanism, which will notify the sender of delivered/lost packets. If you want to send something reliable, you can do this on top of this mechanism, but there is no out of the box support for reliable sending.

Other things that UdpKit tries to improve is for example that it will not allow you to send a message which is larger then the MTU of the path to the other end. You can send larger messages, but you have to split them on a message boundry which makes sense for your application, and not just chop them in half at some byte limit.

The main thing UdpKit misses right now, is UDP-punch-through, as I have not had time to implement this.

 

how do I use the chat example?

 

Pushed a new version, 0.1.2.0 to the public github repo. Adds a few new features:

1) Added a generic UdpSerializer<T> which has statically typed in/out arguments
2) Changed the UdpBitStream struct to be a class instead so there is no need to pass it with "ref " everywhere, and renamed it UdpStream
3) Converted all utility methods in udpkit.unity assembly to be extension methods

And here's the link to the new tag: https://github.com/fholm/udpkit/tree/v0120

 

OK, I have the same question. But instead of just posting "me too!" like some brain-dead AOLer, I'll post my observations and results. Maybe we can all figure it out together, and not have to pester Fredrik too much.

So! I see there is a "proj" folder with subfolders for android, ios, managed, and win32. I know what three of those mean, but I'm not sure about "managed"... I see there's a udpkit.sln (solution) file in there, but when I try to open that with Xamarin Studio, it throws a handful of version warnings, and then displays a project with "Load failed" next to all the interesting bits (including both examples).

In the same folder are also a bunch of .csproj files, but I don't seem to have anything on my Mac that can open those. (However, it looks like the actual sample source can be found inside the src/ folder.)

So then I tried the ios folder next, but that contains only an XCode project... I suspect this is the source code for the library, and not something I should muck with as a user of the library. (I suspect I would need this if I wanted to rebuild libudpkit_ios.a, but I don't need that at the moment.)

I don't see anything in the bin folder for the Mac, so I guess I would need to just build from source? Or, do I need to build a library at all — maybe I can just throw the source itself into a project, and let it compile along with everything else? (But I suspect not, because the src folder includes some C++ files as well as C#... I have no idea how or with what I would build all that.)

I've found the src/udpkit.examples.unity folder, which contains a Unity project with a single scene. When I load this, I get the same error tredpro got above. Fredrik says (above) we need to copy some files to the project... but which one(s)? There doesn't seem to be a pre-built library for Mac.

In desperation, I tried copying libudpkit_win32.dll to the Assets folder of the Unity project (hey, who knows, sometimes a DLL is actually a Mono library, and works cross-platform). But this didn't change the behavior; I'm still getting the "UdpKit' could not be found" errors.

I think I'm stumped at this point. Anyone have any tips on getting started with this thing?

 

OK, I had to take a few weeks off, but I'm back at this now and doggedly determined to get it to work.

I contacted lumen.atri, who (as reported above) managed to get this built and stumbling around on his Mac. He did this by compiling a DLL and dropping that into his Unity project, though he couldn't remember the details and hasn't touched it since. Still, it's very encouraging to hear that it's possible.

On the other hand, the Unity docs on Mono DLLs is less encouraging about this approach, arguing that it's much easier to work with scripts than DLLs in Unity. These docs suggest that you should only use DLLs if you have to (for example, because the code is written in F#, or because you want to distribute a library without source).

I'll post again when I have something to report.

 

OK, I'm pleased to say that I've achieved a bit of success getting all this to build and run. This was by simply dragging the source folders (in the end, these were 'slimmath', 'udpkit', and 'udpkit.platform.managed') into the project. I did have to make a number of changes to the source, though. Here's what I ran into, and what I did about it.

Errors in udpUnityUtils (UdpKit not found):
--> No idea why it can't find UdpKit. It's there. Finally gave up and commented out the entire file.

Errors in demoSerializer.cs: 'UdpBitStream' could not be found.
--> True... searching the source tree, I can't find it either.
Prototypes don't match the abstract methods in the base class:
In udpSerializer.cs (UdpSerializer class):

Code (csharp):

1.         public abstract bool Pack (UdpStream stream, ref object o);
2.         public abstract bool Unpack (UdpStream stream, ref object o);
3.  

In demoSerializer.cs:

Code (csharp):

1.         public override bool Pack (ref UdpBitStream stream, ref object o)
2.         public override bool Unpack (ref UdpBitStream stream, ref object o)

--> Updated demoSerializer to match the base class prototypes.

Errors in udpBitStream.cs: 'SlimMath' does not exist.
--> This is in the slimmath folder, which I hadn't added to the project.
Added it, and these errors went away.

Error in udpLog.cs: No overload for method 'Write' takes 1 argement (line 91).
--> Inserted a new first argument WARN.

Error in demoPeer.cs: Delegate `UdpKit.UdpLog.Writer' does not take `1' arguments
--> Lambda functions and delegates, whee. The argument here is of
type UdpKit.UdpLog.Writer, defined in udpLog.cs. This does indeed take two
arguments (level and message), not one. So the calling code appears to be wrong.
Changed it to:

Code (csharp):

1.                 UdpLog.SetWriter((level,s) => Debug.Log(s));

Error in demoPeer.cs: 'UdpKitUnityUtils' does not exist.
--> This is because I commented it out. Oops. But when I put it back, I
still get 'UdpKit' could not be found. This doesn't make sense; there are
tons of other files which provide (and reference) UdpKit. Stumped.
So I commented udpUnityUtils back out, and replaced the offending line in demoPeer
with this:

Code (csharp):

1.             socket = UdpSocket.Create<UdpPlatformManaged, demoSerializer>(new UdpKit.UdpConfig());
2.  

But this fails to grok UdpPlatformManaged. This lives in a separate
udpkit.platform.managed folder, so added that to the project too.


SUCCESS! It runs now. I'm not sure exactly what it's supposed to do or how to use it, but getting it to compile feels like a victory.

I'm still stumped as to why "using UdpKit" fails in udpUnityUtils.cs, but works fine everywhere else. I've googled for similar problems, but come up empty handed. Does Unity have some odd dependence on where files are arranged in the asset hierarchy?

Fredrik, is there some other way you'd prefer me to submit the fixes above? I'm still only superficially familiar with git, but I could manage making my own fork if you like.

Thanks,
- Joe

 

What would be cool is if there was some sort of client-server architecture, like a node.js library that would talk to this.

 

It's definitely designed for a client-server model right now. (Not ready for P2P yet since it lacks NAT traversal.) It currently works the same way on both endpoints. There's a chat server in the examples.

Would you really rather use Javascript on the backed instead of .NET / Mono? (I'd much rather use the same language on both ends.)

 

With regard to issue #1, my rationale is this:

I would like to use UDPKit in a front-end server that does nothing but handle IO between clients and back-end workers. Ideally, I think it should be entirely event-driven, such that when there is no work to be done it uses no CPU, and it's never found sleeping when there is work to be done.

I don't currently know the best way to do this. BeginReceiveFrom / EndReceiveFrom? Integrate with libuv / libev / libevent?

And of course, I'm only talking about server-side changes. I understand that on the client side we have different underlying implementations, and I think the way it works there is fine.

The current design assumes that the main thread will be busy doing other things besides IO, and will periodically check to see what events have occurred, which is right on for the client, but in a front-end server that would not be the case. Instead, I would like the main thread to block on some kind of event notification mechanism, and call BeginReceiveFrom (or whatever) when an event occurs.

This is an important project, and I'm surprised there hasn't been more activity around it yet. I posted it to r/unity3d in the hopes of getting more eyes on it.

 

I agree about the importance of the project. For what it's worth, I've got it working now on both iOS and desktop. I built the server as a command-line app in Xamarin Studio, and I'm currently running that on a Mac, but will probably move it to a Rackspace server eventually.

For my needs, the server architecture seems fine, but of course if you have improvements to it I'd love to see them.

 

So it doesn't work in Unity Free for desktop?

 

Sure it does!

It works on any platform (and version of Unity) with access to .NET managed sockets... plus iOS and Android, since fholm has built and wrapped native extensions for them.

 

I believe it works just fine (once you fix the bugs I described a couple posts up).

 

Right. It doesn't appear to work in web apps, though. I suppose this is a security restriction. It's curious though, because I thought I recalled someone saying it would work even there. I need to dig into this more deeply, but haven't gotten to it yet.

 

To the best of my knowledge, it works on the following platforms, in both unity free and unity pro:

Desktop (Windows and OS X)
Webplayer (Windows and OS X)
Android
iOS

In practice it should work on all Unity Platforms, but you might need to do some small code changes for windows phone and blackberry support for example. As soon as i get my hand on devkit for blackberry and windows phone, I am going to verify it there also.

There is an update which will drop in a few days which removes all the issues that JoeStrout faced.

 

So, "a few days", turned into a few hours. I was pretty much done and packaged it up on the commute to work this morning. I restructured the entire project so that it's easier to build also.

New version is 0.1.4.0.
All the new code is in the master branch on github
There is also a new tag on github called v0140 which contains the 0.1.4.0 code including pre-build binaries for all platforms, including native libs.

Tag: https://github.com/fholm/udpkit/tree/v0140.

The easiest way of getting just the binaries so you can use them directly is to go here: https://github.com/fholm/udpkit/releases/tag/v0140 and click on the green button marked udpkit_v0140.zip, which contains just the binaries for all platforms.

I'm also working on a quick guide to get started with UdpKit + Unity more easily.

 

I responded to the GitHub issue.

My general stance is that I'm just not convinced about this, maybe if you are building a huge clustered simulation with several thousands of clients that go through some time of proxy/gateway server. But for those cases there are specialized libraries/solutions.

Have you tried using UdpKit to see what kind of performance you get in your scenario?

 

Okay, that's cool. So where do I put the DLL's? Seems that no matter where I put it, it doesn't work. Type or namespace name 'UdpKit' could not be found.

 

Try placing them under a folder named "Plugins".

Special Folders and Script Compilation Order

 

The DLLs should be put somewhere in the assets folder, there's no requirement on putting them in a special folder. You might need to restart Mono Develop/Visual Studio/Xamarin Studio, for unity to pick them up. And you might also need to run "Sync Mono Develop Project" from the Assets menu.

 

The native plugins for Android (.so file) and iOS (.a file), should be put in Plugins/Android and Plugins/iOS respectively. For references, here's a screenshot of a correct folder/file layout:

 

Redownloaded it, built it and now I get just one error about 'UdpPlatformManaged' not being found. At least it finds the DLL now.

 

Did you copy the udpkit.platform.managed.dll into the assets folder also?

 

I'm excited to see this project.

I've moved from one network library to the next and all have major issues that make commitment painful.

Lidgren has been the most promising of the bunch, but after spending some time working with it and combing through the source, I've become wary of continuing my work with it.

fholm, do you have any specific plans for the library in the future? Will you try to make it feature rich, or keep it tight and leave extra features up to the implementer?

So far I've only had time to jump around the repo, but it looks promising!

 

I'm glad

Yes, I have the exact same experience. There's a few good ones written in C/C++, but it always gets a bit confusing/hard to use from C# no matter how good of an P/Invoke API you wrap on-top. So being a programmer my first thought obviously was "I can do this better myself ..." ;P

I agree, again. I use lidgren extensively in SlimNet (github.com/fholm/slimnet). There are quite a few issues with it (weird/plain bad ACK:ing mechanism, very messy code, and the library has a few threading issues)

The two main features which are missing is NAT traversal and Path MTU detection, after that I will most likely just fix bugs and make it tighter/neater and not add any more features. I am intentionally keeping it very light and close to-the-metal. I am using UdpKit for my own networking layer for Unity, which exposes an actor/rpc/state-sync model, and seeing how easy it is to implement exactly what you want on-top of UdpKit, I dont see any need to bloat the library with implementation-specific details.

 

Quick note about using UdpKit in a Webplayer project:

You'll need a TCP server providing a socket policy to satisfy the requirements of the Webplayer security sandbox.

https://docs.unity3d.com/Documentation/Manual/SecuritySandbox.html

UT provides an example policy server in the Tools folder as a quick way to get going.

Maybe this will spare someone a bit of agony.

 

Pushed a few changes to the repo, mostly cosmetic API changes and a tiny issue with start/connect being called right after each other. Going to prepare a new 0.1.5.0 release soon.

Also I'd like to ask if there is interest in a package for the asset store? bundled with a demo, etc. free obviously, as the whole point of the library is to be open source/free. Maybe it will make it easier for some people to start with it?

 

I recommend making a basic demo for the free version in the asset store.

But the thing that is preventing me (and I'm sure many others) from using this is I can't use it commercially, I recommend maybe making a paid version with more of an advanced demo and allow users to use it commercially?

 

You can use this commercially, it's under the MIT license.

 

Just to be 100% clear: UdpKit is 100% open source under the MIT license, it is free for use in any type of project, including but not limited to: commercial, non-commercial, private, hobby, educational, etc.

 

I'm happy with the status quo, especially since you started providing compiled dlls, but I think the Asset Store might be a good way to get more users.

And more users means more issue discovery, and more social pressure to keep UdpKit alive and healthy.

So I think it's a great idea.

 

Tagged a new version, 0.1.5.0, pushed it to github and also attached pre-compiled binaries to the release: https://github.com/fholm/udpkit/releases/tag/v0150

 

Looks good! This appears to fix all the issues I had found in 0.1.2, including a problem with connections never being removed from the connection map, resulting in eventually hitting any connection limit.

However, it also creates a new issue... this was clearly an intentional change, but I don't understand the reasoning behind it. The udpSocket.NetworkLoop method, which is run as a thread to keep the sockets chugging along, used to exit once the socket left the Running state. Now, instead, it contains an infinite (while true) loop. So once the socket state is neither Created nor Running, the thread just sits there logging "socket created", "socket started", and "socket closed" as fast as it can.

For now I'm going to just comment out the while loop, but I'm sure you had some good reason for putting that in there — can you shed some light?

Thanks,
- Joe

 

Joe: I am glad it fixed all the issues.

The reason for the while(true) loop is to handle the case when an exception is thrown inside of the functions being called by NetworkLoop, so that the socket doesn't stop processing just because you might throw an error un-intentionally while packing/unpacking for example.

I pushed an update to the master branch which will handle eerything properly now, version 0.1.5.2

 

Yep, 0.1.5.2 looks perfect as far as I can tell!

Cheers,
- Joe