Assets DarkRift Networking 2: The next level of networking

DarkRift Networking

Update 13/9/17!
Open beta has started*! You can download the latest builds in the most recent posts or visit the community Discord server links channel!

*It started a few weeks ago, I just forgot to post an update!

Update 04/04/17!
Alpha testing signup is available here!

Update 20/03/17!
Lots has changed since this post so take it with a pinch of salt as lots of features have now been changed!

Also, there's more discussion on the community run Discord channel, join here

Update 05/03/18!
DarkRift 2 has been released!

Keep up to date on the new forum thread here!

Original Post
So it's nearly been 2 years since I released the original DarkRift Networking and whilst I'd say it's done a grand job in it's short stint, it's fair to say that I'm a significantly better programmer now than I was when I wrote it and so trying to maintain it is very much a chore. As is trying to add anything to it without breaking it...

So starting almost exactly a year after I started writing the original, DarkRift Networking is being remade from the ground up so that all the small design decisions I took and have now come to regret can be thrown away. I'm not changing the principle of operation, it's still going use messages with a tag and subject because I think that is a brilliant setup, but the majority of the API will be changing to make it generally better.

What's Implemented so far?

• Modular network connections - DarkRift previously had a lot of it's send/receive protocol code in the same class which was a pain, the new one splits this up so that new protocols (e.g. UDP/RUDP etc) can be added without requiring huge rewrites.
• TCP communication - Well that was always there...
• Object based Servers - DarkRift was always based on a static API, which was fine, but there are some great circumstances for object based server instances. So now, if you want a new room for instance, then it's not tricky to just spawn a new server object for that group of players!
• Thread Dispatcher - Trying to embed the original DarkRift into a unity instance was a pain in the backside because DarkRift was very multi-threaded and Unity's API just doesn't like other threads. The new dispatcher allows you to queue tasks to the main thread so anything that needs to talk to Unity can be run on that thread without holding up DarkRift and the rest of it's processing.
• DarkRift Serializers - Essentially, very fast performing serializers that really outdo the old BinaryFormatters that DarkRift used to use. I've had the prototypes for these for ages but because of the old API it would not have been an easy integration. I'll get some better comparisons on these at some point but you get the idea...

• Plugin system - The old plugin system was beautiful and simple, so not much is going to change here. Put a plugin in the Plugins folder and that's it!

What's still to be done?

• UDP and RUDP - My UDP skills are a little rusty so I need to brush up a bit first but this is such a requested feature that it's definitely going in.
• Process based Servers and Server Groups - The idea here is that for Unity it's not always good having new server objects because all your servers are still in one scene; hence server instances will allow you to spool up whole new server processes with their own scene etc. and still be able to communicate across all the instances. Server Groups are going to be the managing process that handles the spawning and hopefully allows server's to communicate across hardware boundaries as well... Hopefully...
• Hot-swap plugins - Swapping plugins in DarkRift currently entails having to restart the whole server which is a pain; therefore I want to make it so plugin state can be saved, a new plugin loaded and the state copied to the new plugin so that play can be uninterrupted during server maintenance.
• Helpers - Essentially just making lots of small helper classes for things like validation, login, database queries etc.
• More... - There's always going to be more to add, suggestions as to what are welcome

I'd love for this to be more than a large block of text but it's not particularly easy to think of photos I can post that aren't just of a console

I'm going to try and keep this thread up to date with the latest progress, which should be a little faster now I have less work, but naturally please feel free to ask questions/make suggestions/whatever really

Jamie

 

Will Dark Rift 2 be similar in terms of API with the current Dark Rift? If I were to upgrade my Unity projects from Dark Rift to Dark Rift 2, would there be a lot of changing?

 

There will unfortunately be a fair amount of changing but it will only be in terms of the API; there shouldn't be any reason to change the structure of your code at all just some function names/class names/classes variables are in/etc (an example being ConnectionService is now ClientConnection). The only exception may be if you're using the embedded server because the way it handles the thread merging is quite different. I plan on writing a guide on how to do all this which will hopefully cover all eventualities

I know it's not ideal in the slightest and I'm not going to just abandon the first version, however it'll just be receiving bug fixes only.

Jamie

 

Hot-swap plugins now sort of implemented, the the code is there it's just that I'm faced with an intriguing threading problem I'm not quite sure of the best way to implement. I know how to do it I just want there to be another way

Essentially this allows developers to update their server code without having to pull the server offline for maintenance as you have to in almost all other servers. Players remain connected throughout and they can continue their games without even noticing the maintenance has occurred

 

UDP communication successfully implemented

Next up is RUDP I guess

 

A little update on what's been happening

UDP is now fully implemented, though there is an rather occasional bug that's creeping around from time to time, and I'm looking into the best way to do RUDP without affecting the UDP code too much (trying to keep it modular and all). I'm now sort of drifting in and out of sections, today I have done a fair amount of the server group code, including implementing the pipe code needed to communicate between DarkRift processes, and I've also rearranged the namespace structure a little and been implementing various bits of API/helper functions/etc.

I'm looking into writing a lot of Unit tests, some running rather complicated routines (I cannot wait for to write them for server groups...), in order to make testing a lot easier and a lot more thorough. I've never really had the knowlede to use them until now

I also spent a large portion of today chasing down an error that seemed to mysteriously appearing from the .NET thread pool and crashing my program without ever entering code I'd written...

ADDED: Oh yeah, and I've been working on a nicer website that uses bootstrap to make it a little more interesting and nicer to use than the old rubbish one

 

So today I made some fairly major changes to the encoding/decoding objects as I realised they were, in fact, more complicated and confusing than the original DarkRift routines were. Now however they're much simpler and should be easier to maintain.

[Removed because it turns out the values may have been more inaccurate than I thought]

 

Just commenting so I can keep an eye on this

 

Well I'm still here, just quiet

I'm currently just doing odd bits and pieces, I've refactored a large part of the messaging layer, added message distribution (and a very nice distribution interface for plugins too), added a custom serialization interface, written plenty of unit tests, added the beginnings of configuration files (thin time in xml) and am also trying to ensure plugins are loaded into a separate app domain for security.

I'm thinking that process server groups and cross-hardware server groups will probably be something added in the version after first as they're a lot of work and will probably skew the schedule quite badly.

There's also quite a few bugs in the networking code still which I need to fix but can't really face it at the moment

 

Nice, nice
I'm so interested in this new features!
I was writing my own solution for distributed networking for MMORPG I am working on (as I said to you in our emails!), but if your solution will be ready in a reasonable time I will continue with your plugin!

 

Today I implemented server commands so that administrators can control the server's workings while it's running, There's a few bits and peices to clean up still and the generic server commands still need to be added but it's good progress.

I've also managed to get plugin sandboxing to work, it may not be a very used feature (and there's still quite a bit to go) but it will allow people to load plugins into restricted app domains to that the plugin can't act maliciously; good if someone's sent you code and you don't trust it although there will be a performance penalty for it (I'm not sure how bad). Also good if someone wanted to launch a cloud service in the future as they wouldn't need to load an OS virtual machine for each customer, instead DarkRift could manage it for them.

I'm thinking about possibly renaming DarkRift for this version, as much as I like the name there's now a game development team that's launched under the same name and it would be a lot easier to rename now than have to rename later because of them

 

I've had a lot of fun learning DarkRift today; looking forward to the continued development of DarkRift 2!

 

The past few days I've been isolating the lowest level networking code in DarkRift, making it so I have 2 classes which provide connection oriented, message based networking for UDP and TCP. This means that they're the only bits that need to worry about sockets and the rest of DarkRift can just worry about sending messages down connections.

This has already fixed a fair few bugs in TCP and once UDP is finished it should make it easier to implement RUDP as well as the rest of DarkRift

 

Nice job! I want to try a beta as soon as possible!

 

There's a good chance alpha and beta will be sooner than expected actually, there's not too long a list of what needs to be completed on the server:

• UDP/RUDP
• Databasing (though I'm going to use OleDb to speed development up and also open up more databases for DarkRift Networking users
• Possibly refactor the message layer again
• Finish commands
• Connect/Disconnect events

Then a load of other stuff that needs sorting:

• Make a Unity specific API
  
• Write shedloads of unit tests that I neglected!
• Write documentation
• Write examples
  
• Finish the new website

That could be about a month sooner than I was aiming for but it's still variable...

Jamie

 

Hi @Jamster,

I want to make online multiplayer game. Sort of like a co-op / battle arena map like Hell divers/Warframe/WorldofTanks
The network I need is a matchmaking in global and local. player 1 creates game and hosting. Player 2, 3, looks at a list of created games and finds player 1 game and joins, player 1, 2, 3 plays together. or maybe auto matchmaking. And also store player data.

1. If I buy DarkRift, do I need setup something else? like I have to rent dedicated server / VPS, server scripting? Do I need to setup something in DarkRift website?
2. Can use this for MMORPG ? and store player info, inventory, status, progress, skill etc.? If not do you have any suggestion to create this ?
3. What is the different between free, pro, extreme ? Only CCU or there more limitation?
4. What happen if I buy pro license, but my CCU over 200 ? player cannot sign in or other player get kick ? or there is notification that my CCU is over ?
5. If I buy extreme, Is it really no CCU or MAU ?
6. 1 purchase for 1 game or many games?
7. What is the different between current darkrift vs darkrift v2 ?

Thanks in advance.

 

Hey,

DarkRift Networking isn't designed for player hosted games (and you'll have a hard time trying to make it work for that). It's intended to be hosted on a VPS in the cloud to which players can then connect to and play rather than players being able to host their own and thus the licenses aren't really going to work for you as none of them allow you to distribute the DarkRift Networking server with your game...

1. As I said, you'd really need a VPS (or similar) to use DarkRift and there would be some server scripting in the plugins if you wanted to create an authoritative server. Nothing needs to happen on the website
2. Yup, DarkRift would work well for that but you'll need to have a good understanding of how they work server-side first.
3. The differences are only CCU limitations, Free 20 CCU, Pro 200 CCU and Extreme Unlimited CCU.
4. It depends on the circumstance: if they were all on the same server new players wouldn't be allowed to connect until one had disconnected (you'd also get a warning message from the server), if they were across multiple servers you'd simply be breaking the license (and thus the law) if you went over. Pro's designed more for closed betas and development servers, you probably want extreme for any released games.
5. Yup, it's really unlimited CCU
6. As it follows the Asset Store license currently it is 1 license for as many games as you like Can't say that will be the same for DarkRift 2 yet though.
7. DarkRift 2 has quite a few differences: Works with UDP, RUDP and TCP rather than just TCP, different API, hopefully will be Open Source with a dual license for commercial use (though I'm not certain), is much more object orientated, much more maintainable and loads more.

Hope that helps,

Jamie

 

I should give an update actually...

I've been working quite a lot on the underlying networking sockets recently, I've abstracted the socket code away from DarkRift quite a lot and I'm moving it to its own library with the sole purpose of providing a uniform API for TCP, UDP and soon RUDP. The library (named, bizzarrely, Hazel after a hazel tree I walked past at the time) provides me with event driven, connection oriented, message based communication for any protocol I write a wrapper for and I guess I'll probably open source it under MIT or something later. It has a similar purpose to Lidgren but will have a bit of a wider scope for other protocols.

In terms of DarkRift not much has happened, work is mainly focused on Hazel at the moment

Jamie

 

Looks great! After asking a few questions in the Photon thread and receiving hateful answers from the developer(because they were excellent questions), I have decided to look for other, better solutions. Forge also seemed like a great answer but the developer appears to be very slow and unreliable. In order for me to purchase this for the extreme license I will need the following questions answered. as this networking solutions sounds great!

1. I seen another thread that said that Unity 4.6 requires the $25 Good ol' Sockets plugin. If I upgrade to Unity 4.7, will I still need this plugin? My games are all currently on Android.

2. Photon was very sketchy and never really told anyone that you need to purchase new licenses for every 100 CCUs. Is this similar? What I am asking is, if I buy this for $100, will I need to shell out more money for servers, 3rd party services, etc like Photon, or is $100 all I need to spend? I am not looking to shell out more than I need to as I simply do not have the budget right now.

3. I made a cannon shooter game for Android. The way I want to implement multiplayer is a 1v1 match where each player takes turns shooting with the goal to knockdown all the players flags. Will this work for this type of game since it claims to work great for physics? Will it be player hosted, server-client, or dedicated? Here is a link to my game just for reference: https://play.google.com/store/apps/details?id=com.Sempiternal.KTF_Full&hl=en

4. Is this developed by a one man team or multiple members?

Otherwise, great job with this! I'm confident this is what I need!

Thanks, Sempiternal Studios

 

For future reference, the thread for DarkRift 1 is here. Let's try and keep this for DarkRift 2 talk please

Nevertheless:

1. With Android or IOS, as far as I know, you still need to either purcahse Good ol' Sockets or get Android Pro (I've not tested with Good ol' Sockets though be warned)
2. You will need to rent/host a server yes, DarkRift Networking is only the software for you to host yourself on a service such as AWS or Azure.
3. You'll have to run a DarkRift embedded server in a Unity instance to get the physics you need for that game, there's a performance bottleneck in there due to Unity only working on one thread. You'll pretty much always get better results with DarkRift if you can run the server standalone instead. Using DarkRift would give you a dedicated server
4. Just me at the moment I'm afraid!

Best bet is to just download the free version off the Asset Store and see if it works for you.

Jamie

 

Work has finally started on RUDP communication! Hooray

I think to play it safe I'm going to release a beta of the low level networking library Hazel as soon as possible, hopefully people can then be checking through that and implementing some test projects in it whilst I finish DarkRift and thus there'll be less to go wrong in the DarkRift beta!

Jamie

 

Today I managed to get a bit done, I redid some of the serialization to make it a little more straight forward and safer. Instead of the DarkRift serializers I've decided that it's only sensible to enforce manual serialization: you get better performance, better control and more flexibility. The DarkRift serializers also didn't cope well with cycles in the object graph and a few other smaller issues. In the end it's just going to be a lot better for everyone I think....

So now there are send routines for a byte array, a DarkRiftWriter and a IDarkRiftSerializable object which will allow you to refer the serialization to the object instead of managing it from outside the object. On the receiving side you can get the byte array, a DarkRiftReader for the array or you can deserialize to a specified IDarkRiftSerializable.

It's a lot nicer and a lot simpler than it was going to be

 

Seems nice! I will be happy to use it in a MMO context

Give me some news!
I can't wait to see it completed, so I can test its features

 

Still working on RUDP code at the moment! Also thinking a lot of things through and doing a lot of maintenance on the old DarkRift

 

Hi Jamie - I have just recently discovered DarkRift and decided to hold out for DarkRift 2 - It seems to be just what I am looking for.

I'm sure you've had several inquiries about help developing your product, and I would like to add myself to the list. If you're interested, PM me and I can send along my Resume - I'm not looking for a job, I would simply like to be part of this project as I've written a lot of server side code for several projects, both personal and professional. Server code is my passion.

Anyway, let us know when a beta is ready, I am eager to test

 

Nearly there with the first stages of RUDP now, you can now send reliable messages which are guaranteed to arrive and will guaranteed to only arrive once! Needs a little tweaking still in order for packet re-transmission to be based of the ping to the receiver and there is a bug with sending data too fast after connecting but looking good so far.

Next is fragmented message and grouping of acknowledgements. I don't think I am going to included ordered messages as with multithreaded systems like DarkRift and Hazel the events may not be invoked in order anyway so there would have to be some form of blocking between receiving each ordered message which just doesn't fit with the concept Hazel and DarkRift are based on.

Jamie

 

Had a little time today just to tweak things with UDP/RUDP and to remake the handshake message a little so that it's acknowledged by the server once any NAT is broken through. Basically it's just a reliable message containing no data and when the client receives the acknowledgement it knows the server is ready for data and has accepted them to the server.

Seems to be working alright...

Spoiler

 

Keepalive packets now implemented for UDP!

 

This is nice!

 

So it's looking like Hazel is very near to beta now, there's a couple of things that I may change beforehand but everything is pretty much done in it now.

Still need fragmented messages but that could be added later, I just need to see if there are any performance updates I need to do that wouldn't be backwards compatible if I left them till later.

Jamie

 

Ahhhh wait I have to do documentation still!

 

Added recycling on any frequently used objects so that if you find you're getting huge amounts of garbage you can easily make Hazel recycle objects, also completely optional whether you enable it or not

Everything seems to be looking good...

Spoiler

 

Really good! I extremely want to try this as soon as possible!

 

Well this seems a lot nicer than PDF files now doesn't it...

Spoiler

I have work to be doing... I probably shouldn't be doing this right now...

 

Yeah! Really hyped for Hazel!!!

 

I'm wondering whether people would prefer tutorials./walkthroughs/articles/guides etc. to be on the same site as this (i.e. with the reference like MSDN do) or whether people would prefer it on some form of publicly editable wiki page?

Or perhaps a mix of both, I post onto the reference site but the community has a separate wiki they can update?

Would people actually contribute to a wiki or should it just be the reference site with the articles etc. on?

 

Probably the mix between them can be the right choice.
An official documentation/reference/tutorial/whatyouprefer and a wiki/forum for the community.

DarkRift is plugin-based, so a place where users can upload their plugins could be useful too. If you provide it on your official site it could be better.
If you need help, you already know my email address!

Anyway, I don't know how different your new API will be. Probably I should prepare a network wrapper for this.
Waiting for a beta!

 

I think that's the way I'm leaning, I like the idea of people being able to edit the tutorials etc I post though but I guess if both are there things can always be moved backwards and forwards pretty easily.

I was going to provide the wiki on the GitHub page for Hazel/DarkRift, I feel like that would be the easiest solution for the time being at least and most people probably already have a GitHub account. It may move to my own site later, I guess the docs will need to be on my own site. For plugins that may very well be on my own site but that would take quite a bit of work.

The DarkRift 2 API is quite different but it should still feel familiar and be easy to get used to, Hazel's API is completely new though

 

Finally got around to setting up the website!

http://www.darkriftnetworking.com/

Any feedback welcome, I'm definitely not a website guy!

 

Ehi, nice job! I want to suggest something!
Here: http://prntscr.com/b9fxpo you should fix your paddings!

 

Haha yeah I'd noticed that! Can't figure out what's causing it, think it's a job for a rainy day!

 

Added IPv6 support in Hazel!

I'm now most of the way through documenting the code and will then make a start on a guide for using it. Few other tiny fixes and we're good for a beta!

 

Oh, this is a nice news!
I'm ready to try it in my project!

 

I have a question regarding licensing
If I get DarkRift EXTREME license, then DarkRift2 is released, will I have to purchase a DarkRift2 license?

 

What is the diffrence between darkrift free, pro and Extreme editions I cannot find the difference anywhere?

 

The CCU limit.
With free version you can evaluate the system and you can have max 20 concurrent users connected.
With pro version you can have max 200 concurrent users connected.
With extreme version you can have max 65535 concurrent users connected (ushort max value).

 

I'm not 100% certain but chances are it would be a free upgrade or at least appropriately discounted

I'm still not sure how it's going to be priced so I can't say for certain

 

Oh ok thankyou .. and I just wanted to know... I am a beginner and I dont know which one to choose from photon or Kii or dark rift and I am a C# programmer so which will be easier for me... But I have some experience with .net like creating DAL layer and a basic factory pattern actually.. So which one will be easier for me to work with?

Also very Important since I am mostly working with WebPlayer and WebGL builds does darkrift support WebGL or WebPlayer builds or atleast will DarkRift2 support WebPlayer or WebGL builds?

 

I don't think DarkRift plays well with WebGL at the moment because you can't get access to the System.Net namespace. I'm looking at ways I can sort that though and I think DarkRift 2 will (probably not at first release) support it.

It's a shame because I suspect you'd enjoy using DarkRift if you're a C# programmer!

 

Ok thanks for your reply. That's too bad to hear Hope DarkRift2 supports it. I was too much convinced to use it. But no problem, you are doing a great job all by yourself Keep up the good work. Looking forward to DarkRift2.