Photon Framework - MMO Client/Server Foundation

My name is Christian Richards, I run CJR Gaming and today, I would like to present my "Photon Framework".

I have been working with Photon for over 2 years now. My first videos were on the Release Candidate of Photon 3.0 At the same time I started the videos, I started the framework. I started simple, with code to get photon to work. Then I rewrote it from scratch when Exit Games added in the Server2Server piece for Photon. Since then I went on to build what this framework has been turned into.

First a little bit about what this isn't:

• It isn't an MMO "easy" button
• It isn't complete
• It isn't simple scripting

That is to say that this framework is the beginning of an authoritative client/server setup. It isn't a simple set of scripts, but a foundation built by a developer who has worked for 13 years building applications for 3 companies.

So if it isn't complete and it isn't an easy button, just what is it?

Right now it is comprised of:

• Proxy Server - main gateway for all communications
• Login Server - Registration, Login, and Character creation/selection.
• Chat Server - Guild, Trade, Tell channels.
• Region Server - All game code.
• Interest Management - Radius based object management - included in the region server code
• Physics Module - Bullet Physics used to control collision

Through my tutorials, website and streams I continue each week to push this code a bit further.

So how is this different from the vast majority of other "MMO posts"?

As I said above, this is a foundation with which to build an MMO. I strive very hard to make sure it is modular, easy to work with, and easily updated. I also work week after week taking in feedback and comments to iron out kinks in the code.

At the time of this writing, I also have prototype code for a World server, client prediction, and I'm working on getting stats working. The world server will eventually handle communication between region servers and will serve as the server that groups are held on so they can go across regions. The only concrete character stat I have is MoveSpeed because it is used by BulletPhysics and the client to determine where others are going to be until the server updates that position directly. All other stats don't even exist until they get written so you aren't locked into a specific set of character stats.

It is my intention to continue to build this framework piece by piece in such a way that people will be able to quickly modify it to suit their needs. Again, this sounds like an "MMO easy button" but it isn't. You actually need experience programming to work with the client and server.

If you think this is something you are interested in, hop over to my site and consider purchasing it. And be sure to check out my YouTube channel where I constantly post new videos adding on to the framework.

Some of you may be asking why I'm selling this on my site and not on the asset store. Really, it boils down to the fact that it relies on more than just Unity3d. It isn't just an asset for Unity3d, but rather a package that contains both the server and client components that happens to use Unity3d for the client portion of it.

In the future as I build more modules for the framework, I will post them up for sale.

 

Glad to see you finally got to a point to "release" what you were working on. I of course have been following you, and thank you for all the learning tools you have provided.. Got a purchase from me, but you already knew that.

 

For those that are interested, the prototype repository is updated every time new code is added.

Tonight's stream on twitch made some changes and got smooth movement up as well as client side prediction for other players.

I've also already begun the modifications to work with stats. Once the core functionality for calculating a stat is done I'll focus my attention on effects (buffs and debuffs) that modify stats and then move on to skills (passives and spells) that apply effects and lastly move on to items (armor and weapons). As I finish different pieces of the prototype, I begin incorporating those changes into the tutorials as new modules.

 

I have been following this series for awhile, its great to see you bringing it to the forums! I can definitely recommend this to anyone who is interested in building a well thought out framework for online gaming. Now you can just get the code but the videos are really great for understanding the reasoning behind the framework decisions. Since it is open source you could always modify it to fit your specific game if you needed.

BTW, the physics module is looking really awesome. I have been enjoying watching this series progress, especially now that we are getting to the fun parts.

-Dane

 

Love your series CJR, they're very educational and a great motivation

I do have one question about your physics module. Now that you've moved over to bullet, you're not longer using a managed physics engine like DigitalRunes so you lose out on web player builds. Any reason why DigitalRunes didn't work out for you? Web player builds is super important to me.

 

The reason I moved over to BulletPhysics was to keep everything using free components. If you notice, I am actually using a C# conversion, not a wrapper. Using this conversion shouldn't be causing web builds to no longer work. I did this specifically because web builds are also what interests me. I'll do some testing, but I was 90% sure that the Bullet Physics conversion was working just fine with Web Builds. If you are running into specific problems, I would LOVE to know what you ran into so that I can correct it, as I want this viable for iOS, Android, Web, and Win/Mac/Linux support.

 

Ahh my mistake, I thought it was a wrapper. Wow, I'm switching over asap!

 

For you, and those interested, the conversion I use is located here I did not convert this, but did clone it, made several changes, and reuploaded to my bit bucket. You can find the original at Google Code. As you can see, the author's last changes were getting it to BulletPhysics 2.79, I updated it with the changes made up to 2.82.

 

So I'm testing out BulletXNA right now on Unity, just to see how I'll be able to handle CSP in the future, and I'm a bit worried about the performance.

Here's a screenshot of my scene:

http://content.screencast.com/users...2d-47c6-a428-96afad02d205/2013-12-14_1240.png

I followed the same setup you did on the physics module, and right now with 32 kinematic character controllers, it's costing 10-11ms per frame which is pretty concerning. Have you spent any time benchmarking the physics modules yet? I'll do a comparison with BEPU later

 

No, I haven't done benchmarking yet. I went looking for anything that was C#, open source, and not a wrapper around a C++ library. The only thing I had found was the conversion of BulletPhysics. I wrote the code in such a way that it should be easy to remove BulletPhysics and move to another physics engine. So if you find better success with BEPU, i can work on a new BackgroundThread, IPhysics implementation, and exporter and present it with the code. Especially if it is better performance wise.

One of the things I don't like about BulletPhysics is that many of the objects are structs, which means any time i say var x = obj.posistion; x is a new copy of obj.position and will be garbage collected.

 

So I have started going through BEPU Physics. It does appear to be much better in terms of memory usage and usability. Once I get it fully functional, I will probably record a set of tutorials to switch over to BEPUphysics.

Right now I'm just getting it to work in my own code.

One of the nice things is that their demos show you how to already get moving platforms working, which is a big plus for many people. I'm really looking forward to seeing what all it can do.

 

That's great to hear I did a test with boxshape rigidbodies, and not only is the performance better by about 0.1-0.2ms after spawning 32 objects. It also produced no garbage, which is a really good sign. I plan on doing a comparison of the kinematic character controllers, but the latest version of BEPU doesn't seem to run by default in Unity, so I'll need to dig in to figure out what's wrong.

 

The "problem" is that BEPU uses .NET 4.0 to handle its threading with Parallel.For

You can do one of 2 things, go into the ParallelLooper and comment out the Parallel.For and use Thread.Sleep(0) (also comment out the #if Windows and other directives) or you can go in and download the TPL backport for .NET 3.5. Then you just have to set the properties to use .NET 3.5 and you can pull it into Unity.

 

The TPL backport sounds promising, I'll try that one out when I get home tonight.

 

Alright I plugged in TPL and it's working great. Tried out the kinematic controller, and I have a capsule moving aroundish. Unfortunately I'm out of time tonight to continue testing. I'll try to set up a benchmark tomorrow and let you know how it performs compared to BulletXNA

 

The Photon Framework has been updated. I have included everything in the videos up to this point. That means that Interest management is fixed and ready, the Physics Exporter is included automatically, and Physics is now more responsive. This still uses BulletPhysics, but in the near future, along side the videos for the stats module, we will cover the conversion over to BEPU Physics.

 

Alright, so with BEPUPhysics, it takes around 0.9-1.2ms to tick 32 kinematic character controllers. Compared to BulletXNA which takes an average of 10 ms. BEPU generated 4.5kb of garbage, Bullet generated 20kb.


http://content.screencast.com/users...ba-40bf-b9af-609d880d667d/2013-12-18_2333.png

So we're looking at BEPU being 10x faster than BulletXNA in this case. I remember someone on the forums had implemented Jitter into their project but I don't think that user ever posted any benchmarks.

Anyhow, you were already going to switch over, I just wanted to have some numbers to throw around in case anyone asks.

 

Wowhoho, just stumbled upon your project. Seems like you dont have own forum so I`m going to use this thread for all my questions, ok?

I`m a noob in a Photon and the Framework - just began to look through tutorials. Sorry if I ask too easy thing but I`m so impatient to know! In fact I try to implement a hexagonal-tile-turn-based-tactics game, if that context does matter for the following questions.
- Is there a roadmap? Could you probably update the 1st page with it?
- What updates are going to be avaliable for 100$ and what will be above?
- Is it possible/easy to have a Photon authorative server created/found on demand by a player in lobby?
- Is it any easier to achieve fog of war than with Unity Networking? I mean enemies positions and animations get updated while visible for certain clients only.

If these things are ok, than your framework is going to save me tons of time and really worths the money.

 

The roadmap isn't "well defined" I know what features I want to cover, but not their order. I'm looking at Smooth Movement, Stats, NPCs, Items, and Quests in the near-ish term. I already have the stats code prototyped out. I want to finish up what I call "StatusStats" which are things that change frequently such as current hp/mp/stamina and experience. Again I'm working on something that doesn't require those specific stats because I want others to be able to say "We don't have experience or levels"

The 100$ purchase gets you everything I've done up to this point. I am not planning on updating it beyond this point except with fixes that I or others encounter and a separate download that will replace BulletPhysics with BEPU physics because it appears to be a superior product compared to the C# conversion of BulletPhysics.

I have not worked on a Lobby/GameServer module. It was something I considered, but my focus has been on things generally used in MMOs. It isn't "hard" to make one: you would start with the login server and after they logged in and moved to "in game" that server would need to know all other servers, create a "key" to tell the game server you are who you say you are and go from there.

the "fog of war" is the interest management piece I put together. right now that works for single players. If you are aiming for an RTS or 4x game or something that can see pieces and parts from different entities, then that is a horse of another color, and while it is possible to add on, it isn't something it does currently.

These are all things I would like to work on and make available, but I only have so much time each day since I do have to work for a living.

 

For those who are interested, this week I spent recording the entire stats module, it is 7 videos long ranging from 7 to 30 min. I am encoding them now and expect them to be posted tomorrow.

This module defines a generic way to build stats without locking you into specific stats. Most places force you into HP/MP and you are stuck with experience, etc being defined for you. This takes that out of the equation by generically defining what a stat is and how you can use it across your setup.

 

After posting 2 hours of new videos covering the Stats Module, I decided to do some streaming today while working on my prototype code. Over the course of 5 sessions today, 6hrs 25min, I was able to build a prototype that involves spawning NPCs, Adding them to Recast's Crowd object. Building a simple Patrol AI, and getting it to patrol around 3 points. With the prototype done there are a few things I want to change about how the prototype works before I cover it in any videos.

So at this point, in addition to initial Framework up to the Physics Export and Interest Management, we have working stats which are defined by you, the developer, and not by what i think stats should be and in my prototype code, waiting to be covered by tutorial videos, we have client side prediction and smoothing, a World Server, delayed logout, and the initial groundwork for NPCs with AI.

In the coming weeks I'll be working on my prototype code to include Quests, Shops, Items, and AI for agro and guarding. Later than that I will be working on transferring players across region lines through the world server for a seamless world experience while allowing scalable regions for those more adventurous.

 

Can you please explain your purchase options.

After a lot of hunting around all I could find was a link to subscribe per month.

I don't mind buying OR typing in the code, but for some reason all the youtube videos and their options are too low rez for me to follow along from Australia.

 

Sorry for the delay in response.

I was producing videos for free for since Oct 2012. It started a bit sporadic since i wasn't sure how many people were actually interested in a framework on top of Photon. Since then I have produce 30.5 hours of code and I make a new video each week covering about 30 min of code as we build each system.

Because of the amount of code and interest I decided to make the code for the first 28.5 hours available for 100$. This covers the parts I outlined at the beginning and can be purchased here.

The Plus subscription gets you access to the code and HD video download for anything not included in the base framework purchase. At this time it covers the Behavior Tree Library, the Stats Module, and the AI code that can be used with the RecastUnity module.

The Premium subscription gets you the same as the Plus subscription but it also gives you access to my bit bucket repository which contains my prototype code. It doesn't overlay perfectly over the base framework because I develop it on a separate machine to keep from accidental code spillage. There are also classes that are renamed because when I record, there are times where I decide a class name fits better to keep others from being confused.

(thanks for helping point out that my framework purchase wasn't visible to non-members, I've corrected the problem so you should be able to have a look.)

 

I highly recommend this server foundation for anyone with technical experience.
Coding a full on authoritative server is no small task and this saves a lot of time getting the basics done.
The code is of good quality and easy to read once you get the hang of it so it is a great foundation to build on. That and Christian has over 50 hours of tutorial available on his youtube channel so you can get to know the code inside out and even code it yourself for free.

I have 4 years experience as a web developer (mainly PHP/Symfony2 the usual) and had no trouble understating the code and extending on it. Very satisfied so far, and imo the product is worth more than the current price, that's 2 weeks of code debugging for any experienced programmer.

 

A quick note:
Right now the website seems to think the tutorials haven't been updated since mid January
http://www.cjrgaming.com/

However, the on YouTube, they continue on:
http://www.youtube.com/user/cjrgaming/featured

 

Yes, being sick I neglected updating the website with the latest source code. That has been remedied and the smooth movement module has been completed. The next module I will be looking at is moving from the BulletPhysics C# port over to BEPU Physics (http://bepuphysics.codeplex.com/) After that I will begin looking at the NPC and AI portion so people can begin looking at writing their own AI to get more interaction in their servers.

 

would be interested in purchasing but i am not smart enough to replace the AI you have with RAIN AI [i am dedicated to that AI].. would this framework work for shooting games? does it have a DB attached?

i was going to build form scratch using uLink before i saw this thread.does it include good client side prediction and input [whatever thats called wwhn its optimized for mmo]

 

Question for you Mr.Richards, how much work do you believe it would be to convert this from Unity to CryEngine? I'm debating between using either or, and felt this would be a good platform for me to jump from since it seems you have a lot of the basic starting issues completed or at least to a point that can be learned from.

Thanks,
Braxton

 

I don't follow your tutorials however I have watched a decent chunk of the ones in the beginging to understand the general basics of the Photon Server SDK a long while ago. Sometimes I pop into your videos now and then if I see an interesting title; as you can imagine it's hard to follow along without knowledge of the full codebase though lol.

Anyway, I'm here to ask why you prefer xml serialization over google's protocol buffers for data serialization? Looking at performance comparisons here I don't see why you'd choose to serialize your types with xml instead. If there are reasons I'm definitely interested as I use protobuf and maybe I should switch to xml serialization instead.

https://code.google.com/p/protobuf-net/wiki/Performance

 

It was much less about what is best and more about what I could easily use to debug code if something went wrong. I intend to switch out the current serialization with something more robust, it was just at the start when I needed serialization but I wanted to make sure the data being serialized was able to be output in a debug when the client got it.

 

Can I please have some guidance in how to start with the MMO Framework. For some reason I am just not fully understanding what has been posted.
Once again I have no objection to paying or subscribing just want some guidance.

Bear with me as I explain how I see it:
* According to the website you can only get the earlier Framework sourcecode (Login, Character Creation/Selection, Chat, Region, and Physics modules) by paying $100 for it. Those modules are not covered by either subscription models so subscribing doesn't obtain it.
* The youtube videos will not go to 720p for me which makes it hard to type in the code when all I can see onscreen is a blur.
* According to the subscriptions you can download modules in HD only AFTER the modules in the $100 package.

So what's the best way of starting? I have no interest in just getting sourcecode files because I really do want to type it in. As I see it, $100 gets me code and I learn nothing. Subscriptions get me nothing for the earlier code but are useful to watch the later ones in HD.

Please correct or confirm....


Of course there is one more question: which is the best order to watch the videos:
* the website is missing the BEPU physics and Framework #15. Is that a miss or deliberate?
* youtube VIDEOS (https://www.youtube.com/user/cjrgaming/videos) shows oldest to newest which seems to be the one to follow....??
* youtube PLAYLISTS (https://www.youtube.com/user/cjrgaming/playlists) lumps all module components together. However FRAMEWORK 15 would be done in a different order than if I followed the chronological sequence.

Thanks for any assistance.

 

I'm going through the videos now.. Technically the Framework playlist is the first module. You will want to play all videos in each module before going to the next. What I'm using as a reference for which module to watch next is Christian's CJR Gaming website. If you mouse over the Framework pulldown, just do the modules in the order that they are listed there and I think it should be good. (Correct me if I'm wrong Christian..)

As for purchasing the source, I'm like you and want to key it in as I go through the videos so I can actually have half a clue what it is doing. However, with that said.. I purchased the source and have found it useful as a reference when I end up with typos that I can't find. If you do that though, keep in mind things inevitably get changed/added to throughout the videos and the source is the final product after all the videos it covers is complete. But definitely worth the cost even if just used for a reference.

Also some things are broke due to Photon changing some thing with their version updates. But all the problems I've encountered so far, the fixes have been covered in his forum (old forum link on the website) and/or in comments on the individual video pages on his website.

 

Thanks Shawn67.
That was the link I mentioned on his website that doesn't appear to link to all the videos. (Framework 15, BEPU, Prototype NPCs with Patrol) and maybe others. I've only done a quick compare.

Did you have any trouble watching his videos? I can barely get 480p from website/youtube and he does mention being able to download an HD version of the latter ones if you are on a subscription.

 

I guess I should have said I do the playlists on YouTube in the same order as he has the modules on the website. I'm not seeing the video you are talking about in the Framework playlist.. There is a video 15 (which isn't listed on the site) but it is titled Release Build Fixes. From looking at his main page, it looks like the playlists are shown in the order I think they need to be watched.. so once you watch all of the Photon Framework ones, then you'd move on to the Login module ones.. and then continue down the line... You wouldn't be doing any physics type videos until after you finish the region server module.. At least that is what I'm thinking from what I've looked at... I've yet to make it that far... lol

I haven't had any problems with reading them.. I'm watching them on a pretty good sized monitor though. There is no way I'd be able to do it on my notebook screen.

 

Sorry for the delay. Let me try to answer these as best I can.

Correct, the 100$ will get you everything up to the the BEPU Physics and stats modules. I was going to try and separate it out and decided I would rather help out those who help me out and so I kept everything up to the world server module. You are also correctin that the subscription model doesn't cover this code.

The early YouTube videos are a bit of a problem. When i first started this series, I wasn't aware of the fact that Camtasia for Mac had an automatic 50% scaling when recording. Because of this, those early videos (I believe up to video 8 ) are at a really crumby resolution and the original files are recorded at that resolution so I'm stuck with them unless I attempt to rerecord them.

Also Correct, the HD videos only started with the Stats Module, Behavior Tree code, and the modules following that.

The subscription gets you the ability to download the videos. I still provide the videos on YouTube in full 1080p once you get past the initial ones with Camtasia's crumby default scaling (good for keeping filesize down, bad for code because ( ) and { } look the same).

The order on the website and on the channel is the correct order.
Photon Framework
Login
Character Create/Select
Chat
Region
Physics / Physics Export
Stats
Smooth Movement
BEPU Physics Update

You are correct, the BEPU Physics needs to be put on the website, I missed posting it because i did that entire module all in the same day. Same thing happened with the Framework video 15. I'll be correcting that shortly.

I tried to make Video 15 just updating the Framework so it can be done right after video 14. Inevitably it does have some affect on the videos inbetween but will get you where you need to go so you can run it along the way. The last fix - updating the PeerId so that you use GUID.ToByteArray() is, as stated, in the forums - http://joomla.cjrgaming.com/index.php/forum/photon-framework/5256-photon-updates

 

Thanks for the clear answer.
Pity about the earlier videos, but that's understandable. I will manage somehow.

Oh, Shawn67, magnifying a blur just gives another blur. My 27" 2560 x 1440 screen will just have to cope.

 

I don't remember how good they were.. I just know I was able to make it through them using my larger monitor.. I was thinking it let me play them all at 720 though... *puzzles* Maybe not... They say memory is one of the first things to go... lol

 

Seems like a really cool framework. Is this still in active development?

 

There has been no updated on the website since May of 2014, it looks like he was last on these forums Dec 2014. I recently purchased the framework myself before I realized he's not been around much. The framework is well written, and I do hope he comes back and does more. Guess time will tell.

 

Is there anyone with the code that can share it ... this guys is not there anymore and i cannot make the server running at all.... I'm at video 14 so past the video that suppose to correct the problem...

 

I'm working on returning. Many things going on in the personal side of life. Right now I need to check out changes in the latest version of photon and new things in unity 5.0

 

Is there any way to get access to the prototype AI and NPC code? I would definitely purchase if that was the case.