Introducing Uniject, a (pure) C# unit testing framework

We have developed a C# framework to facilitate unit testing OUTSIDE of unity in MonoDevelop and NUnit.

You can read more about our approach in this blogpost, but briefly it has the following objectives:

1. Complete code coverage of game code OUTSIDE of Unity in Plain Old C Sharp
2. Dependency Injection (DI) of controllable, mocked UnityEngine dependencies
3. Inversion of control (IOC) - Ninject can build many GameObject hierarchies automatically, simplifying or eliminating many prefabs
4. Familiarity - write testable components just like you do now with MonoBehaviours

In addition to testable code, you get extremely flexible code, and a radically different way of making things in Unity.

What we want to know is:

1. Is anyone else doing this?
   
   We have seen other posts related to DI and IOC, though these seemed to hit trouble and no frameworks emerged.
   
   
2. Does anyone want to see this on Github and get involved?
   
   We have seen a massive increase in productivity by using these patterns, and we feel confident that other people can too. The framework is incomplete, however, and a fair amount of work is required to finish it, including getting a nice Unity editor integration working.

Get it on Github https://github.com/banderous/Uniject.git

 

Seeing as every solution I've come across for unit testing in Unity doesn't cover DI or IOC, this would be invaluable to someone wanting to promote TDD inside their team - and the happy accident of portability sounds promising as well.

 

I use an IoC container that I wrote on my own...probably I will be publish it, but I need some feedback first

 

Definitely yes. I've came into game dev from enterprise software development, where SOLID principles is a must and lots of application are build around Composition Roots and other related DI mechanisms, It would be great to have such modularity in developing games in Unity3d.

 

I've been slowly pulling together an AOP framework for Unity. It allows me to write mocks for whatever I need to (so gryph666 it could help with your TDD), however the discipline of the approach described here will probably lead to better code in the end anyway. With aspects you can still be sloppy, as they allow you to inject/mock anything (although they are still a very useful tool for profiling, logging, error handling, etc).

What are the plans for this framework?

 

Out of curiousity how many actually test games in this manner?

 

Indeed, there's no reason games should be different to any other software systems.

It's impossible to say for sure but I suspect very few, and possibly none.

The question then, if games should be like other software, is why aren't people making them like other software? There are many reasons, but they all boil down to a cost benefit analysis; clearly the cost is too high.

Unity is a major contributor to this cost; it is a significant amount of work to write testable game code, unless we have a framework to help us.

The fact that very few people are doing this actually excites me, because it's an opportunity to make a big difference. The benefits of testability have been extensively documented, we just need to get the cost down.

I haven't touched it for a couple of weeks, Github seems the way to go. I should write some articles on it too, this seems a neglected topic.

 

Note that this is now on Github https://github.com/banderous/Uniject.git

 

I hate unit testing its just a waste of time....

 

Good for you, but this is not a discussion about whether unit testing is valuable. You should distill your wisdom into a research paper, it would certainly be ground breaking.

 

Yeah cause unneeded sarcasm helps the discussion.. Unit tests make you have to write more code for no real benefit. You wrote a framework that no one wants or will use way to go my good sir...

 

So are comments, design patterns, and variable names that have more than 2 characters!

 

do you use them? You seem to support the use of them...

 

Absolutely, here's a sample from a commercial project I'm working on right now:

Code (csharp):

1.  
2. // Create player data
3.         PlayerInGameData playerOneStats = GetPlayerData().startingStats;
4.         PlayerInGameData playerTwoStats = GetPlayerData().startingStats;
5.  
6.         // Move one - Single card
7.         GameControl.GetInstance().PlayCard(0,0);
8.         stats = GameControl.GetInstance().GetCurrentPlayerStats();
9.         if (stats.army != playerOneStats.army + 2) JNAUnit.Fail("Army was not increased correctly");
10.         if (stats.hitPoints != playerOneStats.hitPoints - 1) JNAUnit.Fail("HP was not decreased correctly");       
11.         if (stats.magic != playerOneStats.maxMagic - 4) JNAUnit.Fail("Mana was not decreased correctly");      
12.         GameControl.GetInstance().EndTurn();
13.        
14.         // Check armies clash
15.         stats = GameControl.GetInstance().GetCurrentPlayerStats();
16.         if (stats.hitPoints != playerTwoStats.hitPoints - 2) JNAUnit.Fail("Army did not decrease HP correctly");       
17.  

And another approach on another commerical project:

Code (csharp):

1.  
2.     public void TestSingleItemInteraction() {
3.         // Create a hammer in the scene
4.         Item hammer = ItemHelper.CreateItem("HAMMER");
5.        
6.         Inventory.GetInstance().AddItem(hammer);   
7.        
8.         // Pull hammer
9.         ItemManager.GetInstance().DoInteraction(ItemInteractionType.PULL, hammer);
10.        
11.         // Did we get two pieces?
12.         Assert.True (Inventory.GetInstance().Contains("BROKEN HAMMER"), "No broken hammer found");
13.         Assert.True (Inventory.GetInstance().Contains("HAMMER HEAD"), "No hammer head found");
14.         // Do we still have a hammer (we shouldn't)?
15.         Assert.True (!Inventory.GetInstance().Contains("HAMMER"), "Hammer still exists");
16.        
17.         // Cleanup
18.         ItemHelper.DestroyItem(hammer);
19.         ItemHelper.DestroyItem(Inventory.GetInstance().GetItem("HAMMER HEAD"));
20.     }
21.    
22.  

 

Most of that can be done without unit testing, debugging your code and simply testing features so what is your argument for the benefit of unity testing?

 

Lets take the adventure game example, as its quite a good one.

Manually running through all the possibilities of interactions would take about 1-2 hours. Writing the test cases took about 4.

However I changed the design quite substantially on this one. After the design change. I ran the tests and got a report of issues. It took about 5 seconds.

Doing the tests manually would also require a UI which was not part of the project. Even writing a simple UI which allows me to test the interactions would take an hour or two. It also means that I might discover errors that are UI errors not ItemEngine errors. Even though I dont care about the throw away UI I still need to fix them.

Furthermore this component is something that I own rights to and plan to take to the asset store. If I add a new feature in 6 months time I can ensure it doesn't break anything in 5 seconds, instead of 4 hours. That confidence is pretty important when you have a user base depending on your software.

- - -

Thats me a single dev, working on a tiny commercial project. Imagine you have 10 people working on your project. Or 100. How do you ensure that their changes dont break something?

- - -

There's also stuff that is a bit hard to test manually. How do I test that my items raise the correct events when they trigger their interactions? I'm not a c# class I can't see an event. If I'm going to write an event listener that records the events (Debug.Log, inspector, or whatever), then why don't I just write a test case instead....

- - -

As for the card game, its a bit rough and ready, but a set of tests which covers all the rules in the design doc was written on the train in about an hour. It's really not that arduous.

- - -

Finally, I do agree that not all projects need unit tests and there are some situations in games dev that don't seem well suited to unit testing.

 

This I understand especially if the whole game is coded from scratch, I see unit testing be invaluable but inside unity I don't see unit tsting needed

 

Why? Is unity some magic engine that automatically removes all chance for bugs?

I still have to write my own complex structures for things like AI, inventory, story system, etc. They're dynamic systems that are bound to have errors... completely independent of unity.

Your code on top of unity may be simple, but that doesn't mean everyone's code on top of unity is simple.

 

Wow, was that a direct shot at me? Interesting... Wasn't aware unit test are standard practice.. I dont support your opinion on unit testing and I get slammed.. Seems about right.. Its taboo to have a different opinion than others these days... Instead why not try to help me change my view and see the usefulness of Unit testing like JohnnyA did

 

It's not that you don't support my opinion.

It's because you decided to slam the original opinion:

Have all the opinion you want. I just find it funny that you criticized other people's opinion because their proof for why they like it doesn't suit your standards:

(testing features and debugging... yeah, that's what unit testing is! it's a technique for doing so)

Yet don't explain your evidence for why unity is a game changer when it comes to unit testing...

I repeat, is unity some magic system that removes all chance for bugs? Why is unit testing invaluable until unity comes into play?




Way to play the persecution card...

Is this some fantasy land where you're allowed to throw sarcasm around, but everyone else must stay serious?

 

I'm not getting into this, my first and last reply on this matter;

What proof?

I expressed an opinion

Since you marked something in bold I did to, did you notice I marked "I" bold? I said I felt that unit testing was a waste of time. let me reiterate I said, I feel unit testing is a waste of time and I don't like it, I gave my opinion and I got slammed. or did you not read this post.

After reading that I got snippy and replied as such

I was provoked yes but I was still wrong and should not have posted that.

perhaps your right, but when I was taught to unit testing through code it was boring and time consuming and I couldn't see how it was useful

I thought the question was rhetorical, but, if you must have an answer then, No! I don't recalling me saying it was either..

I thought so lets hope the jury agrees...

Absolutely not! I just failed to see where you were being sarcastic...

 

lol...

good job



oh, so you know... that was sarcasm. I don't want to offend or anything.

 

Interesting points of view by all.
I personally would not use this package, nor do I think (my opinion) unit testing is appropriate for games testing in general.
Why? Because the coverage it offers of the game state tree is too small (and narrow) to be cost effective in proportion to testing a whole game.
Game Engines on the other hand - yes I do.
Working with Unity to do Unit Testing or even OOP is very challenging so I think certainly the creation of this package is a good effort.
Now testing is a highly creative art and I believe this will be another tool in the hands of the apprentice or master, how useful it is, depends on the practitioner.

 

This goes beyond testing; structuring your code using dependency injection confers numerous benefits, even if you never write a single test. You get a modular, decoupled codebase that is flexible and easy to change.

I guarantee there will be countless Unity projects whose codebase has coagulated into an intractable mass of spaghetti. Many more will be wondering why, as their project has grown, their productivity has fallen lower and lower, and why they seem to spend more and more of their time using the debugger - and remember, every minute spent using a debugger is a minute wasted.

My conclusion is that Unity encourages, though does not mandate, terrible software engineering practices, largely due to its emphasis on getting something up and running as quickly as possible.

Leaving testability aside completely, here are my prime suspects:

1. We are encouraged to expose and rely on public variables

This is dangerous enough in itself, but it is compounded by the prefab system. Suddenly, any variable you expose becomes an implicit public interface that risks breaking something if changed, because it could be cemented into a prefab. Worse, there's *no way* to verify you've done so. Prefabs are like a huge, brittle plastercast of the implementation of your code at a given point in time.

2. No constructors

Constructors are a fundamental tenet of OOD; they force classes to explicitly state their dependencies and so avoid the existence of instances in a partially initialised state. Unity brings us an initialisation order nightmare, and the 'solution' is the hideous method of assigning an initialisation priority on a per script basis - yet another way to break our games in subtle, unverifiable and unpredictable ways.

The situation is insidious, because we are never forced to tackle our ever increasing tangle of dependencies; I can access any other script with a single static method call! Constructors force us to think about our dependencies.

Without the ability to mock out Unity it is, in fact, impossible, at least according to Google.

 

I must admit I thought this was just a unit testing framework for Unity, which alone, would have been of little interest to me.

Are you saying your package can enforce some sort of OOP cohesion in our projects? Is it an enabler of OOP? If so, I'd be interested to see some examples.

I can echo the experience had by all of the problems with prefabs and dependencies. OOP in Unity is highly interesting to work with to say the least. What I find even stranger is Unity seem pretty smug and dismissive about it. Perhaps because it will be the large, reused projects that will suffer most, and thats not the typical Unity audience. Idk.

 

I haven't sold it well; this is indeed not just a way of unit testing. It represents a radically different way of architecting your game. The ability to effortlessly unit test your code is but one of its objectives.

Anyone who has tried 'unit testing' using one of the existing Unity compatible testing frameworks, which integrate some form of XUnit into the editor, have not been unit testing at all. Such approaches would be unbelievably painful and utterly futile, and likely put people off unit testing for life, which might go some way to explaining the attitude seen previously in this thread.

It certainly cannot enforce it, though it is an enabler.

I think this would be a sensible next step for me. In hindsight, the facilitation of unit testing was the wrong angle to emphasise, since nobody is actually unit testing!

 

All immaturity aside...

Bless you, sir, for putting this together. I've been aching to use Ninject in my Unity3d project. You just made my day. Week. Month.

 

@Banderous:
I for one am extremely interested in this project. I would like to employ TDD in my own projects, as I find unit testing extremely helpful for CI and instant feedback if something I added/tweaked broke other seemingly non-related areas of my code. I also have tried using the Editor-driven testing frameworks and found them rather cumbersome. (Also do not want to pay to try out TestStar, and from the videos personally would not like to test my code the way they show.)

@Unit testing as a whole:
It really has to do with HOW you test. Arguing one way or the other if unit tests are helpful in one's game (or application...or website... for that matter) relies on testing properly. Simply throwing together a test for testing's sake, with 50 assertions in it does not necessarily mean it's a "good" test-- and having to step through a single test to find where it blew up is also not good practice. But does that mean unit testing as a whole is useless?: No.

Unfortunately the way Unity scripting is set up, and tightly coupling it with the UnityEngine does not allow for full coverage. Especially if you have singletons and/or Managers that exist on a global object-- as well as input/graphics/GUI-- which are prime non-TDD areas. But back to my original point: I am very interested in this approach and would find it very useful in my own practice versus all other options I've seen up to this point. Will be watching this on Github, cheers!

 

I hope you're not disappointed, it's still in skeletal form! You'll almost certainly have to extend it for the needs of your project. I suggest you fork it on Github and send me pull requests, it could use the help

 

Absolutely; scoping is a tricky area in out of the box Unity, which led us down the dark road towards statics. Ninject gives fine grained control over the scope of injected instances.

One feature we have found very useful is the ability to have 'per scene singletons'; to have the same instance injected up until a new scene is loaded, at which point the old instance is GCd and a new one created. This let us have an event manager without the risk of memory leaks that a true singleton entails. Naturally, none of the classes that use the event manager have any conception of the scoping going on.

Good to hear! I will be continuing to expand Uniject and the example project. Many challenges remain...

 

Work on Uniject is continuing, and I have written a post that explores a basic Uniject based project.

In the post we see how Uniject lets us:

*Write a TestableComponent that drives a bouncing spherical light
*Have Uniject create GameObject hierarchies automatically
*Load and verify an audio clip and physic material
*Test all of our code
*Do all of the above without opening the Unity editor!

Let me know what you think.

 

I'm looking into using Uniject. I wrote code for 5 other developers and across all 40 of our products now its very likely no matter what i do one of them breaks. I'd like to solve it by switching to a DI / IOC framework... I don't know if Uniject is right for us though because we still need a designer to jump in and make quite a bit of the software before our coders jump in to make specifications. My first impressions were that we couldn't make a scene in unity and have a designer (with no coding experience or very little) use uniject to make what they needed. Am i using this wrong from the get-go?

 

Thanks for the framework, I'll check it out.

Here's how I've described Unit Testing in the past, with some success.

Imagine you write a new class. As you write it, you test it i.e. use the debugger to make sure everything works as it should. Let's say you test 10 different things. Then a week later you modify the class and test it again, but this time you think of new scenarios/conditions/things that could go wrong, raising the number of tests to 15. Then two months later you again need to modify the class. When you modify it again, will you think of all 15 tests? And what if you are working with other programers. Will they too think of performing all those tests? What if they break something in YOUR class, will THEY get the blame or will YOU? Wouldn't it be great if those tests could be run automatically so you'd know when something breaks (or when someone breaks something)?

 

This looks great. I've been on the fence about applying TDD to my Unity work, but now I might be compelled to give it a shot.

Thanks for sharing!

 

@StephaneB7

This is basically what happens to us. We have a few different "button" like classes that get used in all of our products in roughly 8 different ways. I have no idea which one will break or how it will shatter the application when i need to change something about it. Button classes were taken out of our framework because of this. Of course, there's still 41 other classes i need to double, triple, and quadruple check anytime we need to update something on them.

I think Uniject may be a solution for us because of these.

 

Switching an existing codebase to Uniject will be no small undertaking. Nevertheless, if the code is here to stay then it may be a worthwhile investment, depending on the nature of the breakages; if they are 'look and feel' problems rather than runtime errors, no testability framework will make any difference.

I'd need to know a little more about your product to comment on your situation, but in general designers should still be responsible for creating all visual assets. The difference is that they should create a larger number of extremely simple 'leaf prefabs' rather than a smaller number of complex, brittle prefab hierarchies traditionally used.

Take, for example, something like a UI button. The visual background image of the button could be created as a prefab by a designer using whatever tools they care to use. The code driving the button would then inject the prefab as a constructor parameter and add it to its transform hierarchy. This is a step towards separating the logic that drives a UI from its visual representation.

This addresses one of the other major problems I've had with the prefab system; the impossibility of reusing one piece of a prefab as a part of another. Everything you add to a prefab is immediately coupled to everything else. Use Uniject and smash those monoliths into lots of small, reusable pieces.

@StephaneB7

A good summary of the value of unit tests as an investment vs the 'dead time' of manual verification.

I'd also like to emphasise the massive increase in productivity that comes from being able to write a line of code and immediately execute it - it keeps you in the zone. Hit CTRL-T and you have your answer in less than 2 seconds.

Consider the alternative ways we might get our line of code executed:

1. Context switch to editor, execute my 'unit tests' that actually involve running the entire game engine (10 seconds?)
2. Context switch to editor, play my game until I hit my code (Tens of seconds? minutes?)

Now consider how these costs get compounded. If there's a problem with my code, I will have to repeat my verification step at least once, and maybe more. If I have to manually play my game there's a very high chance I'll make a mistake or get distracted. I might have to attach the debugger, rinse and repeat. With Uniject you can debug in a vanilla monodevelop environment (although strictly you should be writing a test rather than debugging ).

I'd also like to add that Uniject gives you a lot of execution coverage almost 'for free'. The following integration test executes every line of our game logic in the bouncing spheres 'TestableCollisions' example.

Code (csharp):

1.  
2. [Test]
3. public void testExampleSpawnsObjects() {
4.     kernel.Get<TestableCollisions>();
5.     step(10);
6. }
7.  

 

Just want to say thank you for working on this -- I'm very much looking forward to checking this out. It gives me hope that the power of Unity can be combined with TDD and generally maintainable code.

 

I think that this is good idea to have ability to test your game scripts. Thank you

There is framework called "Fakes" that is integrated into Visual Studio 2011, it can mock static methods, private methods, constructors and not-virtual methods. Maybe it can facilitate testing in unity?

 

That looks like a pretty terrifying tool, designed for the gnarliest of legacy codebases. With Uniject you shouldn't need to mock static methods, etc because there aren't any.

 

Just a quick note about inspector variables breaking encapsulation. You can use the SerializeField attribute on private members to make them show up in the inspector.

Code (csharp):

1. // So instead of this
2. public float fooBar;
3.  
4. // Use this
5. [SerializeField]
6. private float fooBar;
7.  
8. // Allowing you to control access with properties again
9. public float FooBar {
10.     get {
11.         return fooBar;
12.     }
13.     protected set {
14.         fooBar = value;
15.     }
16. }

This approach adds a small bit of overhead to your variable declaration, but gives you back full control over the encapsulation of your variables.

Edit: Added C# example code for anyone looking for some pointers.

 

That's a good tip, certainly preferable to making variables public.

However fooBar still constitutes a public interface; it can be baked into a prefab or scene instance, so it is still violating encapsulation.

 

What's not to love about dependency injection?

Now having said that, i think this solution may take away a bit too much, in that you're swapping out an entire workflow for another:
1- Reliance on the resources folder, thus Unity won't automatically cull content you don't use.
2- Almost completely takes away the editor experience. Setting variables per object that are instantiated in code rather than as components uses what as a backing store?

I am torn though. Managing a dependency graph with drag&drop and manually mapping variables is a total nightmare, and every time I make a singleton, I vomit a little.

 

Broadly I agree; Uniject isn't a framework you can have just a little of, it's a totally different approach to building an App. It's based on the idea that testability is more important than any other aspect of software development, and it sacrifices a lot to achieve this.

The Editor workflow is gladly sacrificed because, in my experience, it does not scale to complex projects. A workflow that requires me to actually run and play my game to test my code is hopelessly inefficient. I've written about this in a different blog post, I think, but the engineering practices encouraged by Unity are hideous.

I use XML for all my configuration data. Storing configuration data in prefabs/instances is a nightmare; they're hopelessly brittle in the face of code change.

 

This:

is one of the reasons I decided not to use Uniject on my project. I think it's a nice framework, but it promotes a very engineering-centric approach to development, and I think more people would use it if there was a way to still allow usage of the editor. I believe one of the biggest strengths of Unity is its editor - someone with no coding experience can go in, tweak some variables, create some prefabs, and get something that looks good without writing a line. The impact this has on velocity cannot be underestimated.

However, I completely agree with this:

I rolled my own DI solution using vanilla Ninject and I've basically resigned myself to the fact that there is no getting around some of the inherit limitations if you still want to provide a decent editor experience. What I did was relegate most of my game logic to plain old C# classes that don't derive from MonoBehaviour. The behaviors themselves inject these objects and manipulate them - this lets me keep all the ugly stuff in behaviors, but still have a clean set of code that is doing the real work. Its set up pretty much like a 2-tiered application.

We haven't yet gotten very far in our project, so this could end up not scaling well for us, but it hasn't introduced too much overhead aside from just having the discipline to separate the functionality. On of the things that was done in Uniject that I think is invaluable is the mapping of the horrifying Unity static methods and static state to injectable interfaces so you can swap them out in unit tests.

One question though - what do you guys use for mocking? I haven't been able to get any of the major frameworks to play nice with Unity (EasyMock.NET, Moq, FakeItEasy, probably some others) so I've been manually writing mock classes (painful).

 

I've been using Moq. With Uniject Moq doesn't have to play nice with Unity since it doesn't know anything about it. The tests are run by NUnit outside of the Unity editor, and Moq is only required to mock plain old C Sharp interfaces.

Are you trying to run your mocking framework inside of the editor? Or mock a type in UnityEngine.dll? I can't imagine either of those going well, but if you're using plain old interfaces you should be able to mock away without issue.

 

I'm not mocking anything from unity itself. But in order to get the DLL to be included when Unity randomly decides to rebuild the monodevelop project I have to have Moq.DLL (or whatever) in my Assets folder (otherwise I'm re-adding a reference all the time). So if I stick one of the mocking frameworks in it I get an "Internal compiler error" and I can't build my project from the unity editor (the unit tests work fine from monodevelop though).

I want to figure out how to include a separate csproj in the solution generated by unity that contains my tests, but I haven't found a way to do it and it's really starting to infringe on my game development time So I've elected to remove a mocking framework.

I imagine if you're using Uniject to build your solutions outside of the unity editor you won't have these issues...are you doing something differently?

[edit] It seems like Unity has a problem with Castle.DynamicProxy, which most of the mocking frameworks use.

 

The answer is to create an entirely separate Test solution that contains your unit tests (and depends on your mocking framework) and references the CSProjects generated by Unity. This way you keep all your test paraphernalia out of your Assets folder, while your tests still benefit from the auto generated visual studio projects generated by Unity. Just right click your test solution, choose add existing project and add in the Unity generated projects.

This is the approach taken by Uniject, so you end up with a directory structure that looks like:

root/test/test.sln // references ../../unity/Assets/Assembly-CSharp.csproj, etc
root/unity/Assets/... // Nothing unusual in here

 

Ah - that makes sense. Kind of like "inversion of control" on your solutions. I'll give that a shot and report back.

 

This worked like a charm! Thanks for the advice. Incidentally, I'm also using Moq.

 

I'm very interested in Uniject. I'm come from TDD mindset and have always followed standards and best practices to write software.
I was shocked knowing that the game development practices do not highlight testing. I actively asked famous indie game devs (Jonathan Blow, Tommy Refenes ...etc.) about testing their games and they basically said that they only do build testing and that's about it!

I'm more than willing to even donate money to help you out with this project!

I have few things that I would like to ask:

1- Can you run the tests from the command line? Effectively this would be useful to hook up with a CI server such as Jenkins or Bamboo.
2- Do you have examples of complex interactions that are tested with your test framework? You only have a simple example on your website.
3- Can you test network behaviour as well? e.g. Loading an asset from a server then using the asset in a scene.
4- Does your test framework generate reports after the test has finished? what kind of formats for reports does it output? (i.e HTML, JSON, junit xml ...etc.)
5- Do you have documentation for using your test framework? It would be good to create a video showing how to your test framework could be used.
6- Does it support a specific Unity3d version? or works on all ?


Thanks! Looking forward to your response.
Cheers.