Official Unity Test Tools

Very interesting!

Thanks a lot

 

Very exciting news!

 

Thanks Tomek, this is very useful. It looks like the example unit tests only deal with manipulating basic primitives. Are there any example unit tests that create and check game objects (and their properties) in a scene? By this I mean unit tests written in C# and run through NUnit, not integration tests.

 

Hi,

First of all, thanks for sharing this amazing tool! I am really excited about this after seeing the screenshots. Are there any step-by-step instructions on how to run the tests in the example scene? I opened the AngryBots TestScene.unity, but I am not sure how I can bring up the TestRunner. Even more importantly, how would I add this to an existing application and run tests every now and then?

Ideally, I would like to include this into my continuous build system as well. Is there a way to include it in the build?

Thanks,
Ingo

 

Hi Tomek,

Thank you very much! I can now execute the integration tests that come with the AngryBots example. I've also found the UnityTestTools.pdf file. Looks like to run the tests on a continuous integration server, I'd run them in headless mode as described on page 13/14.

> Unity.exe PATH_TO_YOUR_PROJECT -batchmode -executeMethod BatchTestRunner.RunAllTests -testscene=IntegrationTestsExample

This generates an XML file which I can parse to determine if they passed/failed. Do you recommend any specific plugins for running Unity tests on CI systems? For Jenkins there seems to be at least Unity3dBuilder.

Do tests still have to be stripped from the compiled game? Or will they already not be part of the resulting application? I just want to make sure that none of the test dependencies increase the size of the released game.

Thanks,
Ingo

 

Very nice tool and more stable than TestStar...

Unfortunately it seems that the XML output can't be validate by the NUnit XSD
So my JenKins NUnit plugin don't want to display the test result...
The test-suite element is missing.

 

Well i have added this

Code (csharp):

1.  
2.   private void AddDefaultTestSuite(ResultSummarizer summaryResults)
3.         {
4.             xmlWriter.WriteStartElement("test-suite");
5.             xmlWriter.WriteAttributeString("type",
6.                                             "TestFixture");
7.             xmlWriter.WriteAttributeString("name",
8.                                             "ALLTests");
9.             xmlWriter.WriteAttributeString("description",
10.                                             "ALLTests");
11.             xmlWriter.WriteAttributeString("executed",
12.                                             "True");
13.             xmlWriter.WriteAttributeString("result",
14.                 summaryResults.Failures >0 ?  "Failure" : "Success");
15.  
16.             xmlWriter.WriteAttributeString("success",
17.                summaryResults.Failures > 0 ? "Failure" : "Success");
18.  
19.             xmlWriter.WriteAttributeString("time", "0.0");
20.  
21.             xmlWriter.WriteAttributeString("asserts", "0");
22.  
23.             xmlWriter.WriteStartElement("results");
24.         }
25.  

On InitializeXmlFile and i patched TerminateXmlFile()

Code (csharp):

1.  
2. private void TerminateXmlFile()
3.         {
4.             //Add by XL
5.                 xmlWriter.WriteEndElement(); // test-suite
6.                 xmlWriter.WriteEndElement(); // result
7.                
8.             //
9.                 xmlWriter.WriteEndElement(); // test-results
10.                 xmlWriter.WriteEndDocument();
11.                 xmlWriter.Flush();
12.  
13.                 xmlWriter.Close();
14.         }
15.  

And it's working now...

But adding the test-suite notion to organize the integration tests could be nice.

 

Hi xlarrode,

are you saying (1) you customized Unity Test Tools to fix the XML output and (2) the Jenkins plugin can now detect if the tests passed/failed?

Thanks,
Ingo

 

Yes exactly...

 

There is a problem when trying to build a WinPhone8 project because the integration test framework relies on System.Xml.XmlTextWriter, which is not present on this target platform.

Could perhaps switch to System.Linq.Xml but that would break the web target.

 

That's great!

I think the dependency should also be fine with XmlTextWriter being included in 4.3.3 now for WP8.

On a sidenote, we actually changed our codebase to use the Linq XML package so I wish Linq.Xml was added to WebPlayer for 4.3.3 instead of the missing System.Xml classes added to WP8

 

Any advice for testing custom editor code in isolation? For example with a custom PropertyDrawer, field info and attribute info are only available as read-only instance properties, which can't be set through the constructor of PropertyDrawer. And event data is given through the global current event property. So you can't easily mock that data either.

I guess I could try to create the testing class as a subclass of the class I'm testing, and test the protected methods of the class instead of the public ones. But is that a good idea? Or would you recommend a different approach?

 

Thanks for the offer Tomek. I guess it was a bit silly of me to ask for advice without posting any example code. But I'm a bit reluctant to post the code with which I'm having trouble automating the testing, because it's at the core of a product of which I'm not yet sure if I want to release it for free or not.

I'll try to refactor the code for testing the protected methods (passing the required data as parameters, in stead of having the method accessing global state directly).

If it turns out that I can't manage to figure it out on my own, I guess I might as well post the code here, because it won't be a really reliable product if I can't easily test it with future versions of Unity.

 

Hello!

I'm trying to make some unit tests.

I created a C# file in MonoDevelop:

using System;
using System.Threading;
using NUnit.Framework;

class TextProcessorTests {
}

But I run into an error:

The type or namespace 'NUnit' could not be found. Are you missing a using directive or an assembly reference?

There are no other errors. The sample tests are running properly, with their 'using NUnit.Framework' statement.
What am I missing?

UPDATE:

I noticed that the nunit-DLL:s are only available in Assembly-CSharp-Editor, not in Assembly-CSharp where my tests are located. Is this the problem? If so, how do I fix it?

 

Thanks, it is working! I put it in Assets/Editor/Tests/ .. Can the Editor folder be placed anywhere?
It doesn't really make sense Lets hope TDD becomes part of Unity in the future, and not just an asset.

Also, the PDF contains no instructions on how to run the tests in the terminal on OSX.

 

There is an error in the v1.1.1 batchmode integration test example in the documentation. The argument in the doc is "-testscene" but the BatchTestRunner looks for "-testscenes".

I have a Unity project with multiple scenes using Unity Test Tools v1.1.1. I'm trying to run my integration test scene with the following command...

> Unity -batchmode -projectPath /path/to/my/unity/project -executeMethod BatchTestRunner.RunIntegrationTests -testscenes=IntegrationTestScene

Application.LoadLevel(1) in BatchTestRunner.Start() appears to load only the first scene configured in the build settings rather than my IntegrationTestScene. I can see in Editor.log that this is true.

The following line in BatchTestRunner appears to have no effect. Debugging shows that the array of scenes it creates is correct, but that Application.LoadLevel() just ignores the new scenes array.

EditorBuildSettings.scenes = sceneList.Select (s => new EditorBuildSettingsScene (s, true)).ToArray ();

Any thoughts on a fix or workaround?
Thanks.

 

I'm using Unity for Mac. I've tried this on version 4.3.0f4 and 4.3.4f1. It behaves the same way in both versions. If I only want to run (in batch mode) my integration tests, I need to change my build settings so my IntegrationTestScene is first in the list and make it the only scene with its checkbox checked.

 

Hi,
I 've just updated the UnityTestTools package. ( i like the new GUI )
But now all the tests are waiting the unity editor to get the focus in order to play them.
It seems that you are making a pause ( i guess a time.scale =0) when the focus is lost...

How can i remove that ?


Thanks

PS : You should also put Icons.cs on Editor/ or WebPlayer won"'t build....

 

Importing Unity Test Tools makes building Windows Store Apps player fail with the following errors:

Will there by a version for Windows Store Apps with #defines and Metro dlls?

 

npruehs, we are working on it.

 

Hi! The Test Tools are a great upgrade. Thanks!

However, NUnit test inheritance does not work for me. When I'm trying to unit test interfaces, I create an abstract class testing the interface and a specialized class with just overridden instantiation code (override protected ISomething NewSomething() { return new Something(); }) for each class implementing the interface. As far as I can tell, it should be a pretty common practice, but it does not work for me in Unity.

When there is no other test in the specialized class, I get a pretty ugly exception every time I try to run the tests:
ArgumentOutOfRangeException: Argument is out of range.
System.Linq.Enumerable.ElementAt[UnitTestResult] (IEnumerable`1 source, Int32 index)
UnityTest.UnitTestView.FindTestResultByName (System.String name) (at Assets/UnityTestTools/UnitTesting/Editor/TestRunner/UnitTestView.cs:392)
UnityTest.UnitTestView.UpdateTestInfo (ITestResult result) (at Assets/UnityTestTools/UnitTesting/Editor/TestRunner/UnitTestView.cs:386)
UnityTest.UnitTestView+TestRunnerEventListener.TestFinished (ITestResult result) (at Assets/UnityTestTools/UnitTesting/Editor/TestRunner/UnitTestView.cs:547)
UnityTest.UnitTestRunner.TestRunnerCallbackList.TestFinished (ITestResult fullName) (at Assets/UnityTestTools/UnitTesting/Editor/TestRunner/TestRunnerCallbackList.cs:22)
UnityTest.NUnitTestEngine+TestRunnerEventListener.TestFinished (NUnit.Core.TestResult result) (at Assets/UnityTestTools/UnitTesting/Editor/TestRunner/NUnitTestEngine.cs:129)
NUnit.Core.TestMethod.Run (EventListener listener, ITestFilter filter)
NUnit.Core.TestSuite.RunAllTests (NUnit.Core.TestResult suiteResult, EventListener listener, ITestFilter filter)
NUnit.Core.TestSuite.RunSuite (EventListener listener, ITestFilter filter)
NUnit.Core.TestSuite.RunSuiteInContext (EventListener listener, ITestFilter filter)
NUnit.Core.TestSuite.Run (EventListener listener, ITestFilter filter)
NUnit.Core.TestFixture.Run (EventListener listener, ITestFilter filter)
NUnit.Core.TestSuite.RunAllTests (NUnit.Core.TestResult suiteResult, EventListener listener, ITestFilter filter)
NUnit.Core.TestSuite.RunSuite (EventListener listener, ITestFilter filter)
NUnit.Core.TestSuite.RunSuiteInContext (EventListener listener, ITestFilter filter)
NUnit.Core.TestSuite.Run (EventListener listener, ITestFilter filter)
NUnit.Core.TestSuite.RunAllTests (NUnit.Core.TestResult suiteResult, EventListener listener, ITestFilter filter)
NUnit.Core.TestSuite.RunSuite (EventListener listener, ITestFilter filter)
NUnit.Core.TestSuite.RunSuiteInContext (EventListener listener, ITestFilter filter)
NUnit.Core.TestSuite.Run (EventListener listener, ITestFilter filter)
NUnit.Core.TestSuite.RunAllTests (NUnit.Core.TestResult suiteResult, EventListener listener, ITestFilter filter)
NUnit.Core.TestSuite.RunSuite (EventListener listener, ITestFilter filter)
NUnit.Core.TestSuite.RunSuiteInContext (EventListener listener, ITestFilter filter)
NUnit.Core.TestSuite.Run (EventListener listener, ITestFilter filter)
NUnit.Core.TestSuite.RunAllTests (NUnit.Core.TestResult suiteResult, EventListener listener, ITestFilter filter)
NUnit.Core.TestSuite.RunSuite (EventListener listener, ITestFilter filter)
NUnit.Core.TestSuite.RunSuiteInContext (EventListener listener, ITestFilter filter)
NUnit.Core.TestSuite.Run (EventListener listener, ITestFilter filter)
NUnit.Core.TestSuite.RunAllTests (NUnit.Core.TestResult suiteResult, EventListener listener, ITestFilter filter)
NUnit.Core.TestSuite.RunSuite (EventListener listener, ITestFilter filter)
NUnit.Core.TestSuite.RunSuiteInContext (EventListener listener, ITestFilter filter)
NUnit.Core.TestSuite.Run (EventListener listener, ITestFilter filter)
UnityTest.NUnitTestEngine.ExecuteTestSuite (NUnit.Core.TestSuite suite, ITestRunnerCallback testRunnerEventListener, ITestFilter filter) (at Assets/UnityTestTools/UnitTesting/Editor/TestRunner/NUnitTestEngine.cs:93)
UnityTest.NUnitTestEngine.RunTests (System.String[] tests, ITestRunnerCallback testRunnerEventListener) (at Assets/UnityTestTools/UnitTesting/Editor/TestRunner/NUnitTestEngine.cs:38)
UnityTest.UnitTestView.StartTestRun () (at Assets/UnityTestTools/UnitTesting/Editor/TestRunner/UnitTestView.cs:462)
UnityEngine.Debug:LogException(Exception)
UnityTest.UnitTestView:StartTestRun() (at Assets/UnityTestTools/UnitTesting/Editor/TestRunner/UnitTestView.cs:468)
UnityTest.UnitTestView:Update() (at Assets/UnityTestTools/UnitTesting/Editor/TestRunner/UnitTestView.cs:400)
UnityEditor.EditorApplication:Internal_CallUpdateFunctions()


If I add some test to the specialized class, the exception goes away and the test runs OK. BUT still none of the inherited tests is executed.

Has anyone else experienced this? Or is the inheritance working for you?

Thanks for reply,
Adam

 

I'm on Windows 8 Pro, 64 bit, using Unity 4.3.3f1, Test Tools 1.1.1.

Well, when I have tried to reproduce the exception in a fresh project just with Test Tools installed, I didn't get the exception... And now when I have reloaded the old project, it does no appear even if I remove the additional tests. Strange... (The exception appeared when there was no TestOtherStuff() in the code below, but disappeared when this was uncommented.)

However, I'm more concerned about the inheritance of the test methods. Should they work this way? (I have each class or interface in a separate file.)

Code (csharp):

1. public interface ISomething
2. {
3.     int ReturnOne();
4. }
5.  
6. public abstract class TestISomething
7. {
8.     public abstract ISomething NewSomething();
9.  
10.     [Test]
11.     void TestReturnOne()
12.     {
13.         ISomething s = NewSomething();
14.         Assert.That(s.ReturnOne(), Is.EqualTo(1));
15.     }
16. }
17.  
18. public class Something : ISomething
19. {
20.     public int ReturnOne() { return 1; }
21.     ... other stuff
22. }
23.  
24. public class TestSomething : TestISomething
25. {
26.     public override ISomething NewSomething() { return new Something(); }
27.  
28.      [Test]
29.      public void TestOtherStuff() { ... }
30. }

From what I have read, the NUnit should execute both TestReturnOne() and TestOtherStuff() on TestSomething. But only TestOtherStuff() is actually executed. TestReturnOne() does not produce any output to console, unit test runner, or XML.

My main question is: Do I have something wrong? If yes, what's the best way to test multiple classes with the same interface without code duplication?

Thanks!

 

It states that it requires 4.0.0, but I am getting a bunch of errors on 4.0.1 (to begin with there where two which I fixed, but after that there where about 26 new compile errors). An example of one of the compile errors are:

Assets/UnityTestTools/Assertions/Editor/AssertionComponentEditor.cs(38,59): error CS0103: The name `target' does not exist in the current context

or

Assets/UnityTestTools/Assertions/Editor/AssertionComponentEditor.cs(48,33): error CS1502: The best overloaded method match for `UnityTest.AssertionComponentEditor.DrawOptionsForAfterPeriodOfTime(UnityTest.AssertionComponent)' has some invalid arguments

 

I'm having the same issue. The only way to get the integration tests running in batch mode is by manually selecting *only* the integration scenes in the editor build settings. Tested on OS X and Windows (Unity 4.3.4f1 and Unity Test Tools 1.1.1). Is there any workaround?

No problem with unit tests though. Awesome tools, by the way. Thanks for sharing.

 

I've been having an issue using Unity 4.3.4f1 and the Unity Test Tools. I ran into this initially trying to integrate the tools into my own project, but have tried with a new project after just importing the Unity Test Tools and running the sample tests and am getting the same error.

When running the Unit Tests through MonoDevelop, I receive the error for C# tests:

Code (csharp):

1. Internal error
2. FileLoadException: Could not load file or assembly 'nunit.framework, Version=2.6.2.0, Culture=neutral, PublicKeyToken=93811cf430eaf12a' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference.

Boo tests give the error:

Code (csharp):

1. TestFixtureSetUp failed

And UnityScript tests crash MonoDevelop when trying to access their failed output.

This is using Windows 7 with Unity Free. I have tried cleaning up my Mono/Unity installation. Any ideas on what might be going on here?

 

I should also note here that the tests seem to run fine when run through the Unit Test Runner inside of Unity itself, just not through MonoDevelop.

 

Seems like a few of us are having trouble downloading from the Asset Store. Can anyone upload this package?

 

I guys,

I try to download the free version of unity, but every time I try to start the download.. gives me error 404 not found .. what happens?

http://unity3d.com/unity/download

This is the link

 

The code I'm using are the examples that came with the Unity Test Tools package. A new project, just importing the Unity Test Tools gives me these errors. There are no references to UnityEngine or UnityEditor in my tests.

I have had previous projects work just fine with MonoDevelop, Unit Testing and the Unity Test Tools. This has only been an issue on a clean install of Unity 4.3.4f1 on a new machine.

 

Is it possible to run unit tests, during batch mode, without quitting the Editor after the tests are done? Also, is it possible to know when the tests are done (as they are executed asynchronously)?

Our current issue is that we have Jenkins set to run a custom builder for each of our projects and we wanted to run tests every time a new deploy occurs. What we've done so far was to set build steps to call both UnityTest.UnitTestView.RunAllTestsBatch and BatchTestRunner.RunIntegrationTests. It is really weird to have the Editor open thrice for a single build to run as both of the calls will quit the editor as soon tests are done.

What we want to achieve:
- Call our custom deploy method from batch mode: ProjectBuilder.CIDeploy (it will handle loading a lot of environment variables).
- If there is an environment variable set to run unit tests - and integration tests - call them.
- When tests are done, check if any of them failed. If so, exit the editor with some error code.
- If all tests passed, deploy the player and perform post build tasks.

This way the editor would be opened once to run tests and deploy the player, saving a lot of time.

 

Is there any good video tutorial for using this tool?