Official Unity Test Tools

Well, it's submitted! I got the documentation in Github but ran out of time for the source, asset store resources, etc. That'll probably now have to wait until after the UK Develop Conference next week (anyone here going?)

I also worried that I was spamming and/or commandeering this thread so I've spawned a separate one here for CUTTE.

As noted over there, contributions most eagerly sought. (caveat my finding a contributors agreement!)

HTH, Rupert.

 

I started a convo with Tomek, but it occurs to me that this may be something that other people have encountered and solved. Has anyone tried tying in the unit testing or integration testing pieces with code coverage tools? I almost have the unit tests going thru mono (though I'm still having some issues), but I'm worried that having something similar for the integration tests may be just plain infeasible.

 

Not tried but would love both (or either) to be possible!

Obvious ideal being that they can generate standard coverage report data file that can be graphed, used for highlighting retrograde steps, refusing integrations, etc. by build servers or whatever. (but anything better than nothing, huh)

How does coverage usually work in CLR environment? If it's hooks in the VM (as I believe is used in some other platforms), I guess the question is whether they're already or Tomek can get them exposed for use?

If it's something higher level (e.g. extra IL instructions by pre/post processing before running), then maybe there's no need.

Enough hypothesizing in the dark from me ;-)

 

I'd like to offer 2 tiny patches for UTT:

1. Remove restriction that AssertionComponentEditor doesn't provide GUI for certain Comparer/ActionBase subclasses members (Vector3, etc).

I.e. Create a Comparer with a public Vector3 member and it currently isn't shown in the GUI since typeof(Vector3).IsSerializable == false. The test that does this seems superfluous. Here's a gist...

https://gist.github.com/Arakade/749a2d43e62dcf8beee2

Removing doesn't seem to have caused any problems so far but, if it did, a method that properly checks Unity serializability could be written by checking both IsSerializable and all of the special cases listed http://docs.unity3d.com/ScriptReference/SerializeField.html

2. AssertionComponentEditor should includeChildren so Comparer/ActionBase subclasses can include Arrays, etc.
Here's another gist...
http://forum.unity3d.com/threads/unity-test-tools.218287/

HTH, Rupert.

p.s. CUTTE got declined from the AssetStore because I included the BSD 3 license. I'm awaiting their response on how to free-license per the Provider terms section 6.2. Once fixed, I'll get the source and these patches into the GitHub repo.

p.p.s. <funny> You know you've been testing enough when you typo Assert Store I wonder if Tomek's next work for Unity will be a new store where we sell truths! </funny>

 

unity 4.5.1 import has this error

Internal compiler error. See the console log for more information. output was:
Unhandled Exception: System.Reflection.ReflectionTypeLoadException: The classes in the module cannot be loaded.
at (wrapper managed-to-native) System.Reflection.Assembly:GetTypes (bool)
at System.Reflection.Assembly.GetTypes () [0x00000] in <filename unknown>:0
at Mono.CSharp.RootNamespace.ComputeNamespaces (System.Reflection.Assembly assembly, System.Type extensionType) [0x00000] in <filename unknown>:0
at Mono.CSharp.RootNamespace.ComputeNamespace (Mono.CSharp.CompilerContext ctx, System.Type extensionType) [0x00000] in <filename unknown>:0
at Mono.CSharp.GlobalRootNamespace.ComputeNamespaces (Mono.CSharp.CompilerContext ctx) [0x00000] in <filename unknown>:0
at Mono.CSharp.Driver.LoadReferences () [0x00000] in <filename unknown>:0
at Mono.CSharp.Driver.Compile () [0x00000] in <filename unknown>:0
at Mono.CSharp.Driver.Main (System.String[] args) [0x00000] in <filename unknown>:0

 

Starting to use Unity Test Tools and it's looking very promising!

However one problem had me tearing my hair out for a few hours. I was calling IntegrationTest.Pass but the test would carry on and timeout. Eventually I realized it has something to do with Application.RegisterLogCallback, which is used by Test Tools, but also by ScriptInspector2 and PlayMaker both in my project.

Apparently only one script can register this callback at a time:
http://feedback.unity3d.com/suggest...gisterlogcallback-to-allow-multiple-callbacks

But I'm not sure what the rules are. Sometimes Test Tools works fine with the other tools open... maybe it grabbed the callback first... or last...?

Anyway, I'm investigating Test Tools to replace Test Star which doesn't seem to be maintained any more. I like the design of Test Tools and I love the ability to run in batch mode. However, I'm using it to test PlayMaker, so the RegisterLogCallback conflict is worrying... I can work around it, but hacking a test environment just feels like starting off on the wrong foot

Maybe you have some pull with the core unity team to bump the feature request up in priority? With the growth of the Asset Store lots of editor tools are going to be fighting over RegisterLogCallback and a lot of hair will be pulled out!

 

I'm getting this error testing on Metro with Unity 4.2.2:

 

I hear you! It would be great if you could get this working. Being able to quickly run integration tests across all platforms is VERY useful. BTW, it works fine on WP8.

 

I found a workaround: Make a new layout and add the Unity Test Tools to the layout first - it seems that the first script to grab the callback keeps it... at least in 4.2 so far...

But obviously a real solution would be better

 

Is there a recommended way to load levels in integration tests? Right now it breaks the test runner - the dialog hangs and can't be cancelled. I imagine because the test it was running was unloaded when the new level was loaded... I tried DontDestroyOnLoad on the test object, which keeps it around, but still hangs the test runner...

 

@Tomek Paszek

Internal compiler error. See the console log for more information. output was:
Unhandled Exception: System.Reflection.ReflectionTypeLoadException: The classes in the module cannot be loaded.
at (wrapper managed-to-native) System.Reflection.Assembly:GetTypes (bool)
at System.Reflection.Assembly.GetTypes () [0x00000] in <filename unknown>:0
at Mono.CSharp.RootNamespace.ComputeNamespaces (System.Reflection.Assembly assembly, System.Type extensionType) [0x00000] in <filename unknown>:0
at Mono.CSharp.RootNamespace.ComputeNamespace (Mono.CSharp.CompilerContext ctx, System.Type extensionType) [0x00000] in <filename unknown>:0
at Mono.CSharp.GlobalRootNamespace.ComputeNamespaces (Mono.CSharp.CompilerContext ctx) [0x00000] in <filename unknown>:0
at Mono.CSharp.Driver.LoadReferences () [0x00000] in <filename unknown>:0
at Mono.CSharp.Driver.Compile () [0x00000] in <filename unknown>:0
at Mono.CSharp.Driver.Main (System.String[] args) [0x00000] in <filename unknown>:0


I konw why has this error .

beasuse I use a unity plugin . https://sites.google.com/site/apshaiarts/

when I delete this plugin all is OK

 

@Tomek Paszek

I will delete CodeGuard obfuscation Plugin
Because I find a better one http://www.4infinity.com.hk/tutorial/code_obfuscation

嘿嘿。

 

Using Platform Runner with BB10 I get this error:

Error: Bar file must have .bar extension: IntegrationTests

UnityEditor.BuildPipeline:BuildPlayer(String[], String, BuildTarget, BuildOptions)
UnityTest.IntegrationTests.PlatformRunner:BuildAndRunInPlayer(BuildTarget, String[], String, String) (at Assets/UnityTestTools/IntegrationTestsFramework/TestRunner/Editor/PlatformRunner/PlatformRunner.cs:58)
UnityTest.IntegrationTests.PlatformRunnerSettingsWindow:OnGUI() (at Assets/UnityTestTools/IntegrationTestsFramework/TestRunner/Editor/PlatformRunner/PlatformRunnerSettingsWindow.cs:83)
UnityEditor.DockArea:OnGUI()

 

Cool - thanks!

 

I'm trying to write test scripts that use IntegrationTest.Assert.
I'm setting up a bunch of Asserts in OnEnable and a bunch in Update, but only the first set of asserts runs.
Right now it only runs the OnEnable asserts. If I comment out the OnEnable tests it runs the Update asserts.
How do I tell the test runner to check asserts in both OnEnable and Update?

EDIT: Some code if it helps:

Code (CSharp):

1. using System;
2. using UnityEngine;
3. using System.Collections;
4.  
5. namespace HutongGames.PlayMaker.Tests
6. {
7.     [IntegrationTest.SucceedWithAssertions]
8.     public class Test_SetProperty : MonoBehaviour
9.     {
10.         void OnEnable()
11.         {
12.             TestInitialState();
13.         }
14.  
15.         void Update()
16.         {
17.             DoTest();
18.         }
19.  
20.         void TestInitialState()
21.         {
22.             var target = GetComponent<CallMethodTests>();
23.             IntegrationTest.Assert(gameObject, target != null, "Component not found!");
24.             target.TestDefaultValues();
25.         }
26.  
27.         void DoTest()
28.         {
29.             var target = GetComponent<CallMethodTests>();
30.             IntegrationTest.Assert(gameObject, target != null, "Component not found!");
31.             target.TestValues();
32.         }
33.     }
34. }

TestDefaultValues and TestValues use IntegrationTest.Asserts

 

If I select a child of the test object and hit "run selected test", I get this error and the test runner hangs:

KeyNotFoundException: The given key was not present in the dictionary.
System.Collections.Generic.Dictionary`2[UnityTest.ITestComponent,System.Collections.Generic.HashSet`1[UnityTest.ITestComponent]].get_Item (ITestComponent key) (at /Applications/buildAgent/work/c514da0c8183631c/mcs/class/corlib/System.Collections.Generic/Dictionary.cs:150)
UnityTest.IntegrationTestRunner.IntegrationTestsProvider.FindInnerTestGroup (ITestComponent group) (at Assets/UnityTestTools/IntegrationTestsFramework/TestRunner/IntegrationTestsProvider.cs:87)
UnityTest.IntegrationTestRunner.IntegrationTestsProvider..ctor (IEnumerable`1 tests) (at Assets/UnityTestTools/IntegrationTestsFramework/TestRunner/IntegrationTestsProvider.cs:28)
UnityTest.TestRunner.InitRunner (System.Collections.Generic.List`1 tests, System.Collections.Generic.List`1 dynamicTestsToRun) (at Assets/UnityTestTools/IntegrationTestsFramework/TestRunner/TestRunner.cs:110)
UnityTest.IntegrationTestsRunnerWindow.Update () (at Assets/UnityTestTools/IntegrationTestsFramework/TestRunner/Editor/IntegrationTestsRunnerWindow.cs:250)
System.Reflection.MonoMethod.Invoke (System.Object obj, BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) (at /Applications/buildAgent/work/c514da0c8183631c/mcs/class/corlib/System.Reflection/MonoMethod.cs:222)
Rethrow as TargetInvocationException: Exception has been thrown by the target of an invocation.
System.Reflection.MonoMethod.Invoke (System.Object obj, BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) (at /Applications/buildAgent/work/c514da0c8183631c/mcs/class/corlib/System.Reflection/MonoMethod.cs:232)
System.Reflection.MethodBase.Invoke (System.Object obj, System.Object[] parameters) (at /Applications/buildAgent/work/c514da0c8183631c/mcs/class/corlib/System.Reflection/MethodBase.cs:115)
UnityEditor.HostView.Invoke (System.String methodName, System.Object obj) (at C:/BuildAgent/work/ea95e74f6e5f192d/Editor/Mono/GUI/DockArea.cs:234)
UnityEditor.HostView.Invoke (System.String methodName) (at C:/BuildAgent/work/ea95e74f6e5f192d/Editor/Mono/GUI/DockArea.cs:227)
UnityEditor.HostView.SendUpdate () (at C:/BuildAgent/work/ea95e74f6e5f192d/Editor/Mono/GUI/DockArea.cs:284)
UnityEditor.EditorApplication.Internal_CallUpdateFunctions () (at C:/BuildAgent/work/ea95e74f6e5f192d/Editor/MonoGenerated/Editor/EditorApplication.cs:227)

 

Does the latest version require Unity Pro for mobile (Android/iOS) intentionally?
There are references to .NET Sockets in non-Editor code meaning Unity Indie mobile won't compile one's project if UTT is present.
I'm hopping this was accidental mistake
Thanks, Rupert.

 

I have a similar problem building for WP8, where Sockets are not supported at all!

 

I think the auto-discovery of tests in the project might need to be a bit more discerning...

I made a simple Test behaviour (just to test something, not to write a test) and immediately got these errors in the console:

Assets/UnityTestTools/UnitTesting/Editor/NUnit/Renderer/UnitTestRendererLine.cs(29,46): error CS1061: Type `Test' does not contain a definition for `TestName' and no extension method `TestName' of type `Test' could be found (are you missing a using directive or an assembly reference?)

Assets/UnityTestTools/UnitTesting/Editor/NUnit/Renderer/UnitTestRendererLine.cs(30,50): error CS1061: Type `Test' does not contain a definition for `TestName' and no extension method `TestName' of type `Test' could be found (are you missing a using directive or an assembly reference?)

Assets/UnityTestTools/UnitTesting/Editor/NUnit/Renderer/UnitTestRendererLine.cs(31,46): error CS1061: Type `Test' does not contain a definition for `TestName' and no extension method `TestName' of type `Test' could be found (are you missing a using directive or an assembly reference?)

...

 

What about assertions in code? To simplify my example, this passes:

Code (CSharp):

1. public class MyCodedTest: MonoBehaviour
2. {
3.     void OnEnable ()
4.     {
5.         IntegrationTest.Assert(gameObject, true);
6.     }
7.    
8.     void Update ()
9.     {
10.         IntegrationTest.Assert(gameObject, false);
11.     }
12. }

Some tests are easier to write with IntegrationTest.Assert instead of making an AssertionComponent etc.

Do you plan to support that workflow? Or is it already possible somehow...?

Or perhaps if the AssertionComponent had the ability to call a method instead of just comparing properties...

 

I was running into problems with NUnit plugin and resultFilePath variable in Jenkins because of invalid xml, turned out the xml file wasn't being replaced but overwritten leaving old data (read: larger previous files) intact.
Minor fix in XmlResultWriter.cs creates a new file if it exists then it overwrites the file completely:

Code (CSharp):

1.  
2. diff --git a/Assets/UnityTestTools/Common/ResultWriter/XmlResultWriter.cs b/Assets/UnityTestTools/Common/ResultWriter/XmlResultWriter.cs
3. index abaedb5..326f7ab 100644
4. --- a/Assets/UnityTestTools/Common/ResultWriter/XmlResultWriter.cs
5. +++ b/Assets/UnityTestTools/Common/ResultWriter/XmlResultWriter.cs
6. @@ -316,11 +316,7 @@ namespace UnityTest
7.              {
8.                  var path = System.IO.Path.Combine (resultDestiantion, resultFileName);
9.                  Debug.Log ("Saving results in " + path);
10. -                using (var fs = System.IO.File.OpenWrite (path))
11. -                using (var sw = new System.IO.StreamWriter (fs, Encoding.UTF8))
12. -                {
13. -                    sw.Write (GetTestResult ());
14. -                }
15. +                System.IO.File.WriteAllText(path, GetTestResult(), Encoding.UTF8);
16.              }
17.              catch (Exception e)
18.              {
19.  

 

Have done just that

 

Congrats on getting it open-sourced! Really happy -- thanks! I'm a bit busy with LudumDare atm but hope to offer my CUTTE bits (and scrap CUTTE) once LD's finished
To save a round of pull-request & fix, for an Integration Test Editor Extension helper, is that best to go in a new directory inside IntegrationTestFramework ?

Thanks, Rupert.

 

Hi there, I'm getting the exact same issue you had. My jenkins doesn't recognize the xml report because it thinks the xml is not formatted in a junit format that jenkins understand:



I validated the outputted xml for my tests against the junit.xsd format used by jenkins and it shows that the output is not a valid junit xml format.



I was wondering if you can share your XmlResultWriter.cs file or explain the changes you have done.
P.S: I'm using Unity 4.5.3f and Jenkins 1.576

P.P.S: I would really appreciate any help as I'm currently stuck at this step (getting jenkins to parse xml results generated from unity test tools).

 

huge post there man... check the open source repository has a pull request with my changes

 

@ikriz, thanks for the reply man. I will take a look at your pull request, cheers!

 

Sorry for the "bump" - i believe most coverage frameworks use the .NET profiling API - http://msdn.microsoft.com/en-us/library/bb384493(v=vs.110).aspx

This API allows to do all sorts of weird and wonderful things like manipulating IL code at runtime.

I think this is also how some test mocking frameworks are built (e.g: TypeMock Isolator).

 

Welcome back @Tomek Paszek - hope you had a good hol!

Clarification: it prevents building!
Steps to reproduce:

1. Install indie
2. add test tools
   
3. (optional: create test, run test = fine)
4. create simple scene
5. Build exe (windows, Web player, etc) = fine.
6. switch to android
7. (optional: Tests run in editor (IIRC))
8. Build apk = failure.

HTH

 

Not yet. Hope to address at weekend.
In the meantime, here's the documentation:
https://github.com/CUTTE/CUTTE/wiki#integrationtestsequence

 

Cool

 

If you haven't done this already, I have a start from my own workspace if that's be helpful? I'll clone the repo, push and offer in case so. I'll admit it was a potentially overly simple fix and one that may only work automatically on Unity 4.5+...

I noticed the appropriate sections have code checking UNITY_METRO. Adding extra conditions there seems to suffice.

I say "automatically" since the #define for Unity Pro was only introduced then. One could instruct Pro users of prior versions to set the define in the build settings for Android and iOS if they want net sending of tests. (seems a reasonable workaround for legacy support.)

HTH, Rupert.

 

Pull request offered for the Indie networking dependency (Issue #1).

 

Which is the roadmap for the test tools? Are you guys planning to add some kind of automatic Dependency Injection framework? Seems to me that is one of the big things left. I saw the blog posts about doing DI by hand, and though it looks nice for short examples, it becomes unmanageable for mid to large projects.

 

There is Zenject and StrangeIOC for DI, both available from the Asset Store. There is also TinyIOC on Github for barebones, really lightweight IOC.

Those are confirmed to work on desktop and most mobile (no clue about Blackberry) Other frameworks (Ninject) may or may not work on mobile.

 

Can you add unity-version and build-platform to the Environment attributes in the test results xml?

I'm building a final report that combines all xml files into a single html file, so it's useful to have this info in the xml reports... BTW, it would be cool in the test tools shipped with an example XSLT/Stylesheet for nicely formatted results.

Locally I've added these to XmlResultsWriter.WriteEnvironment:

{"unity-version", Application.unityVersion},
{"build-platform", Application.platform.ToString()},

For some reason Application.platform always returns "WindowsEditor", even thought the test runs on the correct platform...? Any idea how to report the correct platform?

 

Hey,

quick question (not sure if it was asked before): should the package allow importing only some of its assets?

It seems that there's some degree of coupling between its different parts.

For example, in case i want to use only the unit testing capabilities, importing the Unit Testing folder, along with the Common folder did not work (TestResult class is defined inside IntegrationTestFramework). Adding that folder on its own also did not work, since it relies on the Assertion Component class.

Basically i had to import the whole thing. Is there any plan to re-structure to allow importing only some functionality ?

 

Hi,

I'm using this tool with Unity Free. But it gives me error when I was building iOS project.
"System.Net.Sockets is not supported"

I found several files are using System.Net.Sockets in Unity Test Tools. Is there a workaround, or I have to use Unity Pro?

 

@hyperspace_x hi. Yes, it will again support free -- it was a mistake that it stopped. The fix is in the src repository. I imagine @Tomek Paszek will push a new build soon to fix. Until then, you probably need to grab from the repo.

HTH, Rupert.