Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Official Unity Test Tools

Discussion in 'Testing & Automation' started by Tomek-Paszek, Dec 18, 2013.

  1. DigitalGlass

    DigitalGlass

    Joined:
    May 12, 2009
    Posts:
    88
    My integration tests are returning exit code 2 for the examples, even after I removed the failed cases. It's returning 0 after I removed the ignored tests. Adding them back returns 2.

    Is this the expected results? I'm not sure how I would continue my CI build if that's the case.
     
  2. Tomek-Paszek

    Tomek-Paszek

    Unity Technologies

    Joined:
    Nov 13, 2012
    Posts:
    116
    @DigitalGlass
    It's a bug. I will push a fix to the BitBucket repo later today and update the Asset Store package later this week.
     
  3. DigitalGlass

    DigitalGlass

    Joined:
    May 12, 2009
    Posts:
    88
    Great everything is working now!
     
  4. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,562
    Is it planned to add the test's output (from console) to each test?

    Currently, the messages of a test (whatever these may be) are only added when a test fails.
    Also, i am not sure what NUnit considers to be test output (console writeline?)

    Comparing the test runner to other runners (like Resharper), this is standard behaviour (see attached image)
     

    Attached Files:

  5. Tomek-Paszek

    Tomek-Paszek

    Unity Technologies

    Joined:
    Nov 13, 2012
    Posts:
    116
    @liortal This the log information is not coming back from nUnit. I'm not sure if is not supported by nUnit, requires extra setup when initialising nUnit or it's just Unity not liking stdout stream. Feel free to make a request in the Issue Tracker
     
  6. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,562
    @Tomek Paszek Looking at the NUnit code (v3.0, may be different from the version you use)
    NUnit is overriding the Console class's stream to be able to get all text written to it (for both STDOUT and STDERR).

    upload_2014-10-1_13-29-36.png

    If the version bundled with UTT works the same, you can register a log callback and delegate all Debug.Log calls to Console.WriteLine, which is already handled.

    I haven't tried that but theoretically it can work. I will give a look at raising a request for that (unless this solution works and i can submit a pull request for that, if acceptable).
     
  7. Tomek-Paszek

    Tomek-Paszek

    Unity Technologies

    Joined:
    Nov 13, 2012
    Posts:
    116
    @liortal
    The API is different but I got it working. Check the repo. It didn't make it to this Asset Store update though. Can I ask what scenarios you want to use it for? Stdout is not really first class citizen in Unity :)
     
  8. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,562
    It's not a must, but it's nice to have purely for "organizational" purposes - suppose i'm running a test that generates some logging, i'd like to have all the output under the test's output.
     
  9. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,562
    Awesome of you to add that so briefly :) thanx
     
  10. makeshiftwings

    makeshiftwings

    Joined:
    May 28, 2011
    Posts:
    3,350
    So that "as soon as I can" from a month ago about fixing the broken Sockets dependency... how soon is that?
     
  11. makeshiftwings

    makeshiftwings

    Joined:
    May 28, 2011
    Posts:
    3,350
    Also, I just got the repo version, and the Sockets dependency still doesn't work for me, because I have Unity Pro (for building on Windows), but not Unity Android Pro. So this #ifdef will not work for people in my situation:
    #if !UNITY_METRO && !UNITY_WEBPLAYER && (UNITY_PRO_LICENSE || !(UNITY_ANDROID || UNITY_IPHONE))

    I would suggest just changing it to
    #if !UNITY_METRO && !UNITY_WEBPLAYER && (UNITY_EDITOR || !(UNITY_ANDROID || UNITY_IPHONE))

    Or, honestly, take a survey of how many people actually actually need the new .net socket dependent test runner, and if no one wants it, remove it. It worked fine before, I don't know why you had to change it or what benefit anyone gets from it.
     
  12. Tomek-Paszek

    Tomek-Paszek

    Unity Technologies

    Joined:
    Nov 13, 2012
    Posts:
    116
    @makeshiftwings
    I'm sorry the functionality got broken. I'm also busy with other tasks and I can't commit 100% of my time to UTT development. I do, however, prioritise the work on feedback and recently you can submit issue on the UTT's bitbucket page.

    The functionality itself allows you to run integration tests on a device (phone with android) and save the results on the computer where the test were invoked from. It's necessary in CI system.s

    Your problem is actually not a trivial one. I can recommend you using the previous versions of the UTT or changing the code slightly (the #if you mentioned) to disable the usage of Sockets completely in your project.
     
  13. makeshiftwings

    makeshiftwings

    Joined:
    May 28, 2011
    Posts:
    3,350
    I understand it's nontrivial (mostly just because there aren't flags for UNITY_ANDROID_PRO etc, but since UTT is supposed to be both official and not require Pro, it should compile on all versions of Unity. Maybe you guys could distribute a Pro version and a non-Pro version on the asset store, or do what some assets do and have the store install a list of .unity packages and the user then needs to select the correct subpackage to install. I don't think it's a good idea to post non-compiling code and just tell everyone to go muck around in a bunch of separate files to fix the compile errors on their own; if any non-Unity employee tried to get away with that on the Asset Store, they'd have their asset denied.
     
  14. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    6,294
    I've been running into an issue with the Integration tests that I think might be a common one, and one that should be fixable.

    So, whenever you have something in your integration test that instantiates an object, but doesn't set the parent of that object, that object will remain after the test has finished running, and is deactivated. That object can then interact with the later integration tests. This means that the tests are not independent of each other, which is not ideal.

    The most common example of this is something like bullets. If you have an integration test where you test a projectile of some kind being fired, if that bullet isn't set as the child of something in the same test, that bullet will keep flying when the test is over, and potentially hit (and kill!) something in the next test.

    There's two ways I can imagine fixing this:
    1: The integration test runner keeps track of all (game)objects instantiated during an integration test, and deactivates them when the test is deactivated
    2: The integration test runner sets every instantiated gameobject as a child of the current integration test, but somehow mocks in 'null' if those object's parent is asked for.

    Would it be possible to fix this issue? As of now, I have to have a public variable along the lines of "fakeparent", and set every instantiated object as the child of that object. I can leave it as null in production and set it to the current integration test during testing, but it's still a bit icky - especially since test details leaks into production code. I could write my own scripts to grab instantiated objects and set them as children of the current test, but I imagine that I'm not the only one who will be running into this issue.


    EDIT:
    There's another issue I've been running into. The integration test has their transform position hidden in the editor. I can see why you would do that - the test object's position shouldn't be important - but there has been times where I've been wanting to move all the objects in a test to a different position. It doesn't really hurt to have the position available, so why not just leave it there? You can obviously move the test with the editor arrows, so hiding the coordinates makes very little sense.
     
    Arkade likes this.
  15. vincentlo

    vincentlo

    Joined:
    Sep 30, 2014
    Posts:
    1

    Is there a way to automate the integration tests even further? So far I've been able to create a new GUI that lists all the integration tests that I currently have for a project, and when I click on one of the tests, the integration test runner window will pop up from which I can hit "play" to run the selected test.

    Is there a way I can make it so I can select multiple tests in my GUI, hit a button called "Run Selected Tests" and then the tests will run one after another and aggregate the results to be displayed somewhere?

    (As opposed to opening a test scene, opening the test runner, clicking play, waiting for it to finish, opening another scene, clicking play again, etc.)

    I'm a Unity newbie by the way :X
     
    Arkade likes this.
  16. Tomek-Paszek

    Tomek-Paszek

    Unity Technologies

    Joined:
    Nov 13, 2012
    Posts:
    116
    @Baste
    I wouldn't call an issue but I see your point. It's certainly inconvenient and it should be easy to fix. I'll give it a look.
    I also makes sense to enable the Transform component.

    @vincentlo
    There is a batch runner you can execute from command line. Maybe that will help you. You can also write you own API (you can see in the batch runner how to invoke a run).
     
  17. Jiraiyah

    Jiraiyah

    Joined:
    Mar 4, 2013
    Posts:
    175
    as i mentioned in the post on bitbucket issues, I am getting this error even when i try the bitbucket one in unity 5 : http://prntscr.com/5s5smo
     
  18. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,562
    I am using a test (using the unit test runner) that defines a data source like this:

    Code (CSharp):
    1. [TestCaseSource("Levels")]
    2. public void ExportLevel_AfterImporting_ReturnsTrue(string level)
    3. {
    4. }
    5.  
    6. // Returns all asset names under a specific folder
    7. public IEnumerable<string> Levels
    8. {
    9.     get
    10.     {
    11.         return AssetDatabase.FindAssets("", new[] { @"Assets/Resources/OfflineData/Levels" } ).Select(asset => AssetDatabase.GUIDToAssetPath(asset));
    12.     }
    13. }
    This results are displayed in the unit test runner window but scrolling is just horribly slow...

    Have you tested this window with many results (e.g: hundreds of tests?)
     
  19. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,562
    This is what it looks like in the profiler (trying to collapse/expand test categories).
    It looks the same for multiple frames...

    upload_2015-1-15_9-41-23.png
     
  20. konsnos

    konsnos

    Joined:
    Feb 13, 2012
    Posts:
    121
    Do these Test Tools work with Unity5?
     
  21. Tomek-Paszek

    Tomek-Paszek

    Unity Technologies

    Joined:
    Nov 13, 2012
    Posts:
    116
    @Jiraiyah
    Which OS and version you are using?
    @liortal
    Will try it out.
    @konsnos
    There were some issues in recent 5.0 betas but it should be fixed now (in bitbucket at the moment, getting to the Asset Store soon)
     
    konsnos likes this.
  22. makeshiftwings

    makeshiftwings

    Joined:
    May 28, 2011
    Posts:
    3,350
    In Unity 5, if I open Unity and the Unit Test Runner window is part of my layout, it spams the error log with the error below and the icons are all missing from the Unit Test Runner window until I close and open it again:

    Recursive Serialization is not supported. You can't dereference a PPtr while loading. (Constructors of C# classes may not load objects either eg. EditorGUIUtility.TextContent should be moved to OnEnable. See stacktrace.)
    UnityEngine.Resources:LoadAssetAtPath(String, Type)
    UnityTest.Icons:LoadTexture(String) (at Assets/UnityTestTools/Common/Editor/Icons.cs:76)
    UnityTest.Icons:.cctor() (at Assets/UnityTestTools/Common/Editor/Icons.cs:54)
    UnityEditorInternal.InternalEditorUtility:LoadSerializedFileAndForget(String)
    UnityEditor.WindowLayout:LoadWindowLayout(String, Boolean)
     
  23. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,562
    I also see the same with latest RC build (RC3).
    The same behaviour is not seen with 4.6.2...

    This occurs due to loading of objects from a static constructor.

    I didn't see anything related to this changed behaviour in the release notes(frankly, I was pretty sure you cannot load any object from a static or non-static constructors anyway).

    I know that there's a 5.0 branch on BitBucket, I can't recall seeing a fix for this issue there though. The fix can probably be initializing these textures from the context of the main scripting thread.
     
  24. Tomek-Paszek

    Tomek-Paszek

    Unity Technologies

    Joined:
    Nov 13, 2012
    Posts:
    116
  25. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,562
    What is the source of this error? what behaviour was changed in 5.0 regarding asset loading ?
     
  26. Tomek-Paszek

    Tomek-Paszek

    Unity Technologies

    Joined:
    Nov 13, 2012
    Posts:
    116
    The serialiser got some performance improvements that made loading assets in static constructor impossible (not only not recommended ;))
     
    Last edited: Feb 24, 2015
  27. Anatolii_Landyshev

    Anatolii_Landyshev

    Joined:
    Aug 10, 2012
    Posts:
    31
    Hi.

    I've downloaded Unity Test Tools version 1.5 from Asset Store.
    The Unit tests are working fine in editor. However I need to run unit tests on iOS device. Is that feature supported?
     
  28. liorrrr

    liorrrr

    Joined:
    Mar 10, 2015
    Posts:
    5
    I've never tested this actually, but it looks like it's supported.

    You can find this option under Under Unity Test Tools -> Platform Runner menu.
    When selecting "Run on Platform", you get a choice of which test scenes to build and for which platform. iOS is also included in that list.
     
  29. Anatolii_Landyshev

    Anatolii_Landyshev

    Joined:
    Aug 10, 2012
    Posts:
    31
    @liorrrr Thanks for quick response.
    As I understood Run on Platform runs only integration tests. I couldn't get my unit tests results using that option.
     
  30. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,562
    Unit tests usually are desired to be run in isolation, and as such don't require the actual game engine, etc to be executed.

    Why do you need to run a unit test on the target platform?

    BTW: I'm the same as liorrr, it was just my other user (with my work email address).
     
  31. Anatolii_Landyshev

    Anatolii_Landyshev

    Joined:
    Aug 10, 2012
    Posts:
    31
    I'm developing a LINQ to iOS library: https://www.assetstore.unity3d.com/en/#!/content/18131
    System.Linq works fine in editor. However it causes JIT compile errors on iOS. The aim of my library is to provide LINQ implementation that works on iOS. My unit tests are running fine in editor, but I need to be able to run them on iOS in order to check the code for JIT compile errors.

    Earlier I was doing that with NUnitLiteRunner for Unity: http://wiki.unity3d.com/index.php?title=NUnit
    But unfortunately it's not working with IL2CPP. So I need another tool for unit tests with ability to run them on iOS.
     
  32. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,562
    Well you can probably achieve this with the integration test framework.

    Create a single scene with an empty game object that will have a script on it that instantiates whatever unit test case you'd like to check.

    If your library compiles fine for iOS you should have no trouble in building and running that on device.
    The only point here is that if you're using NUnit's assertions or anything else that comes with it, you'd need to include that in the runtime code (Unity Test Tools includes NUnit only for the editor).

    Other than that I can't see any reason that it won't work.
     
  33. Arkade

    Arkade

    Joined:
    Oct 11, 2012
    Posts:
    654
  34. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,562
  35. Arkade

    Arkade

    Joined:
    Oct 11, 2012
    Posts:
    654
    I'm not at the computer to check but from memory the test checks a value on its own component against a value in code. Leave the code value unchanged and modify the GUI value.
    Make sense?
     
  36. Boris0Dev

    Boris0Dev

    Joined:
    Apr 27, 2015
    Posts:
    10
    Hi,
    I'm using Unity 501f1 personal 64 bit and encountered 2 issues:

    Under /assets/scripts I've created "HealthManager.cs":

    Code (CSharp):
    1. using UnityEngine;
    2. using System.Collections;
    3. public class HealthManager : MonoBehaviour {
    4.     public float healthAmount;
    5.     public void TakeDamage(float damageAmount) {
    6.         healthAmount -= damageAmount;
    7.     }
    8. }
    Under /assets/UnityTestTools/Examples/UnitTestExamples/Editor the Test "HealthTests.cs":

    Code (CSharp):
    1. using NUnit.Framework;
    2. using System;
    3. using System.Collections.Generic;
    4. using System.Threading;
    5.  
    6. using UnityEngine;
    7.  
    8. [TestFixture]
    9. public class HealthTests  {
    10.     [Test]
    11.     public void TakeDamage_PositiveAmount_HealthUpdated()
    12.     {
    13.         // Create a health component with initial health = 50.
    14.         var health = new HealthManager();
    15.         health.healthAmount = 50f;
    16.  
    17.         health.TakeDamage(10f);
    18.  
    19.         // assert (verify) that healthAmount was updated.
    20.         Assert.AreEqual(40f, health.healthAmount);
    21.     }
    22. }
    Everything works fine, the test is recognized by the test runner and executed.

    If I move the test to /assets/scripts/tests/HealthTests.cs or ANY other location I get the error "Assets/Scripts/Tests/HealthTests.cs(1,7): error CS0246: The type or namespace name `NUnit' could not be found. Are you missing a using directive or an assembly reference?"
    But: I've added the reference to the project too. Please have a look at the first 2 attached files.
    Any ideas how I can solve this issue? Or is this a bug?

    Second issue:
    In the assertion compoment the attached script "MoveBall.cs" is not listed. See 3rd attached file. Maybe the reason is because the script is in the project "UnityVS.sandbox.CSharp" and not "UnityVS.sandbox.CSharp.Editor" (see 2nd screenshot)

    Thanks editor 1.png editior 2.png editor 3.png
     
    Last edited: Apr 27, 2015
  37. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,562
    @Boris0Dev plz format the text as code, makes it easier to look at it here.
     
  38. Boris0Dev

    Boris0Dev

    Joined:
    Apr 27, 2015
    Posts:
    10
    Update: the issue with the assertion component I solved now on my own. It wasn't clear to me, that only methods are listed that are compatible with the comparer type.

    Update 2: I solved also my 1st issue. Unit test _have_ to go to /assets/Editor...
     
    Last edited: Apr 27, 2015
  39. npruehs

    npruehs

    Joined:
    Jul 20, 2012
    Posts:
    67
    Yes, they have to, as they rely on the NUnit.framework.dll which lies in Editor/Plugins as well. That library has to go there in turn in order to allow to make builds for all the different platforms (Standalone, iOS, Android, ...) which would otherwise blow up due to the Windows-specific code in NUnit :)
     
  40. yoyo

    yoyo

    Joined:
    Apr 16, 2010
    Posts:
    112
    Note that your unit tests can go in *any* Editor subfolder anywhere in the Assets hierarchy, for example Assets/Scripts/Editor/Tests/HealthTests.cs should work fine.
     
  41. Lars-Blaabjerg

    Lars-Blaabjerg

    Joined:
    Apr 5, 2011
    Posts:
    54
    I would like to use Unity Test Tools, especially the Integration tests. My problem is that my project includes the Windows Phone 8 platform. And compiling it fails if Unity Test Tools is even in the project. Since I can't exclude stuff from the build. I have to remove the integration test tools AND the integration tests completely from my project to compile it on that platform.

    It would be really nice if that was fixed. Even if it means providing non-functional stubs and/or throwing not supported exceptions. I can live with not actually being able to DO the tests on WP8.

    As it is, it is too much trouble.
     
    Alex-Chouls likes this.
  42. Newcomma

    Newcomma

    Joined:
    Feb 10, 2015
    Posts:
    89
    I'd love to integrate this somehow with a code coverage analysis, has anyone tried this / is it planned or possible?
     
  43. Newcomma

    Newcomma

    Joined:
    Feb 10, 2015
    Posts:
    89
  44. pretender

    pretender

    Joined:
    Mar 6, 2010
    Posts:
    862
    Unity Test Tools are great, but when I'm building for Android I get an error:
    for I guess editor code isn't properly enclosed in ifdefs...how to properly fix this?
    thanks!
     
  45. Arkamis

    Arkamis

    Joined:
    Apr 8, 2013
    Posts:
    4
    I came across this issue today. My hack fix is this:

    #if UNITY_EDITOR
    Undo.RegisterCreatedObjectUndo(go, "Created test");
    #endif​
     
  46. Arkamis

    Arkamis

    Joined:
    Apr 8, 2013
    Posts:
    4
    I am trying to run Code-based assertions from the command line for CI, and I am having a number of problems.

    1.) No results file ever gets written, no matter what I use for the -resultsFileDirectory options.
    2.) Even when using the -quit option, sometimes it just executes the game and leaves it running.
    3.) Watching the diagnostics feed using DebugView, I get entirely different behavior every time I run things.
    4.) Occasionally, the tests never seem to get run, the game just builds. The exit code is successful even with a deliberately-failing test included.

    The command I am using to run everything (on Windows) is:

    Unity.exe -projectPath C:\myProject\PathToMyProject\ -batchmode -executeMethod UnityTest.Batch.RunIntegrationTests -testscenes=MyDemoScene -targetPlatform=StandaloneWindows -quit
     
  47. Arkamis

    Arkamis

    Joined:
    Apr 8, 2013
    Posts:
    4
    In fact, I cannot get the PlatformRunner to work at all. It is in no way clear what it is trying to do, and no amount of changing properties seems to have any effect.

    EDIT: well now it seems to be working. I don't know what I did.
     
    Last edited: Jun 9, 2015
  48. pretender

    pretender

    Joined:
    Mar 6, 2010
    Posts:
    862
    Thanks I'll try this!
     
  49. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,562
    Hey,

    I am using Unity 5.0.2 and just started to use stripping for iOS (bytecode).

    Getting an error now in Xcode, i guess that some of the methods were stripped away (from IntegrationTestFramework):

    upload_2015-6-11_10-14-13.png


    upload_2015-6-11_10-14-22.png

    Not sure what's wrong here. On one hand, we're not using IntegrationTests so it should probably be stripped, but looks like not all of it is included or something's not right.

    Anyone else experienced this issue ?
     
    danhamel likes this.
  50. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,562
    Please disregard my previous post - the bug is due to something else - (Case 704069) IL2CPP (iOS) build fails when removing scripts between builds without restarting editor