Feedback Why the regressions to Test Runner in Unity 5.6?

We have recently moved our project from Unity 5.5.0 to Unity 5.6.1 and noticed the Test Runner was changed and now supports play mode tests and updates the results as it executes. That's great! However, we also noticed that:

1. Tests are running much slower now. For few lengthy tests, about 3 times slower (of course it depends); for many quick tests it's actually much worse by orders of magnitude (see post below).
2. There are no icons to select only tests that failed or that were not run yet. This was very useful!
3. There is no number that counts the number of tests (just a detail).

Why these regressions? Will those be fixed in the future?

Thanks and cheers,
Daniel

 

It seems that due to the results updating while executing, there is a large interval between running each test. If you have 1000 very quick tests they will now take a very long time, whereas before they finished almost instantly.

In fact, I tried this with 1000 tests that look like this:

Code (CSharp):

1. [Test] public static void Test1() { Assert.True(true); }
2. [Test] public static void Test2() { Assert.True(true); }
3. [Test] public static void Test3() { Assert.True(true); }
4. [Test] public static void Test4() { Assert.True(true); }

They took about 21s on my machine (i7-3632QM), having only trivial code as seen above.

Most of our actual test scripts went from taking a few dozen ms each in Unity 5.5 to taking hundreds of ms in 5.6. And as I've just now optimized them, they should run in a few ms but take about 6 seconds total.

Can we return to the old test scheme somehow? As it stands this is having a hit on our development time.

- Daniel

 

Just reported this as a bug. At the moment I have no suggestions for other people running 5.6... I haven't tried to get old versions of the unit testing tools into our project. Maybe that could work it in the meantime...?

EDIT: just tried the tests in Unity 2017 (2017.1.0b6) and it's even worse. Hangs for a while before even starting to run the tests, depending on how many you have. Also reported that. D:
UPDATE: for the hang problem in Unity 2017, it had already been reported by someone else: this is the page. The general slowness in 5.6 is still a problem, however.

 

I would like to note that I have the exact same problem. We have around 1400 unit tests in our project and recently upgraded from 5.3.41f to 5.6.1f. The unit tests that formerly took a few seconds now take over a minute (!!) to run.

 

Ouch! That's even worse than our case. Let's hope this gets fixed. If I end up trying an alternative I'll let everyone here know.

 

I'm seeing the same problem. As soon as you start using just a few [Values(...)] attributes or similar, as the amount of permutations grows it becomes unbearably slow.

It's easy to see this with the following test:

Code (CSharp):

1. using NUnit.Framework;
2.  
3. public class PerformanceTest
4. {
5.     [Test]
6.     public void ManyTests(
7.         [Range(0,1000)] int testInt)
8.     {
9.         Assert.IsTrue(true);
10.     }
11. }

This is just 1000 executions that should be sub-millisecond each, but in Unity 5.6 this takes ~18 seconds for me.

Also, the lack of filters is super annoying -- we used the category filter all the time.

 

Also, is there any way of starting the test runner from script? I just recently implemented ITestRunnerCallback / Batch.RunTests() into our build script, it worked nicely and allowed us to check the unit tests on each build. All those classes appear to have been removed in 5.6, is there any replacement for this use case (running tests from scripts)?

 

We're facing the slowness issue as well, but it seems to vary between machines, even after clean installs of Unity (including deletion of all settings).

In addition, we've noticed the following regression: The unity test runner no longer treats NUnit's Explicit attribute correctly. These tests are supposed to be run if selected by a category, but now they only run if executed individually - both in batch mode (build server) and in the editor. This is the same issue as noted by @sybo_jon above, about the category dropdown disappearing from the test runner GUI. For us, this is a big deal, as we have a significant number of tests that are unity-independent and run in the IDE, so we need a way to run different sets of tests in different environments.

 

I just want to say that we're experiencing the exact same problems. Its also very annoying that you can't alt-tab away from Unity while the test runner is running (it will stop and wait until you tab back).

We've gone from being able to run all our tests in a few seconds, to it taking 5-8 minutes.

I've noticed that this has increased the amount of failing tests, since developers don't want to wait that long for them to run.

I think its time for Unity to tell us what the plan is, and when we can expect improvements.

 

Another one: The XML format generated by the test runner is no longer valid NUnit XML, because the required root element is missing. Build servers tend to love that...

 

Anybody at Unity care to comment? I think @Tomek-Paszek was doing related work..?

 

The comments everyone is writing just make me realize this is even worse than I thought! Would also really appreciate if someone from Unity could talk about this a bit.

 

I reported that months ago in a 5.6 beta and it's still active... I had to workaround it in Jenkins rewriting output XML...

It's annoying that something so easy to solve takes months to fix.

 

I also think that Unity should create a subforum for test runner because this threads get lost...

 

The new test runner also doesn't support tests in managed plugins (DLL assemblies). We use DLLs exclusively for our project and this is making our tests totally unusable.

It's clear to me the new Test Runner is not quite ready for prime-time. Can someone from Unity comment about this and potentially revert back to the old Test Runner somehow?

 

Also documentation regarding batch mode is outdated and I had to submit bug report to learn the undocumented new parameters for play mode tests.

 

First of all, thanks for your reply.

I thought that was the case, but it really is a huge difference for non-frame dependent tests (or non-Unity-dependent tests, for that matter). Though I realize the following may be a little maintenance nightmare, I suggest adding a toggle so that we can run tests sequentially and without breaks, using the old system (while the new one isn't up to par).

Secondly, do you know of a way that we can, right now, run the tests in the old Editor Test Runner - or similar tool - while keeping Unity 5.6? I believe everyone in this thread would like to know this. Anything Unity Technologies can provide in separate, like the old tests module itself?

Thanks!

 

Thanks for replying. I could use the Batch-class with reflection but what about the callback interface, is that still around somewhere? In 5.5 I had a nice progress dialog that listed the number of tests that succeeded, failed etc. so I need the callback to be able to do that.

 

@Tomek-Paszek reading you I'm thinking that there is no dedicated team working on test runner because you broke so many things months ago and there are still broken...

 

the performance regression in the test runner is huge, we do need a workaround or for this to be fixed in an upcoming beta, as it affects our ability to deliver high-quality products with Unity. Can this please be prioritized?

EDIT: Just to be clear, the tests themselves actually run quickly, but the problem we are seeing is that they take a very long time to actually start running (from the time 'Run All' button is clicked) - Unity.exe takes 100% CPU of one core for a long time, before showing the modal 'Running tests' dialog. For example, we have 477 unit tests, which take 10 seconds to run, but Unity.exe hangs at 100% cpu for 4 mins and 30 seconds prior to executing them. This is on Windows 7 and Mac using Unity 2017.1.0b1 (64 bit). It happens when we select to run all tests or even just 1.

 

Thanks @Tomek-Paszek, we are using Unity 2017.1.0f1 (released a few days ago) but in fact it turned out it was our bug. We are using nUnit's TestCaseSource attribute to generate a lot of tests which yield return each test case, and i discovered that some code that deserializes a large json file was being run during creating of every test case. So instead I just made a static field with the deserialized result so that it was only done once, and the 4.5 mins reduced to about 15 seconds. Sorry for the false report.

 

This new test runner was clearly rushed out. It should not have been released with so many regressions. Very disappointing and causing headaches for many teams. Not having enough resources is no excuse for releasing something so broken that is vital to many teams.

 

Could you guide me how to access the Batch class via reflection? I did something like this and could not find either Batch class or UnityEditor.EditorTests

Code (CSharp):

1.         Assembly unityEditorAssembly = Assembly.Load("UnityEditor");
2.  
3.         StringBuilder sb = new StringBuilder();
4.         foreach (Type type in unityEditorAssembly.GetTypes())
5.         {
6.             sb.AppendLine(type.FullName);
7.         }
8.         Debug.Log(sb.Length);
9.  
10.         EditorGUIUtility.systemCopyBuffer = sb.ToString();

 

I haven't actually tried yet, with more important tasks at hand. But I was going to use ILSpy to find out where the classes were hiding and work from there. Trying it now, I can't seem to find them either. Maybe @Tomek-Paszek can point us in the right direction?

 

Wait, the massive delay before running any number of unit tests hasn't been fixed for the final release of Unity 2017.1?

In our project I click a test and it takes almost one minute of high CPU thrashing before it runs any test (I have no idea what Unity is doing with it). Then it happily says that tests took a few hundred ms themselves. I reported it during the 2017.1 beta, and it was apparently already reported by someone else during beta version b1, here:

https://issuetracker.unity3d.com/issues/editortest-running-a-single-edit-mode-test-is-really-slow

I'm baffled, really. The Test Runner is completely unusable now. This is why we shouldn't update to new versions, really... Why even have betas if critical problems are left unfixed for months?

Sorry, I'm really mad at this.

 

Hi Tomek,

Thank you for replying to my angry venting moment. :| Personally, we don't need play mode tests, so seeing everything getting worse performance (in this case a bug) each version is really maddening for me. When you have more news about this please do tell us.

Cheers,
Daniel

 

Just to chime in, the issues still haven't been fixed. I'm like filing bug reports since May or so, but nothing has improved so far. We had to go for weeks without getting test reports back from our CI until I took my time to postprocess the results XML file in a sensible way. We have about 6000 (edit-mode) test cases and running those in the editor just takes ages... (2017.1.0p1).

Also: https://forum.unity3d.com/threads/unity-test-runner-batch-mode-broken-workaround.482145/

 

Yeah, it is a shame that I am just getting into testing practice too. Currently I have to stick to only PlayMode test because I can afford to wait in that mode. I wonder when will the EditMode test become sensible again. (Did you tried 2017.2 beta?)

 

I suspect the issue is still the original problem that made me create this thread, which is that the new Tests Runner is very inferior to the old one, performance- and feature-wise. Where tests ran almost instantly before, now there's a few frames between each one. In a project with 6000 tests this is now probably an insane wait (likely takes minutes).

I haven't tested p1, but it says this: "Editor: Fixed a performance regression that cause very long startup time when invoking a test run." This seems related to the bug that appeared in 2017.1; not the runner regression.

 

We also had the problem that the test runner doesn't support tests in DLL assemblies and made a testproject for this case. It's reported under Case 948723.

 

@Tomek-Paszek we have recently moved to TDD however we require to upgrade our unity project to use 2017 (we were previously using 5.4.5). I've just spent nearly a week upgrading all our various plugins to play nice together and with our project. I've now been fixing our tests (having finally made the thing compile). I double click a test, wait 3 mins and then it runs. The test took a few ms. We are using 2017.1.0f3. Should I be using a different version? This performance hit will seriously hit our team where it hurts.

 

We used to utilize 'Application.temporaryCachePath' for some of our unit tests, this also seem to have regressed. Is unit tests not performed on the main thread any more?

 

Any news on this, @Tomek-Paszek? I was needing to do more tests and it all takes soooo much time to run (using 2017.1 and 2017.2 alike).

This thread was created 6 months ago, and apart from Unity fixing the performance bug introduced in 2017.1.0, nothing seems to have changed in this regard (to my knowledge, since I haven't tried the 2017.3 beta). How is this issue going?

Cheers!

 

@Tomek-Paszek What about the Batch-class and ITestRunnerCallback, I couldn't find them to use with reflection (as you suggested)? Where are the classes these days? They shouldn't have been removed if there is no replacement..

 

Simply to reply to this as we are using unity latest LTS 2017.4.1f1 and it seems have been not fixed.

Running tests is pretty quick in itself though there are still 2 big heavy issue :
- Startup always takes around 15s each time. (before the progress bar appearing)
- After test are run (we can see results in the window) unity hangs for 60s !

This is a pretty unbearable situation, this is reproductible whatever number of test we are running, even 1.

60 runs of test in a day equals 1h15 minutes of work lost hanging. Right now its one of our most major caveat.

As you are working on the LTS it should really get fixed for it to be production ready

 

As additional info. When hitting the cancel button (for instance when making an infinite yield return null test) the 60s of hanging are still there.

 

I haven't tested 2017.4 yet, but the general slowness and waiting frames between tests is still there.

Now there's a progress bar so you can't even do anything else in the editor, but that's okay. However, if that is the case why not run every Edit mode test at once in the same Editor frame and skip all the waiting? Ever since Unity 5.5 the performance of Edit mode tests never recovered. It ran almost instantly and now takes a very long time.

Maybe give us the option of running a configurable number of tests before updating the progress bar, such as 10, 100 or infinity. So if you want to see the progress bar you can choose, but we value our time much higher.

Daniel

 

There is a regression again in transition from 2017.3.1 to 2017.4/2018.1 beta. EditMode test has a very long start up and tear down time. Running multiple tests is slow only before the first test and after the final test. By slow it is like 5-10 seconds, depending on how large your project is. (I think it depends on how many scripts you have, even if they are not related to the tests) This make rapid iteration impossible and I would rather test manually.

PlayMode test is not affected by this. I have 2 related bug report submitted with no reply yet
https://fogbugz.unity3d.com/default.asp?1010794_q8u3dlr4095r6dqm
https://fogbugz.unity3d.com/default.asp?1021076_uhcsjltr5gpv1g97

 

Our issue is definitely this one. As our project has a lot of scripts it may be what takes 60s for tear Down. Please we need a fix

 

Another "still an issue" issue with the testRunner is the lack of automation. The "run on recompile" option was removed and has not been restored. The only reference I see to this issue is this thread, now nearly a year old, stating that the option would be restored soon. There's a suggestion in the thread to reimplement it in an editor script... except there's absolutely no reference as to what API calls can be used to run the tests in a script! I've been digging around in the available source code and the assembly browser but I can't find anything useful for this purpose.