Search Unity

Unity Test Runner Batch Mode broken / Workaround?

Discussion in 'Scripting' started by einWikinger, Jul 7, 2017.

  1. einWikinger

    einWikinger

    Joined:
    Jul 30, 2013
    Posts:
    97
    We've been reyling on automated tests for a very long time, and before the nunit test runner integration in unity we've been rolling our own solution which we then ditched for the far better nunit test framework support.
    Now, we're without a working CI because the batch mode test runner got broken somewhere in the 5.6 release cycle. Bugs have been filed etc. but nothing changed, and for months I have a bad feeling in my stomach because we don't get the results of our automated tests, let alone that nobody writes tests anymore because -what's the point when they're not run?

    Traditionally we've been running the TeamCity CI, and thus used the "-editorTestsVerboseLog teamcity" switch to get realtime feedback of the tests. This has been broken since at least 5.6. As TeamCity can also interpret nunit test result files directly, I've been trying to use that as a workaround, just to find out that the result XML files generated by the test runner doesn't conform to the XSD (the root test-run element is missing, see fogbugz issue #910845).

    Does anyone have the same problem? I've been looking for workarounds for ages now (e.g. invoking the test runner manually with a custom results generator, ...) but everything I thought about just wasn't worth the time or just far too complex for such a "simple" issue. I'd be more than happy to be proven wrong here!

    *ranton* Hey, Unity Technologies, how'd you feel if your automated testing pipeline just shut down for the next three months? *rantoff*
     
    Last edited: Jul 7, 2017
    CanisLupus likes this.
  2. HaraldNielsen

    HaraldNielsen

    Unity Technologies

    Joined:
    Jun 8, 2016
    Posts:
    139
    Hi einWikinger,
    Nunit dont have XSD for version 3. You can see the nunit issue here: https://github.com/nunit/nunit-console/issues/13
    We fixed the missing element and went trough the documentation and made sure that the xml is valid compared to that. We are backporting this to 5.6 as well, so you should see this fixed soon.

    editorTestsVerboseLog are not supported on the new testrunner, but we will be adding something else. I don't know yet if that will be backported to 5.6, it depends on how easy it will be.
     
  3. einWikinger

    einWikinger

    Joined:
    Jul 30, 2013
    Posts:
    97
    That's great to hear. We upgraded to 2017.1 in the meantime. Having a valid results XML will be fine in our case, I liked the realtime feedback on executed tests with the verbose log, but can live without it. Thanks for the update!
     
    HaraldNielsen likes this.