Automated Testing

Hi there,

[I don't know if here is a right place to it (I saw "General Discussion and Ideas" and here I am)]

I'm from Brazil, I'm a master degree student and I'm willing to use Unity as base tool for my thesis!

I come from Web Development/Software Engineering but I'm a lover of game development! So, I'm a newbie in game development, I know Unity for about 2 years and always wanted to try it for real. Now I have a chance!

As you might already know, automated testing "became a fashion" (a really good thing actually) nowadays, leveraged by agile methodologies!

I have read a review about studies related to game development and something that is not well explored is testing in general (automated test, my objective).

So, finally, what I want is a discussion about it (automated testing in game development, not agile methodologies). As I don't have the knowledge, I really don't want to enter in something unfeasible!

I would like to know your thoughts. When you use Unity (in your job), what kind of automated test you do in your workflow? Any kind? And, if you could do something (but you don't have time) that could be very cool and feasible, what would be it?

I want to make a extension to Unity related to it (if possible)!

Thanks!

salaniojr

 

I think that's a really interesting subject. I guess one reason why there's not that much written on automated game testing, yet, is that games are heavily presentation oriented - and that's rather difficult to write proper automated tests for.

Maybe you can contact UT and see what kind of testing procedures they have in place to test the game engine itself.

 

We so far generally did only do very limited automated testing if any at all.
Thats not only related to the fact that the whole input - timing relationship makes it hard to reasonably test anything (The only available and at even barely usable automated testing solution I'm aware of is actually Test Complete which offers visual based testing which is required as you can simulate input for testing purpose) but also the fact that there are major limitation on using Unity with Unit Testing suites in general (might have gotten better, have not rechecked it lately as trying to use unit testing at the end of a 6 month+ project makes no sense), the only ones I know that they worked really well were user created in Unity 2 days and extremely limited compared to the "common solutions".

But generally I think it would be a very interesting topic and a solution that would work well would definitely be great.


Until then I will continue my "user driven unit tests" with isolated test environments for distinct aspects where single buttons and alike test whole aspects and the user has to verify the output

 

Thanks jashan, I really appreciate your response, but maybe I'm a bit lost, what you mean by "presentation oriented" and why it hinders the automated testing?

Yeah, it's an interesting option!!! Again, maybe I'm not following you, but isn't it two different types of applications?! Because what I would like is an extension to help end users of the engine to test their games by inserting the extension on their workflow, or something like this...
What do you think? Engine automated tests could help me this way? If yes, there's some specific person that could help me at UT that you could recommend?

Thanks!

 

From blogs and other sources UT's automated test suite does playback of recorded user input, and screen grabs followed by per-pixel comparison with a threshold for catching rendering errors. They probably have timer-dependent rendering going directly by framecounts, or it's possible they just set timescale to a very lage value to keep timer-inaccuracies down to a minimum.

AngryAnt is part of QA I believe.

 

Hi cannon, thanks for your reply. If it happens like you said maybe it's not what I really need! But I think every experience worth looking, I'll try to talk with AngryAnt to confirm...

Right now I'm working with an example that could help clarify what I really want to do, to show you with images.
Hope you guys follow it...

Appreciate all your help!!!

 

@cannon: thats the kind of Unit Testing currently available through Test complete, and calling it a major pain is an understatement, cause visual stuff is totally useless over the course of a game where visuals change as the slightest delta makes the test fail just due to "input not equivalent". Its good for "all is finalized" situations to test against regressions but thats where it ends.


@salaniojr: looking forward to it

 

When you're doing testing, there's various layers into which you can hook the tests. Unit tests typically are very simple tests checking whether basic functionality works the way it is intended to work. Unit tests are comparatively easy to implement and are usually rather solid. When you want to do Unit testing in games, you'll probably usually want to have the "core game mechanics" in a separate project that doesn't need the engine - so that things are being kept simple and you can easily implement the Unit tests.

Another approach that one could think of is some sort of special scenes with a special GUI that test stuff that does run in the game engine; still more or less "Unit tests" based.

However, since games are heavily audio-visual and interactive, a much more interesting thing to test will in many cases be the actual presentation. In other words: Do thing look correctly? Do things sound correctly? And that's much more difficult to implement tests for - as dreamora has already explained.

In a way, yes. But still, the way test their engine might provide a good foundation for doing actual game testing; something to get you started.

 

Having no knowledge on the subject, I'd say you could set it up so that inputs are pressed and everything, and check if the camera ever clips through a wall, or if a player can actually get stuck, or something. That shouldn't be too hard to set up, and you can set time to run much faster so it gets through more stuff.

 

Yeah Muzz5, it is related about what I want and something more in the next post...

 

Sorry about the late with the ideas! I have been busy with a report... and sorry about the size of it... couldn't be smaller, is a master thesis needing help...

So, here’s my, how can I name it....... uh, little illustrated report.

Just to recap, you can consider me as a game development newbie, I have experiences with enterprise software development only...

I really can’t, certainly, tell this isn’t utopic. I feel that it is feasible (somehow), but you guys can help me realize. I don’t know if it is useful either.

I want to test this scenario shown in Figure 1. I know, it’s simple, but it’s for the sake of illustration:


Figure 1. Game Level – Top View

• The blue ball, with the camera attached to it, is the player;
• The darker gray lines are walls;
• The yellow squares are just crates to enable hide spots to the level;
• The red balls (in fact are capsules, Figure 2 shows it) are enemies with guns;
• The green balls are pickable items (provide health, ammo, armor, etc.);


Figure 2. Game Level – Perspective View


First Thought
Functional Tests [1]:
– Test more units functioning together at higher level;
– Requires more time to build a Functional Testing Framework (yep);
– Can capture bugs generated by the interaction of different units;

Let’s say that I want to check if the enemies (red balls) are well programmed. At first, we could simply drive the player through all the paths where enemies lies (using Pathfinding). If the player just walk, without shooting enemies, what happens? The player dies? Or maybe, “damn, he crossed all the way to the end and didn’t died”! I know that it maybe sounds stupid, but it’s a simple programming mistake that could be tackled by easily running an automated test. Yeah, you could really try it for yourself in a level this little, but if the level is bigger, with a lot of paths to choose, and you want to balance it? If you could have something like this by automation: “Man, something is wrong, he shouldn’t have gone trough that first enemy easily, the enemy didn’t noticed the player”... you could report it in log files or in another user friendly way that you could notice that “He was shot just by the second enemy (objectName: enemyPrefab02), uh, but it took to much hits for him to die (action: player dies after 50 hits), he almost scaped by just running, I don’t want it, too easy, maybe it fits in newbie level (easy, normal, hard...)”.

And you could set up the player to act reasonably too, shoot enemies, look for hide spots (behind crates or wall), maybe setting waypoints in the level to AI work in a targeted way and see how it works with “light speed” running and view the report, after execution, to spot some hardly “spotable” manually bugs (I don’t really love the spot word).

Second Thought
Smoke Testing [1]:
– Load different levels looking for crashes;
– Run scripted sequences looking for crashes;
– Generate random inputs looking for crashes;

Besides the examples above, an example of this could be mixed with the “running through the level player” example that I gave in the last thought. While it is going through the level it could try any possible input, try running through walls or even through enemies to check collisions, trigger scripted sequences while testing inputs, etc.

Third Thought
I could do it in a more simple manner! That means, create testing components to objects. Something that I could call “Test Units”.

Imagine that you have created the level above presented in Figure 1 (you’re not a good level designer, are you?), and you just want to assess a single object that is not working properly (you manually checked). Let’s choose the Pickables.

And you complains “Geez, I don’t wanna write a Unit Test”, and I thought about creating a Test Unit. It could be something like this: you put a component to the pickable item asses it by writing an assessment in a more natural way.

In practice it could be something like this:


Figure 3. Test Unit in action.

Notice that is an immature idea, that is a prototype of a future polished product. Is not close of a good thing, but try to feel the idea!
So, in Figure 3 you can see in the Inspector a “Test Unit” script attached to a Pickable item with a specific language used for testing, in other words, a Domain-Specific Language (DSL) for the Testing Domain that you can use to apply scripts for Testing Framework use. The floating window called “UTest”, maybe, doesn’t makes any sense, I just created it to illustrate the framework being enabled, that means, enabling every Object that has a Test Unit to be managed by the framework to give feedback about what happened on action following the assesment script (DSL).

So if you don’t know what DSLs are (are part of my thesis), simply saying, DSLs are languages, and languages must have parsers, there is ANTLR for parser generator and there is a implementation for C# and as you know, Unity supports C#. So, I want to create a language that is Testing specific that could ease the testing process. I don’t know, maybe using the Reflection API from C# to implement testing tasks.

References (only one?)
*I know it's not well supported, but if I would go to much on this I could loose your feedback because of late. I could put something about DSLs, TDD, Automated Testing stuff and... you can go further by searching google!

[1] Francesco Carucci http://yetanothergameprogrammingblog.blogspot.com/2010/06/aaa-automated-testing.html

PS.: the utopic thing that I talked about in the beginning is because I really can’t find something like this anywhere in the web, maybe because it is a silver bullet and no one would release it for free or it is just unfeasible to do and not a single company would apply it in their workflow. Tell me if it could be feasible to do and helpful.

Thanks!

 

Certainly unit testing libraries can work for some instances, if your game relies on something formulaic, it can be tested. Expected paths in pathfinding code, expected results in a turn-based combat situation, etc. Though it gets harder if your game is real-time, more so if you rely on complicated physics because of the unpredictability.

I remember a gamasutra article about this. They even unit tested their rendering engine by having it render debug stuff, then comparing the resulting render to the desired picture (from a file), pixel by pixel, and it worked for them.

 

Yeah, I think you're right, but I think about pathfinding in an straightforward manner, to seek for basic mistakes like I said in the report in my last post.

So, maybe, you think I could input some random variabilities in the pathfinding algorithm to accomplish more possibilities or is it too hard to do?

Yeah anomalous, maybe is this article: http://www.gamasutra.com/view/feature/2269/automated_tests_and_continuous_.php, you're talking about, I've already read it! It's nice, but is not what I want as you may know from my last post, but it's not least interesting! In fact, I don't have know idea how they do that, and might be quite complex do accomplish!!! But it was a nice reference!

Thanks for your reply!

 

anomalous, your project's links aren't working with me, I couldn't see them! I'm getting "Bad file length."

 

Sorry about that, herobo is a free web host so you can imagine the amount of bandwidth it has to manage, so usually my download/upload there goes timeout.

I've since moved to using dropbox but not all the stuff has been migrated. You can go here for my latest work in Unity: http://anomalousunderdog.blogspot.com/search/label/Unity

I'm not sure what you mean, pathfinding shouldn't give unpredictable results, but you can create variations and test those, example: getting the path with least movement cost instead of shortest path (make sure it skirts around mud which would have slowed down movement)

Perhaps you can also automate testing of AI, but if you want the AI to have unpredictability on-purpose (maybe you don't want the game to become too predictable), be a bit lenient on the testing and have it accept more than one condition to satisfy the test. i.e. perhaps in a certain situation you expect the enemy to either flee or use his health potions, so you just make it that if it did either then it would deem the test successful.

 

Hey guys. Got a PM poke and stopped by. While I am no longer in QA, I can talk about our coverage, given that I used to work on our test frameworks.

Today we have several levels of testing - the higher the level, the slower the test and therefore the less often it is run. All of them are tied up in to a continuous integration setup - base on a software solution called TeamCity. The setup allows us to specify frequency of- and relations between build targets and test frameworks - of course in the end based on version control checkins.

The fastest and lowest level stuff is probably the C++ unit tests. These are basically as advertised. Higher up the chain we have C# unit tests. Figuring out what they are should also be fairly straight forward. On the high level, however we have a bunch of different stuff with some relations.

Our integration test framework provides an NUnit based API which enables test writers to launch and instrument an editor, load or construct a project, build and run for any target and instrument the resulting runtime. The editor instrumentation has a complete editor scripting bridge and the editor log is wrapped - to facilitate two-way communication between editor script and test runtime. Obviously you can also check stuff like asset folder structure and so forth.

The runtime has no built in test-to-runtime communication, but the player log is wrapped - just like the editor log. Furthermore, the runtime can be queried for screenshots and audio recordings.

The integration test framework is commonly used for creating tests that verify the process of creating a scene, building it and correct execution of it on target platforms.

Sharing some of this functionality is the graphics tests suite. Here, pre-constructed projects are built and run and screenshots are checked for pixel differences. The concept is that this test framework runs faster than the integration tests and can therefore check for rendering engine bugs fairly often.

There are a lot of extra features an nuances of course, but this is more or less the core concept. The most well known framework, which was mentioned earlier, is the record-playback rig. Here a bunch of projects are built using a special runtime which can either record input, downloads, screenshots and audio or play back input and downloads and record screenshots and audio for comparison to an original set. The point is to maintain compatibility with user projects in the web player - throughout new releases.

I've probably missed a lot of stuff, but I hope you find this useful.

Oh and we're hiring in QA

 

Thanks AngryAnt, it's really nice to have these overviews from inside the HQ!

I really didn't get it right! What is editor scripting bridge and what is log wrapping? What is a two-way communication between editor script and test runtime?

Here too, what is built-in test-to-runtime communication? And again, log wrapping... when you say you can query the runtime, you're saying that you record the play action and further you can look for specific parts of the recording?

I really would love to work with you guys at UT, but unfortunately I don't think I have what is needed!!!

 

The scripting bridge to the editor is a way for the test to send editor scripting code to the editor instance - to have it executed. The runtime does not have a setup like that.

The logs are wrapped so that the test can for instance go:

Code (csharp):

1. editorInstance.Eval ("Debug.Log (\"Expect this message\");");
2. editorInstance.Log.Expect ("Expect this message");

 

Got it!

Angry, as you have already worked with Testing and I'm really needing to define my thesis' theme, can you please take a look at this post http://forum.unity3d.com/threads/95054-Automated-Testing?p=625117&viewfull=1#post625117 of this thread and tell me what you think, if you have time of course!!!

Thanks!

 

Hi Guys,

I am also struggling automating Unity Games. I am using Editor test runner for automation but its not solving my purpose.
Is TestComplete is the best tool for automating Unity Games?
Can someone please help me understanding how to automate UI of Unity Games? @AngryAnt

 

Seems that someone made one more solution
http://forum.unity3d.com/threads/automated-game-testing.425806/

 

I am currently working on framework that gives ability to find a GameObject and emulate some action with it. The framework calls Puppetry. It is a open source and the source code can be found here.
You can find a GameObject by its name (or partialName) and place in the hierarchy and then emulate Click, SendKeys, Swipe, DragTo, etc.
To make asserts and waiting for conditions, you can check if GameObject is present, is active and get its component by name of component.
Basically, tests with Puppetry should be executed in Editor, but also you can adapt it to test on built version.
Additionally, you can use Puppetry with Appium, to test game on mobile device. For this purpose, you can get Coordinates of the gameObject by Puppetry and then make click on these coordinates by Appium.

More detailed documentation is here.