What has happened to the QA in Unity?

Im not sure whether to carry on with Unity as my main development tool anymore. I feel the direction the company is going is the wrong one. After paying a lot for Unity Pro over the years (from version 3) I feel like the product has got worse. I now spend most of my time figuring or googling workarounds when the editor throws an error or some bug appears in the build, or Unity crashes.

I run Unity 5 on a Mac and it is still a mess months after release. Crashes for no reason, the “new” UI is buggy and requires a lot of messing about to get it working. (Masking is now broken on some screen resolutions). I also feel mobile development has been sacrificed for so called AAA features.

- Pro licence is a joke and you get nothing for the $5k you spend on it

- The terrain system is ancient and crap, runs poorly on mobiles still!

Recent problems while working on a project

The file 'MemoryStream' is corrupted! Remove it and launch unity again!

[Position out of bounds!]- Unity Crashes , Jesus even the bug reporter crashes!

- Asset store shows up white can’t load anything
- Android runs slower than unity 4

sorry for the rant. Just frustrated at spending so much time working around all the bugs.

People say submit a bug report. How often should I do this? Every day? How do I submit a 3gb project, ridiculous.

 

Sounds like the complaints of someone who is trying to get something done... ahahah i love it

 

Are you able to recreate any bugs in a new project?

 

A 3 GB project for a mobile? Wow.

The best option is to vote with your money. There are decent alternative to Unity now. It's up to you to choose if porting is worth eliminating the frustration of using Unity.

I personally don't have that many issues with the engine.

 

Agreed. The grass is not always greener.

 

Hi Matt,

I like your feedback. I think it's honest and Unity needs to hear it. If you still want to submit your project, Unity can set up an FTP for you to drop it off, or find another way to meet you halfway. There's a ton of bugs in software as complex as Unity and they're getting nailed each day.

Sorry yours isn't yet nailed but if we can get your project to Unity, something can be done at least. Also, why not tell the forum what bugs you have - perhaps there are known issues?

 

Ha yes that was a real rant. Feel better today. I know it's not the fault of Unity but I live in Sydney, Australia which is a tech backwater even in the city centre. Im not kidding it would take 70 hours to FTP a 3gb file!

I guess I just feel Unity should concentrate on a few platforms and get the bugs ironed out, then I mean how many people develop for PS vita, Blackberry and Windows store etc.

 

Is it possible to reproduce the problem with a small scene? Again, I'm sorry you're going through this. I'm not paid by Unity or anything, but I do appreciate what you are going through and have the luxury of a little bit of insight being a moderator, in how Unity works and their staff. They really are a bunch of cool people who do go that extra mile, so it's easy to get angry if you can't see the human side of it.

If it's possible to make a small reproduction of the issue, ie just a test scene or smaller part?

Let us know what can be done to help.

Off topic: Unity had a massive uphill battle to fight on multiple fronts, they had competition from Unreal Engine, Cryengine, everyone asking for different things (hobbyists, professionals, 2D games, 3D AAA techniques, better performance, better everything, more platforms) and they had to do this in one year (for Unity 5) - so that's an incredible amount of pressure, things can go wrong but we're here now and it's just a matter of fixing everything up and bug reports are such a valuable and essential part of that.

 

Do you use a remote server for version control and/or backups? You could simply pass the Unity devs a username and password set aside specifically for them.

 

I'm gearing up for an Android release soon and I feel your pain. Shadow performance is terrible and trying to get lightmapping looking 'correct' on an Android device has been nothing but painful which has cost me countless hours of development time.

I just upload a repro scene that highlights the issue, usually you can strip out art, scenes, plugins etc and anything that doesn't need to illustrate the issue. I use my OneDrive account to sync any uploads, and for massive projects I would just let it sync overnight, then send Unity the download link in the bug report.

Also I'm sure you are aware you only need to send your project's 'Assets' folder right? You don't need to send any other files. A lot of people include the Library and Obj folders which are unnecessary.

 

Sounds like something an automated test on the client side should pick up.

 

So true! I spent over 15$k in assets and licenses for Unity over the last years. 2 months ago i decided to dump Unity and replace it with Unreal4 instead. I had 2 bug reports there on their forums and on both i got a DEV from Epic looking into it - 3 weeks later both got fixed.

The big difference between both DEV teams is that Epic actually makes games and they know what you need. With Unity i just feel lost as a customer. I tried ordering premium support which costs you about 10$k a year and they needed over 3 months just to send me an offer - for UE4 you'll get a better support for free...

For the XB1 Support i had to set the MS Staff on CC when sending an email to the Unity Staff to get a reply, awfull. And after working with the UE4 engine with every week i just regret i didn't change the engine earlier.

The engine is degrading, too many things are broken or not on industry standard (terrain, shaders, speedtree implementation, GI etc etc etc) and support is awfull.

 

People exist that check in Library and Obj files to source control, nuff said

 

The source control isn't a part of Unity, though. The bug reporter is.

 

AAh I get you now, I was talking about submitting a seperate download link, not using the bug reporter attachment feature. So yes, it shouldn't include library and obj folders, or does it?

 

I've never checked, so I don't know. I expect that the built in collator would get the minimum required files. I was more thinking about when people supply their own archive.

 

I agree with @mattSydney here.

Unity seems to be crumbling under its own size : my bug reports rarely get any coverage ; even the simple ones about typo & errors in the documentation are ignored ; Feedback site is a 4 years old cruel joke (or maybe even more old, I arrived on Unity 4 years ago) ; new features keep being added while the existing ones are still incomplete years after their 1st release (eg: Mecanim that was released in v4.0 and needed to wait v5.1 to be finally relatively feature complete).

Personally, I wouldn't recommend Unity to a team. As a single developer, it's still a good choice, but one has to plan for the inevitable roadblocks he'll get if he doesn't use Unity in the generic ways it was intended to be used (or not so generic => can't even redefine the input from within the game..!!).
So Unity will make you save years of development, but will also likely make you lose months of development to bypass its issues & shortcoming, and recreate from scratch a feature that you had thought to already be in Unity, but was unusable because it missed something indispensable (and often it's an obviously needed thing).

Just lately, I got 2 annoying bugs : 1 has been reported 1 month & half ago (with a project, but not by me) and there's still no fix so far ( => http://forum.unity3d.com/threads/update-to-unity-5-1-too-bright-gui.332655 ), and another one, with a project by me, 2 weeks ago, and still didn't get any answer from Unity (about previewing in the Editor small audio file, which gives totally bugged sound).

Always asking "post a project" when reporting a bug, when a normal project will be several GB big, requiring several days to upload, and containing a lot of confidential data (I'm not willing to give my source to anybody) is not a good way to proceed.

Asking to "create a project" is also not a good way ; it takes time, and as right now it feels almost no one check the bug reports, it feels like a total waste of time anyway

v5.1.2 seems to be the 1st version ever that I can publish my game in good conditions (if no nasty bug arise, I still didn't check too much) ; I started to work on my game more than 3 years ago, and I'm near release, so it seems I got lucky, but I can't imagine the stress I'd have been if I was ready to release 1 year ago, as what I had in mind wouldn't have worked in U4 without a lot extra painful & sub-optimal bypassing.

And no, I couldn't have known at 1st that it wouldn't work in U4 because the documentation didn't have a word about the limits I hit during development (about physics engine).

Did I mention the lacking documentation ?

Anyway, I could go on for hours about all the shortcomings of Unity. If you want to see some of them, you can check my post history, as usually I post about them in the Forum...

Thus said, it's still a great engine, it's just the unfinished parts that are (really) annoying !

 

The terrain system has always been a bonus extra. Go model your own terrain in one of the 1000 tools out there. Or even use Blender + a heightmap generated from somewhere.

 

As a developer, can you suggest a way for them to more efficiently get the information they need? If they can't reproduce it they can't fix it.

I agree that uploading projects is an utter pain for exactly the reasons you've mentioned, but I can't think of a less painful way to send an actually useful report.

 

@angrypenguin,
1- I always provide steps to reproduce the bugs I report ; in ~30% of the cases, it should be enough for the programmer that created the bug to find out what's wrong without actually reproducing the bug ; in another ~30% of the cases, it should be enough for the programmer to do a mini project by himself and actually reproduces the bug and fix it ; in the last ~40% of the cases, then they could request a project
2- it should be possible to post a project without its textures nor its sounds ; this would save easily 90% of the upload size ; it should also be possible to obfuscate the code (with the possibility to select to keep in clear the scripts related to the bug, of course !)
3- Issue Tracker search should be integrated into the bug reporter, so it'd be easier to know if the bug we want to report is actually already known or not (and the search should be enhanced) ; we could then add notes to the existing report instead of creating a new one

I fixed hundred (if not thousands) of bugs in my games based on report from my users, so I get a sense of what is needed.

I paid for Unity, and I take time to fill reports ; asking me to create a sample project is too much, especially if I don't have the guarantee it'll be useful and lead to a quick bug fix.

 

No, you are wrong, the Terrain engine should be standard in a game engine, as should visual scripting and a robust GUI

 

What other games engines provide these three things, so that it should be called standard?

 

I still believe Unity is the best and engine for small teams I just feel some changes in the QA and the halting of feature creep that is affecting the base product. I remember when I first used Unity 3.x and I couldn't believe how stable and easy to use the program was, i never worried about errors or half implemented features (to my memory anyway!)

But as Hippocoder says they are having huge competition from the likes of Unreal, but I feel Unity's strength is in the mobile market, it is never going to be used for AAA games.

 

2 and 3 are pretty reasonable. I don't think it's pragmatic or realistic to expect Unity staff to make projects for themselves to reproduce bugs, though. Imagine how long that would take even for your pretty decent bug reports. Then consider that the vast majority of bug reports won't be so solid, perhaps not even including a bug. You're suggesting that a programmer makes a new project from scratch to explore every possible bug, instead of clicking "Open Project" and then running it (which will already take long enough).

 

I'm pretty sure UE4 and CryEngine has these?

 

Only Unreal 4 provides all three. Crytek lacks a visual scripting tool.

 

Hi all,

I don't think Unity is crumbling under it's own size. I've outlined briefly what the problems are in my above grey text - chiefly that it's been a massive engineering task and its ongoing. Every beta improves with hundreds of bugs squashed. This isn't the Unity of 2013 or 2014, it's really actually doing what we're asking - just we have to be patient.

 

I think Unity 5 has been more unstable than previous iterations. But that said, it's getting better and better with each release. It doesn't crash as often. Though, I suppose some of the crash-prevention-rituals I have developed may play a part in that. I'm not really sure as I do them subconsciously.

 

The license cost topic has been beat to death already, you can look up threads about it and make the choice for yourself depending on your project size and expected profitability for your specific project.

Unity 5 crashes sometimes, there have been big changes and many systems have been completely revamped. If its stopping you entirely maybe you should consider rolling back to 4.x. The only crashes I have personally encountered since 5.0 post-beta have been related to light baking.

If you upgraded a project from 4.x to 5.x you should expect to make significant changes to compensate. Many projects run slower until properly optimized.

The built-in terrain system is quite outdated. You can find extensions on the Asset Store to suit your needs. It would be nice if there were significant upgrades made to the built in system immediately but it isnt happening immediately.. However while I don't think it is on the immediate roadmap they are indeed working on updating it for some time now. Until then use the relatively cheap 3rd party Assets and middleware as this is what everyone else does.

Bug reports are expected to be systematically reproducible. You should be tracking down ways to consistently cause the error, then narrow it down as much as possible, eliminate unnecessary data, then submit a bug report. Submitting a crash with a project attached and just saying "i crashed" is not even remotely as effective as a good bug report that you have spent the time to investigate the cause of.

 

We're making massive efforts to fix bugs, actually this is our top priority and we'd rather sacrifice the inclusion of new features than release something riddled with bugs.

The sustained engineering team is dedicated to creating patch and dot dot releases which only contain bug fixes, we churn them out at a pretty crazy rate http://forum.unity3d.com/threads/unity-patch-releases.246198/

Thomas Peterson aka @QA for life goes through everything that we do in our Keynote at Unite Europe

But still after all the effort we put in, there are still bugs of course. Unfortunately we get far more useless bug reports than useful ones, often users will not write clear reproduction steps, it's not actually an issue with the editor, there's no repro project attached etc.

We know creating a repro project and narrowing down the cause can be a hassle but it's essential for us to be able to reproduce and pin point the cause of problems. Most of the bugs you find are the result of a specific set of circumstances that cause the issue. I'd love to say we had enough man power and time to make new projects for every bug but we just don't, it's not feasible.

Also, if you're a pro user, please please please try the beta, tell us what is wrong and what is right, this is really important to us http://forum.unity3d.com/forums/5-2-beta-forum.89/


This is actually not a bad idea, we definitely need to improve search too.

 

Actually no it doesn't.. Out of all major engines, Unity is the only one lacking:

http://docs.cryengine.com/display/SDKDOC2/Flow+Graph+Editor

 

Mind you I did manage to get a look at an early beta of Unity's new terrain system . Here >> goo.gl/ysX4cA

 

Then what ? When throwing a 8k heightmap or even just 4k at Unity you'll spend days at splitting, slicing, building a proper LoD around, you'll struggle putting trees on it because either SpeedTree in Unity is actually SlowTree or when using native Trees creating multiple terrain tiles on the same position because the billboard atlas blurrs your trees.

While in Unreal4 you import the heightmap and it's done, it asks you for some parameters like how many tiles you want and features you want to be activated. So there is a workflow of a many hours against a few seconds.

And don't get me started on painting the terrain or importing splatmaps, remove tiling etc. You'll need assets like RTP and TC to make it not look like it's 1998. In UE4 you'll have a visual material editor that allows you to creating stunning looking scene in a few hours... It's not even about the graphical engine features, it's about having basic tools available.

And once you are done with that good luck building GI for terrain... Lots of crashes and hours wasted due to adjusting settings "to get it somehow done without crashing or lagging the scene to death".

You can create stunning visuals in Unity but the workflow to archive that is way too time-consuming even if you know exactly what you are doing and how to archive certain effects.

 

I appreciate the effort, but not all issues can be submitted as repro. For example, in my big project, an empty scene takes 40 secs to run when I pressed Play. The Asset folder is about 30 gb, and It's a fairly high end system, and I tried with several high end systems. In fact , many of my projects gets slowed down after a certain size.

Now I can't submit a repro bug, but if you had tried doing a project of such scale, I'm sure you will notice these issues, so the question is, are you being made aware of is it filtered out by QA.

http://forum.unity3d.com/threads/pro-users-are-you-getting-better-support.340187/

 

@angrypenguin,
yes, it's not possible for Unity support to build repro project every time, but they could do it in many cases, just by having standard projects ready to be slightly customized.
Here's a example : the too bright textures with Legacy GUI issue. Having a standard project using the Legacy GUI and a couple textures in it, they'd need less than 5 minutes to reproduce the bug : it'd be as easy as adding a GUI.Drawtexture() call in their OnGUI() and changing back & forth the linear/gamma color space in the project settings.
This technique would work with a lot of small bugs that don't happen in too special & unique cases.

 

So it'll make the easy-to-catch bugs only a little bit harder to catch? What about the hard to catch ones?

 

@angrypenguin,
the hard ones are the ~40% cases left, where Unity team can ask for a repro project without the textures nor the sounds..!

 

Programmers don't look at incoming cases, QA do and they don't have that deep knowledge to know what's up behind the curtains.
I don't mind creating a project if precise steps are provided, unfortunately often some little things are missed out in those descriptions, which lead to not being able to reproduce the bug and still needing to ask for it.

Furthermore, if you are getting some error - well, you already have a repro for it.

It's not automated right now as it's impossible for us to know what is actually needed to repro your exact problem, but you can right click the scene where the bug occurs and use export package. This will extract only that single scene, which usually is quite small.

You are completely right and thinking in same direction as we do. Search in issue tracker and few other unity related sites (forums, answers, docs) are already being integrated to bug reporter and I've already tried first version of it.

Furthermore, we are continuously working on the crash analyser tool, which groups all the crashes based on their stack traces. Even the current prototype of it allowed us to close more than 2 thousand duplicates of crashes last week while providing explanation, workaround or information on how to fix it to users.

 

That's the best case scenario we hope for and really love all the people who do that.

It's simply a question of time usage. Where have been times where I spend a day or two investigating a single case due to it having huge project and not always reproducible problems.

Alternatively, I could have looked at couple dozen cases with small repro'es.

 

I'm quite certain that we have a number of tests for big projects, ranging in size from 3gb to 32gb and amount of assets from 4k to 87k.

I'll ask person responsible if he faced any similar slowdowns.

EDIT: Nop, not a common issue. Do you have any editor scripts which might perform actions on entering of play mode?

 

The point of one of my earlier posts is that I can't send my project due to size & confidentiality, thus the need to create a small project that reproduces the bug. Plus you don't want to jump into the mess that is my full project...

It's a good idea, but unfortunately, it doesn't apply to my game : my whole game is procedural (more exactly, I have my own map editor & my own map files).
But you should add a button for that in the bug reporter, with an estimate of the resulting size, it might help people to know they can do that and convince them to do the upload.

 

Well, yes, so far getting repro from users is our best way to make sure we find a bug, but our focus right now is on visibility and removal of duplication, so that only one person has to go through the bug reporting and everyone else can just find the issue on issue tracker and add his comments/concerns.

Preview actually does exactly that, it shows you all files included and their size.

We have a number of other features already being added to bug reporter and a big list of future improvements planned. Everything to make the process as smooth and painless as possible.

 

It sounds a lot to me like a manual page on "How to easily make a bug repro project" would be super useful here.

You've already written a whole bunch of stuff in this thread alone which is really neat and that I, as a Unity user of many years, didn't know about. (Eg: Select scene -> export package -> you now have a much smaller project to repro from.) It also sounds like you've missed out on a bunch of potentially useful stuff, which people have put time and effort into, because of simple misunderstandings (like using a link instead of uploading directly).

So a how-to focused on that might help a lot of people save a lot of time and get you better repros with minimal engineering effort, if any. I'm sure your QA team could write that document, and they'd know all of the best things to do and worst things that should be avoided from the bug reports they get every day.

If/when that document is written, it should be linked first thing in the bug reporter tool, and maybe even have a check box confirming you've read it before the rest of the form is accessible.

 

It seems you didn't get what I meant, so let me rephrase it.
I don't want to do : Select scene -> export package + select the file in the bug reporter, and then finally know how big it is.
I want instead a big button in the bug reporter window whose explanation or tooltip is : "remove the project folder from this bug report and instead create and attach an asset package from the current scene, whose approximate size will be XXX MB".
So I'll need 3 seconds to read all the information I need to decide if I want to proceed with this way or not...

Bonus : everybody will become easily aware that it's a possibility to get a smaller upload.

 

@SpiriTx, I found the cause of slowness. It was a Assetstore package searching through the whole project folder to find one of its setting file.

However, this is the topic of not just only submitting to QA, but how are bugs reports being looked at and fixed. If the bugs are left unfixed for years, of course you keep getting tons of bug reports on same bug and you get overwhelm.

 

I'll offer to add something like that to http://unity3d.com/unity/qa/bug-reporting

 

We can't blindly export package, because we don't know if isolating the scene will still reproduce the issue. That's why manual check if it still reproes in empty project is required from user side.

Yes, that's a very good point. We are addressing this by making bugs public with Issue Tracker and planned integration of search into bug reporter to remove all the duplication and make information transfer easier and more effective both for us and users.

 

I think Unity 5 is a BIG step forward for Unity. There are bugs in every engine. The guy who said UE4 team fixed his bug after 3 week, is lucky, there are requests in UE4 forum for improvements for about 1 year, and nothing have been moved. You see, open source have it's disadvantage, you need that, go make it, you have the sources. This is absolutely true, except, nobody can spent so much time to learn millions of line codes to make an improvement.

Flow graphs are a bit outdated, I sincerely don't want something similar in Unity.