Where are the Unity performance testers?

simple question - Do the QA even run performance tests on each Unity release. Its just painful after every release to see people writing in forums frame rate dropped. No good for us mobile developers.

I would like see some post on this from Unity or some explanation?

 

Every version will have quirks. That's why many experienced devs don't change versions mid project.

 

That isn't necessarily true. Now if you were talking about changing versions Unity 4 to Unity 5, then yea that would be something to avoid mid-project. But developers don't disregard updates and patches, which get released here every week. Sometimes something is broken or less-than-optimal in the first release and in x.1 or x.2 that fix is implemented.

 

I'm talking more about the QA testers, dont expect developers to do performance testing on top of there development work, but just do good practice code

 

All I will say is that with the UI API changes Unity made in 5.2.x they either didn't do even a middling QA on the UI or they did and haven't advised us properly with the guidance, for example, "Oh, by the way, with Unity 5.2.x and above, you need to rewrite your Unity UIs."

 

indie devs dont count

 

Actually it's perfectly true. He did specify qualify that as "many", he's not claiming that everyone does the same thing.

"Disregarding" something and "choosing not to use it" aren't necessarily the same thing. @BoredMormon is right, at work (and at home for me) versions aren't downloaded and installed every week just because they're there. Those weekly updates are for a purpose, and that purpose is fixing bugs. If those bugs don't effect a project then there's no reason wasting time and effort, and increasing risk, just to be on the latest version.

Same deal with any other kind of update. The changes are evaluated, and if they're of enough benefit to current work then we roll out the update. If they're not then we don't spend the time and effort doing that because... why the heck would we?

It's not about keeping up with the Joneses. It's about equipping ourselves with effective tools.

 

I've personally got about four versions of Unity installed on my system. Mainly around not needing to update for some largish freelance projects on the go, but wanting the latest features for some other projects.

 

On my workstation at the office I have Unity versions all the way back to 3.x installed. Fun!

 

Elaborate please...

Confirm they do the below quick test:
Unity Internal Test Project -> Open in latest build of Unity before public release -> IF FPS have dropped -> Release to public anyway..

I'm not sure if I'm seeing something whole of Unity is not, but I know your on the forums each day, alot of performance issues, and you just keep telling customers to wait for patches each week, if we keep doing this then each week will have nothing out....Remember this is MOBILE development, we need all the performance one can get..

 

Some peoples don't really care about mobile dev, and only PC.. you see.. mixed opinions.

 

I understand that, but Unity should care considering everything moving to mobile these days

 

Okay that makes sense to have these.

But this is not enough at all, you need to have some real projects and do some tests to see how things interact when combined. It seems to me when the integration happens thats when things go bad.

My personal opinion is not enough is being done - you need Unity in house projects and check these projects with each new release. End day its just pretty poor performance testing from QA! (not getting the developers involved in this one)

 

That wasn't the point I was disagreeing with.

If you mean in the sense that you're choosing not to use it after looking at it, then yes. If, however, you aren't even looking at it before making that decision, then it's precisely the same thing.

I'm not arguing against this at all. In fact, I'm pretty sure I said something very much like it in my post. My point is not that we *should* be keeping up with the Joneses. I'm saying that developers aren't stuck with the version they have if they're mid-project. If it's between Unity 4 and Unity 5, then yes, moving from one to the other while mid-development is most likely too risky to do, but for patches that provide fixes that just isn't the case. Simple as that.

 

Personally, I learned the hard way not to upgrade mid-project. Had a game that ran perfectly on iPhone 4. Upgraded to U5 and apparently iPhone 4's unofficially not supported anymore because of the brand new GPU costs. *Sigh* Maybe it was about time for me to update my iPhone anyways.

 

My Unity 5 projects still run on iPhone 4S. I'm not sure how 4 and 4S differ, though.

 

I haven't tested U5 on 4s but an empty scene ran at 16 fps for me on the 4.

 

Unity are moving parts of the engine onto other threads. this is tricky business to get right and we need to give unity some time to get this working 100% perfecto.

 

Agree, but does not answer or help with the topic at hand..

 

He just gave a few examples, those aren't the only tests being run. You can really only test for certain things that you know will be in a large number of projects. What if they do have real projects, but there aren't performance losses? There are so many variables that it becomes impractical to test for everything that everyone could possibly be using before pushing things out. This becomes especially true if you add in all of the possible assets from the store or random blogs, without any guarantee of following coding guidelines or practices.

 

@MrEsquire

9:50 Goes over the types and number of tests that QA runs daily. There are 61 different performance tests as of a month ago, that does seem low granted but we don't know how much each test covers.

 

Thanks for the link, did you listen : 31.47, the guy in audience asks the perfect questions.
The answer from the QA Director = we need repo, or use the forums!!, click crash submit button.
I'm so happy he got asked these questions.

Even with this, something is going wrong, I can copy and paste you multiple threads in the forums, complaining about speed, ftp slowdown, choppyness, lag etc.

 

Yes, the 4 is a dead device really, but even so, for an empty scene that seems crazy, and really should be investigated.

Is the default skybox in 5.x doing something very expensive (by mobile standards)? - I'm still using 4.6, so haven't come across this yet.

For comparison, a few years back, I made this - using C++, Marmalade, and my own GLES2 rendering code - and it ran at 60fps on an iPhone 4 with only the occasional dropped frames. (I was rather careful with shaders and limited use of alpha blending, of course)

It does feel like mobile has become a lower priority for Unity - despite so many mobile developers depending upon it. With support for 4.6 ending (but many/most mobile developers still using it?), the lighting changes in 5.x still seem like quite a setback on mobile, with no clear gains from it - it's not as if any of the new rendering features are viable (if even supported) for real-world games on mid-range mobile devices.

 

I tried the BS demo out over the weekend and tried some of our larger stuff. In Game mode everything runs sweet, but noticed that in Editor everything just comes to a grinding halt and judders.

It's simple to re-create, you can just stitch together some 4096 X 4096 HM's in the terrain system. Or just load a scene up with a boat load of meshes (like a bit forest on a plane mesh). As said it runs fine in game (60+FPS, 15MS) and in builds.

Does this happen for others, or is it just me? Tried it on two machines, (Specs I7 4810HQ, GTX 980M, 32GB DDR4, M.2 SSD other machine was a 5820K, GTX 780TI, M.2SSD, 32GB DDR4).

 

I lock up the 5.2 Editor completely with an FBX that used to work in Unity 5.1 & prior. I found out it specifically a clash between Unity 5.2, upwrapCL, and Intel HD Graphics type GPUs.

But scenes that do work are getting better and faster...

 

Of course. Many of us use entirely custom shaders. But we still need (basic, oldschool, diffuse-only) lightmapping and lightprobes to work well. This is an area where 4.6 still appears to outperform 5.x, especially with issues like this taking a long time to be dealt with.

I'm sure 5.x will get there eventually, but as somebody working on a project with a whole lot of baked lighting, if feels like the end of support for 4.6 is coming a bit too soon...

 

Keep in mind that there is likely a kind of a survivor bias situation here. Those that experience the performance issues between version changes are much more likely to say something about it, than those who don't have performance issues.

 

How many people are actually upgrading projects, doubt its how many you think.
Because if its working fine with lets say 4.6 and customers having no issues then no need to.
Its always safer to start a new project with a new major build.
This way one would think everything is normal, its only those who had projects in the past working great and now wish to upgrade to newer features (Win10 mobile support, more mobile options) and maybe take advantage of other new features will really feel the change and difference as they know before there games running better.

If there was some kind of guide for smooth upgrading then that would make things better.

Currently I;m getting the feeling, customers are asking Unity whats going on with performance...getting back response its okay must be your specific project, send it to us so we can see it...

 

I'm simply comparing to other bugs (for example 5.2's various animation bugs) which seem to have lot more people reporting and commenting on them. Most of these reports seem to come from people who have upgraded projects.

But I acknowledge that performance issues are likely the last in the chain and so will conversely get less attention (if your app doesn't work its a lot more noticeable than your app running slowly).

 

I think there are two topics being discussed here, I do agree firstly on the fact if someone is having a breaking issue, they have to explain and submit a bug report if they feel its Unity related. Yes they should be discussed in forums because it could be something simple and user error, hence its not good to submit report and waste QA time without first checking forums for similar issues etc.

We clear on that, but lets discuss performance of each release. If ones game works fine and not broken in a new release of Unity, then technically one cannot complain that there are breaking issues. If they see performance decrease then one would investigate the issue, but then spend hours doing this and realise its not the project but something within Unity. Then looking in the forums, more and more people reporting similar problems...After long discussions on the forums/threads, what exactly has QA done about the existing reports? You say most are not constructive in the forum discussion of course as most are complaints.. But those which have been submitted? ? So you wish for people to wait till release 5.3/4 this is a joke as time is not on our hand. I just dont understand how with all these complex tests and QA processes one cannot open publish a Unity in house game with FPS counter on a Android phone and check before and after FPS. I think you just use the customers as indirect testers as you simply cannot get your QA teams to do such tests as it mean they have to do work that is no necessary or out there general working standards...

 

If you're building a new project and run into a bug, you'll probably work around it. Maybe stick to legacy anims instead of Mecanim (or vice-versa). Or maybe building an entire custom lightprobe system....

If you upgrade a project, it's more critical that the existing features work, as they become so deeply embedded within a project, avoiding problem features becomes much more work.

 

Check out my thread on a my simple Unity benchmark called Cube Mark.



It only tests basic rendering and physics collider performance, but there is nothing stopping you or any other Unity user from writing a simple benchmark that will test the performance of one or more of Unity's sub systems.

There is quite a bit of variability in the performance of Unity even for this simple benchmark.

Actually their WebGL benchmark is quite good as it tests a range of sub-systems for performance. UT should open it to the community so we can all check out the performance improvements with each version, or UT could publish the test results along with each release.

 

On the topics of performance I have high hopes for full DirectX 12 multi-core rendering due out next year.

And think that UT should provide parameters for their batching system that can then be tailored for/or have templates for optimal performance on different platforms.

 

What about hopes for Mobile device performance?

 

I'm starting to wonder whether the future of Unity would be better (at least for mobile devs) if it became two separate products - 'Unity Hi-Spec' (based on 5.x), and 'Unity Mobile' (based on 4.6)

 

Unity spent a lot of resources to just have one product. It would unnecessarily complicate the situation for people who deploy on multiple platforms.

@MrEsquire, what prevents you from identifying the bottlenecks or differences in Unity 4 and Unity 5 in your game, such that you can submit a bug report? Unity is constantly extending the performance test suite. There is a few blog posts about it. And I am sure they consider performance regressions and accordingly extend their performance test suite.

 

That's a terrible idea, and how Unity used to be when they first developed mobile support. It also solves absolutely nothing whatsoever.

If it's running badly for a mystical reason, it's due to a bug and everyone can help fix that with bug reports. In an ideal world no engine would have bugs, but every engine has bugs. Tonnes of them, because the software (os/driver) and hw is always changing, and people always want new Unity features. Unavoidable.

 

With Metal on IOS and Android adopting Vulkan they should both be good, especially as mobile CPU's are ramping up their core counts. It's just a matter of providing enough mobile GPU power and the current high end models have DX11+ features already.

Actually I think the mobile GPU's use tile based rendering systems that could work really well with multi-core/threaded CPU's, a dedicated CPU for each quad/hex/oct of the screen.

Mobile is catching up with desktop, so no point really as high end mobile is probably overtaking low end or older desktop systems. And the graphics API's are standardizing around multi-core low level APIs DirectX 12 and Vulkan across all platforms.

What we need are easier ways/tools to build a high level high performance product in Unity for the top level hardware that will scale elegantly on lower end hardware and allow for a range of input devices out of the box.

 

Is it, though?

Yes, mobile devices are still improving at a fair place. But desktop/console isn't standing still.

 

I see the performance gap reducing but not fully closing as the desktops can utilise higher power usage.

 

It would do little good for both of us, because I would go around the forums and find all the major bugs and performance submitted issues and paste you a long list here, and then you would reply by either saying they being looked at by QA or your not sure when they will be fixed, probably in next major release.

But thanks for the offer..

 

In terms of being hardware in user's hands - yes. People typically change a mobile every year, but a desktop every 5 or even 10 years. We're talking about real, normal people ie the majority. Not you. Not me.

 

Does that mean you didn't submit any bug reports on your own?

 

Personally I think the gap is getting a little large to maintain a good balance for mobile. If your targeting a Iphone 4 for instance the gap between that and what apple just released is similar to the PS2->PS4 in terms of ratios.

 

Personally, I still see more people using the iPhone 4S than 5(s/c) or 6(s) combined.
But after taking a look at the following article, in general, your statement seems to be quite realistic:
http://info.localytics.com/blog/res...iphone-model-with-the-highest-user-engagement