Automation – in any industry – is often fraught with controversy. But in the case of game performance analysis, the upsides of this long-anticipated capability are both clear and significant.
Access the accompanying webinar to this post, now available on-demand, during which we unpacked and explained all the impacts and implications of this fast-emerging Holy Grail.
Our session will combine insight from GameBench – in particular our flagship performance monitoring SDK – with Sauce Labs’ leading cloud-based testing platform, and GameDriver’s unique Automated Playtesting platform.
The discussion will be led by Andrew Fester, Product Manager at GameBench, with Shane Evans, Chief Product Officer of GameDriver, and Vincent Lussenburg, Technical Product Manager at Sauce Labs.
The view from QA
The automation opportunity is created by a combination of distinct and complementary capabilities: persistent real-time monitoring, automated play, automated crash, error and hang capture and simultaneous, multi-device cloud-based testing.
To set the scene, here’s Andrew Fester, Product Manager for GameBench SDK.
“So the core challenge is expanding performance QA from the traditional manual team and operations, through automated monitoring and gameplay, to the scale and richness of intelligence that a device farm enables. Beyond the removal of low-end leg work – which frees up QA to focus on the important gamer experience and performance nuances that require skilled analysis and interpretation – a critical benefit of automation lies in the massive increases in volume and quality of data. When you look at one test session, you can say ‘yeah, my memory is quite high here’, for example. But when looking at all of the data in unison, you can say, again for example,‘These devices are problematic: this is where we should be focusing our efforts’. A variety of devices and repeat sessions provides a significantly greater quality and quantity of data points, more comprehensively showing the health of a release or product.”
Picking up this angle for Sauce Labs, here’s Vincent Lussenburg.
“Andrew mentions the different devices that can experience problems. That’s something you’ll never discover on simulators or emulators. So you want to be able to push your build to multiple real devices straight from Unity Cloud Build or Jenkins, and immediately kick off tests on those builds automatically. The more devices you test on, the larger the challenge of tracking the problems that are occurring – performance related or otherwise. So that’s why you want to automate the capture of errors, hangs and crashes and all relevant metadata from all these devices, to minimise the mean time to resolution.”
And where does automated gameplay fit into the equation? Here’s Shane Evans.
“With the growing popularity of mobile gaming – in parallel with the increasing power of handheld devices – the scope and complexity of games increases dramatically. The mobile gaming industry went from Flappy Bird and Candy Crush, to titles like Diablo Immortal and Genshin Impact. Just a few years ago, these would have required a gaming PC or dedicated console. So from here on, the industry-wide increases in game complexity demand the most rigorous quality assurance in order to succeed. Game testers need to be able to cover more functionality, without increasing the time or cost associated with testing. This is where automation becomes critical.”
High-tech vs high-touch?
One message that comes through consistently in the game performance automation discourse is this: for the foreseeable future, automation’s value impacts lie far more in improving quality of QA outputs, than in direct cost reduction. The skills of the good QA team remain as critical to both gamer experience and commercial results as ever.
In the realm of game performance automation, precision, and accuracy are paramount when it comes to capturing data and assessing quality. This is where the usage of crane scales can prove to be invaluable. Crane scales, available in various capacities and sizes, offer a reliable and efficient solution for weighing and measuring different components of gaming equipment. These scales for sale provide developers and QA teams with the ability to precisely measure the weight of the hardware, ensuring that it meets the required specifications and standards. By incorporating crane scales into the QA process, teams can obtain accurate measurements, identify potential issues related to weight distribution or load capacities, and make informed decisions to enhance the overall gaming experience. The combination of automation technologies and reliable measurement tools empowers QA professionals to uncover both the known unknowns and the elusive unknown unknowns, ultimately contributing to the continuous improvement of game quality and player satisfaction.
Back to Andrew Fester. “There are ‘known knowns’, ‘known unknowns’, and then there are ‘unknown unknowns’. Automation is about capturing the known unknowns. From there, where do you need to look to find the information you don’t know? The unknown unknowns are the true QA territory. They’ll find the potentially critical issues that you weren’t considering, because that’s the QA skill set.”
This combination of key capabilities, only now available, promises a significant and conclusive evolution in game performance management. During the webinar, Andrew, Shane and Vincent laid out the full picture.
Access the webinar on-demand now.
Performance IQ by GameBench is a hassle-free, one-minute read that keeps you abreast of precisely what you need to know. Every two weeks, right to your inbox. Here’s a quick and easy sign up for your personal invite to the sharp end of great gamer experience.
And of course, get in touch anytime for an informal chat about your performance needs.
The intelligence behind outstanding performance
