A new playing field
If 2021 has been the year of anything in gaming, the shift of focus to networks is surely a big headline. Quite suddenly, across the CSP (communication service provider) space, ISPs and mobile operators are playing catch up, or planning new offerings that serve the notoriously demanding needs of the gaming audience. Or, in many cases, both.
Cloud Gaming, of course, has rightly captured a lot of our attention. But its undoubtedly immense promise still lies some way in the future, and through cold commercial eyes, it would be easy to make the case for “watch, wait and see”. After all, current estimates for cloud gaming’s growth, even with a CAGR of close to 50% from 2021, suggest a market value of little over $5bn by 2025. Certainly a lot of money. But not to every player in the ecosystem.
Clearly, there’s more going on here, and we don’t have to look far for much larger and immediate trends. It’s well-known, for example, that mobile games already represent 50% of global gaming revenues, and there are currently some 1 billion online gamers worldwide. And, startlingly, in 2021 over 50% of gamers report playing an MMO weekly.
Upping our game
So game performance over networks has landed, and landed hard. Naturally, GameBench is supporting many of the leading ISPs and mobile operators, as they get fully to grips with gaming – no longer as just a proxy for overall network performance, but as a critical market sector in its own right, and a driver of critical new offerings, revenue and future growth.
In a previous Performance IQ bulletin, we asked “What Will it Take to be the #1 Mobile Network?”. The overall conclusions – derived directly from our own extensive international work this year testing major networks against real game play – were stark and sobering. 5G, at least for now, is far from the magic bullet that has commonly been assumed. And while, as we would have anticipated, ISPs tend to perform well overall, most, if not all, mobile networks have a way to go yet.
Another league
Performance measurement is nothing remotely new, of course, for any network operator. But, as our own clients have rapidly come to see, the traditional metrics and methodologies that serve more than adequately for communications, and even the less-demanding forms of streaming media, fail to measure what matters, uniquely and passionately, to the gamer.
This is due, not merely to these ever-increasing expectations, but to the complexity of the overall gaming ecosystem, which ranges across developers and publishers, through platforms and networks, to device manufacturers and silicon vendors. Each one of these verticals contains its own particular performance variables. Furthermore, each one of these variables interacts with all the others, to create a perfect storm of performance – and therefore, experience – variability.
We begin to understand the nature and scale of the gaming challenge for networks, as they engage with the requirements of this hugely lucrative, profoundly demanding environment.
Getting fit for purpose
Traditional network testing methods measure only the performance of the network itself, excluding other critical performance drivers such as individual games and devices and their interplay. They also calculate the game performance of the network using proxies that are more than adequate for traditional telecoms functions – voice, text and some streamed media – but unfit for gaming’s far more demanding requirements.
GameBench captures and computes user experience metrics using video capture software on the device during actual gameplay. As illustrated in Figure 1 above, these metrics accurately reflect the true user experience, unlike the data extracted using a traditional average or aggregated engineering approach. In Figure 2 below, we lay out the differences between network performance and gaming performance metrics in full.
| Metrics | Network Methodologies |
| Median Download Speed | Predict game user experience using indirect metrics – Median FTP downloads, UDP packet loss and jitter measured using a reference server (not the actual cloud gaming platform server). Measure RTT network latency during web and app testing to reference server. |
| Packet Loss | |
| Jitter | |
| Latency | |
| Mean opinion score (MOS) model | Predicted game experience MOS score using crowdsourced FTP download and ping data collected. |
| Metrics | GameBench Methodology |
| Animation Fluidity | Capture and store actual gameplay frame rate – calculate variability and sustained fps drops to grade fluidity experience. |
| Application Latency | Measure actual application gameplay latency (ie: Time between OS processing input and reaction rendered on screen). |
| Image Quality & Consistency | Calculate Fractal and Spatial complexity from captured gameplay images – calculate actual % degradation in image quality compared to a reference. |
| Launch times | Calculate actual time game took to reach gameplay directories. |
| Streaming Data Rate | Measure the actual network usage by the local client application to/from the cloud gaming platform server. |
| Figure 2. The stark contrast between the demands of traditional network performance metrics, and those required for confident game performance management. |
As we can clearly now see, not only are performance proxies an entirely inadequate approach for networks to measure and benchmark their current and future gaming experience, but the inherent complexity of the entire ecosystem, driven by the interplay between both new and emerging variables, demands an unprecedented degree of rigour.
Performance IQ by GameBench is a hassle-free, one-minute read that keeps you abreast of precisely what you need to know. Every two weeks, right to your inbox. Here’s a quick and easy sign up for your personal invite to the sharp end of great gamer experience.
And of course, get in touch anytime for an informal chat about your performance needs.
The intelligence behind outstanding performance
