All The Metrics That Matter: Part Two | The Key Performance Management Benchmarks

The four pillars of game performance quality

This is the second of our two Performance IQ bulletins dedicated to the essentials of GPM. PIQ 21.05 provided a quick reference glossary of all the fundamental concepts that every gaming professional needs to understand.

Here we go into greater depth, unpacking and explaining the key GameBench benchmarks that define and differentiate the “Great” (enthusiast), “Good” (standard), “Basic” (casual), and “Poor” (out of scope) levels of actual gamer experience. These are best understood using the four pillars of experience quality: Fluidity, Responsiveness, Image Consistency, and Duration.

Where did these recommended benchmarks come from? They’re derived from eight years of dedicated GameBench performance analysis of experience quality, based on real gameplay – over 70,000 hours of testing, across over 20,000 game sessions, with over 7,000 device models tested.

Fluidity

When the FPS dips below the developer’s target or becomes unstable, users immediately begin to notice stuttering images and lag in the game. The three key performance metrics here are Median FPS, Minimum FPS, and FPS Variability.

Median FPS

The middle-most frame rate during gameplay, representing the typical visual fluidity that the gamer experiences. (Higher is better.)

Minimum FPS

The worst frame rate experienced during gameplay, typically happens during a moment of heightened gamer activity or a computational bottleneck. (Higher is better.)

FPS Variability

The average jump between consecutive frame rate readings taken each second, reflecting the amount of variation in visual fluidity that the player experienced, (Lower is better.)

Responsiveness

When a game is described as “laggy” or “slow”, players’ actions are not immediately acted upon, and the immersive experience is naturally compromised. This is a responsiveness issue, and the primary performance metric here is User Perceived Latency.

User Perceived Latency

The measured time for a response to a user input, rated according to fixed benchmarks set by research. Example: time between a player clicking the mouse button, and seeing the muzzle flash of an in-game gun.

Image Consistency

Disruptions in consistency of image presentation (for example, macroblocking, a common distortion that appears as large pixel blocks) directly undermine the enjoyment and quality of gameplay.

Figure 1: Here we see results of around 0.33-0.39 for both the Note 10 and iPhone 11 Pro Max, reflecting artefacts typical of upscaling from a 720p native image to a 1080p display image without effective anti-aliasing.

Our Jaggy Index

Algorithms are used to monitor key metrics of image consistency across a gameplay session. Two complexity indices, fractal and spatial, help to spot differences like texture detail and lighting effects in a scene.

Our own GameBench Jaggy Index (see Figure 1) measures the degree to which low-res gameplay creates aliasing artefacts in geometrical elements containing straight lines. (Lower is better.)

Duration

This is the obvious cornerstone of mobile gamer experience, and while battery life continues to improve year-on-year, the need for carefully managed trade-offs between power and performance remains. Here the key performance metrics are Available Gameplay Hours and Loading Time.

Available Gameplay Hours

The average mA consumption of the whole system, measured during gameplay and combined with battery capacity (mAh) to rate the expected number of gameplay hours on a charge.

Loading time

The time taken to launch an app after it has been removed from memory, but is not freshly installed (i.e., a “cold launch” but not a “first launch”). Measured from tapping the game icon on the home screen to seeing the first interactive screen.

These two latest Performance IQ bulletins naturally go together, with a third to follow that combines both our GPM glossary and these key benchmarks. Keep an eye on your inbox!