HOW BUNCH ESTABLISHED THEIR SDK QUALITY STANDARDS

Jason Liang & Saniya Anand - Bunch.live

Special Guest Post by Jason Liang, CTO and Co-founder & Saniya Anand Quality Lead of Bunch.Live

Bunch offers players the ability to video chat with friends while playing games on a mobile device. Not only does the app offer first-party game content that players can launch right away, but also it supports third-party games that players love, such as BrawlStars and Call of Duty Mobile.

When it comes to mobile gaming, performance is a critical component of the gamer’s experience. While Bunch aims to enhance the playing experience by facilitating a social element through which players can video chat with friends in the heat of the action, the team also seeks to minimize any adverse performance impacts on gameplay. As well-known management thinker Peter Drucker once stated,

“If you can’t measure it, you can’t improve it.”

Our customers need to know their players will have a great time playing their game and chatting with their friends free of performance problems. Therefore, we must understand the performance impact of our SDK in great depth.

In the search for a tool that will allow Bunch to measure performance impact systematically, we are looking for a tool that can profile apps on iOS and Android devices and enable us to measure game performance with and without the Bunch app features. The metrics include FPS (min, max, avg), FPS stability, CPU usage, GPU usage, device heat, power consumption, and network consumption.

Various tools measure one aspect of the performance but not another, and some of them only focused on one platform. GameBench provided these numerous measurements for iOS and Android, collected hundreds of hours of data, and used said data to focus our solutions and optimizations in the video chat layer. While using GameBench, we observed some unexpected results; however, this has provided valuable opportunities for the team to reevaluate fundamental assumptions that we would have otherwise neglected.

While using Bunch android video overlay, one of the unexpected results was a decrease in CPU usage for some of the games tested. The team was ‘perplexed’ by this result. It only makes sense if the CPU usage is going up due to using more phone features. The team realized that the video overlay might have a constraining effect on FPS which means the system has less work to do, resulting in a lowered CPU usage when Bunch is enabled.

The team is now taking the investigation learnings further to understand the limitations and working on new solutions to better the Bunch SDK integration.

We can now fully integrate the performance measurement process in our QA team with the GameBench team’s support. The Bunch QA team first completes a series of tests for a game with various time lengths without Bunch running and then three comparison tests with:

Bunch running in the background with just voice chat on iOS and Android,
Bunch video running in the foreground as an overlay on Android only, and
Bunch video running in the game as an SDK on iOS and Android.

Figure 1: Game Performance Summary with Video On showing FPS Chart and CPU Chart during Call of Duty Session

Figure 2: Game Performance Summary with Video Off showing FPS Chart and CPU Chart during Call of Duty Session

Taking advantage of the cloud dashboards and our internal test plans, we can execute and export raw data files into our data warehouse for subsequent measurements and further analysis. Although many of the current test results are pretty good in terms of the minimal performance impact Bunch has on most game sessions, we must continue to find opportunities to improve performance with new software updates. We feel very confident that these performance tests can be reliably repeated on GameBench by our QA team in a uniform manner across iOS and Android as Bunch continues to grow.

Wrap-up

GameBench allows us to profile each of the Bunch apps via wired or wireless connection types. Each type offers multifarious capabilities; battery usage is easier to capture in the wireless mode, and the wired mode allows the test to run without network instability concerns. We may consider alternative methods to measure battery usage and temperature to take full advantage of the wired mode.

However, as GameBench improves these capabilities and compatibility in the wireless mode, we will probably do more. We also look forward to more support for odd Android devices that behave differently during testing, speeding up support and documentation for the latest operating systems on iOS and Android, and further stabilizing capturing sessions to reduce wasted testing time.

After learning from the various “gotchas” associated with benchmarking on different phone types, we are excited about the possibility of automating these performance tests in the future as GameBench continues to improve on the toolset and connection mode. These automation tests will allow us to continuously monitor the performance metrics of products and services for third-party games and guarantee that our players will always have an amazing party on Bunch.