Search Unity

How to A/B test game difficulty with UGS Use Cases

June 15, 2022 in Games | 8 min. read
Different level bars
Different level bars

Perfecting your game’s design can be difficult when you can’t see how your players interact with it. By doing A/B testing, you can make design decisions based on how your players really play the game.

The Unity Gaming Services (UGS) Use Cases are a collection of samples that implement typical backend game use cases and game design elements, show how to resolve specific development tasks, and highlight the efficiency you can achieve in your game backend by integrating different UGS packages in your project.

One of these samples is about A/B Testing.This lets you segment players into multiple test groups in order to determine which version of a specific game element is the most engaging or intuitive to your players. The example we show is testing the amount of XP required for leveling up. This particular example is suited for something like alpha playtesting of a single-player game. But this is just one example, and there are many potential uses for A/B testing with UGS, even in published or multiplayer games.

What is A/B testing?

Through A/B testing, two or more versions of an experiment are compared to see which one will perform better. By presenting users with similar options, you can compare and contrast variants to determine the changes to implement going forward. A/B testing allows you to determine which optimizations of a variant will lead to your desired results. 

The two variants in testing are often referred to as the A variant and the B variant. To prevent any types of biases that may skew the data, users are randomly assigned to each variant.

How to implement A/B testing in your game

Plan your test

You may already have an idea about part of the game’s design that is ineffective. Maybe part of the user interface isn’t intuitive, or maybe a couple of beta test players have said that the puzzle in level 42 was way too easy and could be moved earlier in the game.

When planning your study, you’ll need to decide:

  • Which alternate configuration(s) to test.
  • Which data points to track to indicate which is the better configuration.
  • How long the test will run.

Configure base values in Remote Config

Using the Remote Config Dashboard, you can create key-value pairs to configure parts of your game.

Remote Config dashboard

Set up your experiment groups in Game Overrides

After your base values are configured, you can now use the Game Overrides Dashboard to configure variants for testing. You aren’t limited to just A and B, you can create many variations or combinations of changes and perform multivariate testing if you like.

Game Overrides dashboard

Update your code

Now that you have some remote values, some overrides, and some ideas about what you want to track, you need to make sure your game’s code is integrating these systems.

You can use the Remote Config SDK to download the configuration values to use in your game, as shown below. This will automatically download the Game Override value depending on which group the player ended up in.

await ConfigManager.FetchConfigsAsync(new UserAttributes(), new AppAttributes()); var abTestID = ConfigManager.appConfig.GetString("A_B_TEST_ID"); var abGroupName = ConfigManager.appConfig.GetString("A_B_TEST_GROUP"); var levelUpXPNeeded = ConfigManager.appConfig.GetInt("LEVEL_UP_XP_NEEDED")

(code examples based on Remote Config SDK version 3.1.0)

Finally, use the Unity Analytics SDK to send events to the backend for reporting. Each event can contain custom parameters to describe what happened. These parameters can be used for filtering reports later on. You want to make sure to include a parameter in your event to indicate which testing group this player was in when they sent the event:

var playerInfo = new Dictionary<string, object> {{"abGroup", abGroupName}}; AnalyticsService.Instance.CustomData("PlayerDidSomething", playerInfo);

If you are already using Remote Config for various configuration values, and have analytics events configured for important actions, then you could decide at any point to start up a new A/B test for one of those values at any time from the dashboard, without having to recompile the game and deliver an app update to players.

See the results with Unity Analytics

Using the Unity Analytics Data Explorer in the dashboard, you can create a report that shows the progress of your experiment. Based on this report, you can then make your design decision.

Unity Analytics Data Explorer

Finalize the design change

Now that you can see how your players actually play the game, you can optimize the game to be more fun and engaging for your players. Since you were testing something that was configured in Remote Config, you can simply change the base value and delete the overrides. You don’t need to release a new version of your game.

Get started today with our collection of samples

The Unity Gaming Services Use Cases collection includes several other samples apart from A/B testing:

  • Loot boxes: Reward players with a random Economy currency using Cloud Code to perform the Economy grants.
  • Loot boxes with cooldown: Grant players random collections of both Currencies and Inventory Items at timed intervals.
  • Starter packs: Allow players to purchase a Starter Pack using Cloud Code to implement the one-time-only purchase.
  • Seasonal events: Update game content remotely based on timed special events.
  • Idle clicker mini game: Update server authoritative game state in real time, similar to idle clicker and social games.
  • Cloud AI mini game: Server authoritative gameplay in a simple Tic-Tac-Toe game played against AI running on UGS with persistent state, Currency rewards, stats, and straightforward AI.
  • Command batching: Group game Commands into a queue and process on the server in a single batch to reduce the volume and frequency of server calls made during gameplay.
  • Battle pass: A seasonal reward tier system with a free track and a premium track. Learn more about using the battle pass sample in this blog.
  • Rewarded ads with Unity Mediation: Offer players opportunity to boost level end rewards by interacting with a reward booster meter and watching a rewarded ad.
  • Daily rewards: A prevalent engagement feature that can boost retention by showing players an escalating series of rewards incentivizes them to keep logging in to claim better and better prizes.
  • Virtual shop: Demonstrates a key feature in many games as it allows players to use in-game currency to purchase items and resources to facilitate a server-authoritative in-game economy with multiple store pages and server-managed badges.

Make sure to check back in the GitHub repository as this project will continue to grow to include more use cases for Unity Gaming Services.

To learn more about the Unity Gaming Services Use Cases, make sure to watch our latest bootcamp video that walks you through all the steps to get started.

June 15, 2022 in Games | 8 min. read