On June 8, 2018, Rob Pedrosa, Director of Operations and Services for Hi-Rez Studios felt good. Realm Royale had gone live a few days prior and everything went silky smooth.
He’d planned enough capacity for the launch and his liveops team was only dealing with minor issues. All was well, until he got a call from someone on his team with words that all CEOs want to hear but Ops teams do not.
“Ninja is playing Realm Royale! This is going to be huge for us – everyone is going to download and play!”
She wasn’t wrong. Within those first few hours, player numbers increased by well over 1000%, and kept rising. So how did Hi-Rez keep up with server demand, with no scaler?
“I didn’t sleep for 72 hours!” recalls Rob, with a smile. He and his team called every server vendor they knew to get their hands on as many servers as possible, desperately trying to stay ahead of the skyrocketing demand curve. “My team worked miracles, but we couldn’t keep up and, ultimately, that cost us. We’ll never know how many players we lost to queues; but regardless, we vowed it wouldn’t happen again.”
Once player numbers returned to manageable levels, Rob spoke to the CEO and said they’d need 40 more people if they were going to continue to operate their server fleets using bare metal servers, as they used back then.
But after investigating a number of solutions, including scaling up the team and speaking to leading public-cloud providers, Hi-Rez settled on Unity's Multiplay instead.
“After a few hours with the Multiplay team it was clear they weren’t just knowledgeable about servers, but also about game development, and what it takes to do both right.”
Fast forward to July 2020, and Hi-Rez has just launched its most adventurous title yet: Rogue Company. This time though, they’re ready for anything (even Ninja).
Rogue Company is a third-person, tactical action shooter developed by Hi-Rez’s First Watch Games studio. As a Rogue Company mercenary, a player grabs their weapon of choice and dives into iconic locations to compete online in various PVP game modes.
Launched in open beta on July 21, Rogue Company presents a number of challenges for the studio in terms of capacity planning. With a multi-million dollar marketing campaign, a hungry player base of 70M existing players across 10 titles, and a launch strategy that includes every region on the planet, how do you forecast?
“Forecasting a new venture like Rogue Company is next to impossible, but with Multiplay we’re not worried. Their Hybrid Cloud gives us the flexibility to scale up and down with demand as needed,” says Rob.
Multiplay Hybrid Cloud works with 150+ data centers worldwide and all major public clouds for resilience and performance at scale. As more players join, the scaler allocates more servers and, if players decline, servers are deallocated. This multi-cloud approach means no player is ever left waiting for a game.
This, combined with the fact that the whole operation is managed 24/7 by a distributed support team, means the Hi-Rez Ops team can focus more on the quality-of-life aspects of liveops.
When Hi-Rez launched its first game, Global Agenda, they turned to Vivox for voice comms support and haven’t looked back.
“The best thing you can say about any backend service, like voice, is that you don’t even notice it’s there. Vivox has been part of our stack since we launched back in 2005 and I don’t think I’ve ever raised a ticket with them, so they must be doing something right!”
Vivox is the voice chat solution in all Hi-Rez games, including SMITE, Paladins and Rogue Company.
The tools and services Hi-Rez use are part of Unity’s growing suite of multiplayer tools. Multiplay is a battle-tested game server hosting solution, which scales servers for Hi-Rez as needed. Vivox is a trusted voice comms solution, which delivers crystal clear comms for Rogue Company players. Learn more about both here.
This interview is part of a profile of Hi-Rez Studios and their backend infrastructure for Rogue Company. Learn more about how Hi-Rez works with Unity by downloading the case study here.