Search Unity

AAA vets share advice on setting up a scalable DevOps toolchain

March 9, 2023 in Games | 18 min. read
AAA vets share advice on setting up a scalable DevOps toolchain | Hero image
AAA vets share advice on setting up a scalable DevOps toolchain | Hero image
Share

Is this article helpful for you?

Thank you for your feedback!

Monster Closet Games is a small studio with big ambitions – and the experience to match. Most of the core team has been in the industry for 20-plus years, and they’ve worked on a number of gaming’s biggest franchises, from Assassin’s Creed and Prince of Persia to Far Cry and Halo. They’re currently developing an online multiplayer title codenamed Project Shrine, with plans to launch on PC and current-gen consoles.

“High-level, it’s a third-person co-op dungeon raider,” says Monster Closet CEO Graeme Jennings. “You and your group get together, build synergies between your characters, and raid dungeons for treasure. It’s about working together as a team.”

Teamwork is at the heart of Monster Closet’s approach to game development, and the studio plans to stay tight-knit and focused. “I’d rather have 40–50 developers who love working together, who love the way we work, and who build great games because of that,” says Jennings. “I have a genuine belief that great teams, with the right tools, can build great games.”

Monster Closet’s artists and developers are used to working with a powerful tech stack of proprietary solutions. Starting over from scratch meant this wasn’t an option, so in the first few months, the team carefully curated a toolchain that would scale with their ambitions.

For Project Shrine, Monster Closet is accelerating production with Unity’s engine-agnostic DevOps solutions and automations, including Unity Version Control for source control and Backtrace for error tracking. We interviewed Monster Closet’s lead online programmer, Patrice Beauvais, and CTO, Thomas Félix, to learn about what they’ve been creating and how they built a tech stack designed to scale.

Pic of team members working at Monster Closet offices

Creating multiplayer games and the need for source control flexibility

What did you consider when you started building your tech stack?

Thomas Félix: A few of us have had different experiences with live games, and we wanted to have a solid DevOps foundation that could support that in the long term. Even though GAAS [games as a service] is not at the core of our game, we wanted to make sure we had a powerful tech stack that would help us release and iterate quickly.

We all had experience with Perforce, but we’re not necessarily big fans – it works well enough, but we had been doing things that weren’t really meant to be done with it. We were also looking at Git as a solution, but then we found Unity Version Control.

On paper, Unity Version Control mixed the great approach you get with something like Git, but also something much more powerful like Perforce to manage data. We were seduced by Unity Version Control’s task branch workflows; after about six months of evaluations, we decided to give it a try. At that time, we were around six or seven people. Because the team was growing slowly, it was a nice, smooth ramp-up. We’re now at around 43 team members, and so far, so good.

Still from Project Shrine

What was your process for testing this version control system?

Patrice Beauvais: For us on the tech side, we didn’t want to have half our project on Git and half on Perforce. We know how much of a burden it is to have to use and maintain two different source code management tools – it’s common with live service games where the online systems and game data aren’t necessarily fully integrated.

And a team of your size just couldn’t support that type of approach – the fact that Unity Version Control can facilitate both of those workflows is helpful?

Patrice Beauvais: Totally.

What kinds of problems have you encountered previously using two different version control systems?

Thomas Félix: In game development, you always need to build customizations for the tools you use, no matter how good they are. Unity Version Control is a great example – even though it works well, we still found ways we wanted to tailor it to our workflows. If you have to support two source control systems, you double that work, which is always painful. Someone who’s good with Git might not be as familiar with Perforce, and vice versa. Training people takes twice as long.

Patrice Beauvais: For me, the worst part is integrating data between two different version control systems. If you’re using Perforce but your data library is stored in Git, that data will need to go back into Perforce, so the two need a way to interface. Even though there are many solutions, these interactions aren’t really meant to happen, and sometimes you lose the project history. A bigger team can make it work, but I’m not going to spend six months building a solution to migrate data from Git to Perforce.

It sounds like your team has used many different version control systems over the years. What are the benefits and challenges of some of the solutions you’ve explored?

Thomas Félix: Let’s start with Perforce – it’s super resilient, it manages data very well, and it’s not that complex for nontechnical team members. You don’t find that anywhere else, really, except with Unity Version Control. On the other hand, the big monorepos you see in Perforce aren’t really suitable for game development – fast integration, multiple branches, that kind of thing. You can manage with Perforce, but it’s far from ideal, especially if you want to build a robust CI/CD pipeline.

Patrice Beauvais: Git’s UI is great for programmers, but I probably wouldn’t ask an artist to work with it. It’s not ideal for managing large files and data, and it doesn’t support locks natively very well, yet.

Thomas Félix: Unity Version Control is a better solution in many regards – the UI is tailored for content creators, so it’s great for usability. We see Unity Version Control as the perfect marriage of Git and Perforce.

Programmers usually want to be in Git, and you can get pretty much the same workflows in Unity Version Control. For nontechnical content creators, it’s easy to submit their data, which solves one of the biggest problems teams run into with source control.

Data loss is the worst thing that can happen to us. Code is easy to handle in every source control solution, but data is always tricky. We cannot afford to lose work, and each mistake made on the data side means paying for it a thousand times later on. We try to be very, very careful with that. With Unity Version Control, it’s a win for both our programmers and content creators.

Screen capture of Monster Closet work within Unity DevOps tool

Avoid broken builds with a task branch workflow for continuous deployment

Do you have any best practices you can recommend for maintaining build integrity?

Thomas Félix: For a small company like ours starting out, we knew we couldn’t afford to have broken builds because we submitted bad data or code to the main branch.

With Unity Version Control, we never work in the main branch. We’re always in control of it, it’s always stable, and the mergebot actually does most of that work for us. That really resonated with us when we were trying it out, and it’s one of the first things we put in place, even when we were just five people working on the main build. It’s worked really well: The main branch is almost never down, and it’s been like that for almost two years now.

How does Unity Version Control handle speed when working with large files and switching between branches and workspaces?

Thomas Félix: In terms of task branches and switching back and forth, that works well, too. It takes a bit of time for people to get used to this workflow – task branches are a new concept to many people, and it’s maybe not as fluent immediately for artists as it is for programmers.

That being said, every week – not every day – we do catch small problems through mergebots and our CI/CD processes, but they never enter the main branch or break the build. It takes a bit of time to get used to, but working in one branch will always be quicker than working across two – not by much, but if you step back and look at your pipeline as a whole, you start to realize it’s a much, much better way of working. At least for us, as a small-to-medium-sized company, it’s perfect.

So there’s a culture and learning change you have to make to move to continuous deployment, but it seems like you’re saying you’ve already caught a lot of bugs or other potential issues before they even hit main.

Thomas Félix: Totally. I would never go back to one-branch development. A team like us just can’t afford to spend days debugging or fixing problems that hit main. 

When we interviewed Apocalypse Studios, they discussed the “culture shift” that task branch workflows can require. They were using Perforce before Unity Version Control and talked a lot about branches versus streams. What’s your take on that?

Thomas Félix: Branches and streams are quite different to me. If Unity Version Control didn’t exist, we could probably build something around streams and try to get the same thing going, but it would be complex and error-prone. In Unity Version Control, it’s much easier and much safer, because branching is what it’s built for.

In Perforce, streams are the equivalent of tasks. If you’re super technical, you can make it work, but I would never put that in the hands of artists. With Unity Version Control, currently we have more than 1,000 branches – most of them are archived, and we have about 10–15 open at any given time. I’m not sure I’d like to have 1,000 project branches in Perforce.

What challenges do you anticipate as you move further along in development? What challenges have you faced already?

Patrice Beauvais: As we mentioned, people aren’t immediately used to the task branch way of working. For artists, it’s really new to them, so we’re careful to explain how it works and why we’re doing it.

Thomas Félix: That’s true. People weren’t resistant to it or anything, but it’s definitely a cultural switch. Anyone looking to switch to Unity Version Control, like we did, needs to take that into account. It’s a better way of working, but you have to be willing to think outside the box. We started fresh, from pretty much nothing – no office, no infrastructure, and a very small team – so it was a little easier for us than it might be for other studios. Building your infrastructure in the cloud sounds cool, but it comes with challenges in terms of iteration time, costs, setup, security…. In the end it’s a win, but it took us some time to get a reliable workflow up and running.

Still from Project Shrine

AAA-grade dashboards and observability

You’re also using another of Unity’s engine-agnostic solutions: our verified solutions partner, Backtrace. Can you tell us what your error tracking pipeline looks like?

Thomas Félix: We use Backtrace to track every single bug in most of our applications – the first ones, obviously, being the game and the Editor. We mentioned before that we built some tools around Unity Version Control – Backtrace is integrated there, too.

It didn’t take long to set it up, and it gave us access to some top-class tools, dashboards, and workflows. We were able to get a lot of the things we had in place at previous companies up and running pretty easily. After being operational for around six months, we already had visibility on all the crashes in the game, the Editor, and our tools. It wasn’t something I expected to get so early when starting a new studio, to be honest.

Patrice Beauvais: It’s a super good tool. At Ubisoft, I worked on a proprietary solution like Backtrace for two or three years. Backtrace is really feature-forward – it’s even faster than what I was working on, and was easy to implement. Again, we did add our own customizations for custom data, and worked to integrate it with our server, which is on Linux.

Thomas Félix: We were quite impressed by the time it took to set up Backtrace. Two or three days and we were already receiving crashes, so we decided to move forward.

View of Unity's Backtrace from Monster Closet

Advice for indie studios on setting up your tech stack

What did you do to ensure the process of implementing Unity Version Control went smoothly?

We’ve shipped a lot of big games, and we try to use that experience to think about how we can apply it in new contexts. That’s why we ended up going with Unity Version Control, and with Backtrace as well. The tricky part is making sure we don’t invent problems we don’t have – we’re not a 1,000-person studio anymore!

We’re always trying to find a balance between how we leverage our experience while reminding ourselves that we’re not trying to build the next big AAA game. We still want to make something great, and to do that, we need the best workflows – and Unity Version Control fits perfectly.

What was your process for testing this version control system?

Thomas Félix: The tricky part was making sure we could put it in the hands of artists, both in terms of the UX and data integrity. We worked with several artists on the team to make sure they understood how to use it. It was really important to us to nail data management for our project. The more people we added to the team, the better the feedback we got – people were happy, and we knew we were onto something.

How are you using Unity Version Control’s Gluon workflow for artists?

Thomas Félix: We do use Gluon, but for something else, what we call the raw data – data that’s not tied to the engine. Let’s say you’re an artist and you’re modeling a mesh: You’re using the raw data, the source file, in something like Blender. This doesn’t have to reach the engine; only the data you export from it does. This data is managed in a task branch, but we manage the source files in Gluon.

These files can be really heavy – character artists using tools like Zbrush can generate files that are 2, 3, 4 GB per asset, if not more. You don’t want programmers having to sync 1 TB worth of original character meshes, so we manage those in Gluon using partial workspaces. Character artists only synchronize character files, modelers will only synchronize model files, and it’s the same for audio, textures, and so on. It’s all stored in a separate repository, away from the task branch workflow.

Capture of Monster Closet team in-office while working with Unity

So, to recap, you’re using Gluon for scenarios where you’re working with huge files so someone doesn’t have to download the whole repo – they can just use a partial workspace.

Thomas Félix: Exactly. It’s an archived version of the original data, so we don’t use task branches for that. We don’t need to have a task branch for those materials, as long as creators submit their latest work every once in a while.

What advice do you have for smaller studios looking to scale up and tackle ambitious projects, like you’re doing?

Thomas Félix: That’s a good question. From day one, you need to know where you want to go. For us, we started with a small team, and we knew we wanted to grow, but we didn’t want to scale to 1,000 employees – even 200 isn’t our goal. We made many decisions – decisions that we’re really happy with! – that we might not have made if we had different ambitions.

Building your infra in the cloud does make it easy to scale – just be careful, because it can cost you an arm and a leg. Always try to be in control of your workflow. If something doesn’t work, do the work to understand why. Make sure you have strong foundations, basically.

Looking to optimize your game development pipeline? Get started with Unity DevOps, built to work with any engine.

March 9, 2023 in Games | 18 min. read

Is this article helpful for you?

Thank you for your feedback!

Related Posts