 All right, so thanks for joining us today. My name is Vinicius and I'm a Cloud Advocate at Microsoft focusing on containers and Kubernetes. And today I'm joined by Tyler from Return 10 Studios. So Tyler, thanks for joining us today. No problem, glad to be here. Awesome, so you wanna introduce yourself to people probably know you from the customer story that was published about the fours of five, but please introduce yourself in your own words. Sure, so I'm a Principal Software Engineering Lead over at Turn 10 Studios. I help build and maintain and run all the cloud services that power all of our ports are rising and our ports and motorsport games. Been at Turn 10 Studios for about eight years and Microsoft for about 19 years. And that's me. That's a long time. I've been at Microsoft for almost 11 and I can imagine how 19 years must feel like. It's flown by. Yes, I can see that. So all right, so let's talk about Forza, right? So Forza, the customer story was published on the public website. You guys are running on AKS on Windows containers. So let's talk a little bit about the history first. So what was the state of Forza prior to AKS and what made you guys choose AKS and continuing on the Windows platform? Sure, so when I joined the team, we were actually running on-prem services talking to an on-prem SQL database. And as our games continue to grow and become bigger and have larger audiences that just really, we weren't able to continue to maintain the services in that fashion. So a few years back, we migrated over to Azure Cloud Services that definitely had some benefits over the on-prem solution. We're able to have elastic scalability and scale up and down. Given right around our launch windows and holidays, we have humongous peaks of users and then that dies down a little bit in between those periods. And so previously we'd have all this hardware for the launch and then we'd still have to pay for it and maintain it, but it wouldn't really be used in between launch and holiday. And so moving to Cloud Services helped us out a lot. And but we really wanted to be on a modern future-looking platform and Cloud Services is just not that anymore. And so early last year, we started looking around to see what's kind of out there. We were looking at Azure Service Fabric for a little bit and then Azure Kubernetes. And just given our timeframe of where we're at with different titles we were working on and different deadlines, we really landed on Azure Kubernetes as the solution for us. It seemed kind of natural, just the large community base around it. At the time, we weren't even positive whether that meant we had to go to Linux because that is the larger community. But as we investigated it, we found that we really wouldn't have to change that much of our logic if we went to Windows containers on AKS. And so being able to stay on Windows and stay on .NET framework, running in IIS, everything that we had been using for the forever, it really allowed our transition to happen very fast and able to get it all up and running and stress it and feel comfortable about it before our Forza Horizon 5 launch last year. Yeah, so the team had a lot of knowledge on Windows already and running on the Cloud Services. So you wanted to continue that knowledge and then if you had to learn something new, it was the Kubernetes portion of that, right? Yeah, exactly. Our team was very new to Kubernetes and Docker. So we had a lot of ramp up there. If we had to move to .NET Core or move to Linux or pick up something else, it probably wouldn't have fit in that timeframe. But the fact that we were able to reuse a lot of what we already knew and just learn those portions around Docker and Kubernetes, it allowed us to really move fast. Yeah, so I was going to actually ask what were the challenges in moving to the AKS platform? I'm assuming that learning curve of Kubernetes was already something that the team saw as a challenge, right? Yeah, I mean, just going from Cloud Services where it kind of packaged it all up for us and it was just one single package. Whereas now as Kubernetes, it's extremely more powerful but with power comes a little more responsibility and learning those differences between services and deployments and pods and nodes and config maps. Yeah, it's a little ramp up, the new area, new technology. The fact that AKS is kind of built as a platform though, we were able to build some great dev tools on top of it that interacted with it that just kind of helped the team out to onboard to it. And then, since we're already in Azure DevOps for our deployments, our deployments are still just one click in ADO. We just had to rework how that release pipeline worked a little bit. And so we were able to reuse a lot of the technology we already knew and just kind of switch out that platform. Great, so you touched on something that is actually true for a lot of our customers, right? Which is the seasonality of how many users you have using your services. So it's holiday, if it's a specific holiday during the year or if it's the end of the year, they will see an increase of number of users and then it will die down after that holiday or that specific date. That is true for you as well. And from reading the customer case on the Microsoft website, we noticed that you had like a million users using the service at the same time for the launch, right? So tell us a little bit more about your architecture and how you guys achieved that one million users concurrently. Yeah, so Fort Horizon 5 was the first time we really, we knew it was a great game. We knew it was gonna be a big launch. And so we set our stress goals ahead of launch to be considerably higher than we'd ever done in the past. And so we actually tested to 3 million concurrent users. And even just the first part, the hard part of there was to be able to simulate those 3 million concurrent users. We actually doubled down on AKS there and wrote our test harness in AKS since we were able to deploy and scale up our clients, which in turn would cause our services to scale up. And that architecture really is spread out to 17 different microservices. And so some of the common ones that are used in our game, the most popular is leaderboards and UGC, so liveries and tombs, those kinds of things, our drive at Tars or auction house. And so all of those are kind of spread out using all different kinds of Azure storage tech. So whether it be table and blob store or SQL managed instance or Cosmos DB. And so having all of those spread out has helped us scale tremendously. On top of that, one of the key things that got us over that hump in stress to get to that 3 million was using the Redis cache in Azure. That allowed us to really kind of bypass some of the lower latency calls were needed and just pull from our local cache. And so yeah, all that in combination was, we went into this knowing it was gonna be our biggest launch ever, but also probably feeling the most comfortable about any of our launches. And as you said, we broke a million concurrent for the first time in our franchise, which was very exciting. And at that time, all 17 microservices were auto scaling with no intervention and work great. That's awesome. Yeah, the auto scaling feature on AKS is something that customers take advantage a lot and just the peace of mind that knowing that you're gonna scale up and down as the load comes and goes. It's really a peace of mind for customers. But like one of the things that we're really interested to hear more about is, it's a Windows platform, right? So you continue on Windows running on AKS. And the question that customers have a lot when using that approach of taking an application that already exists moving to AKS is how much of the architecture needs to be rearchitected, I'm sorry, how much of the application needs to be rearchitected to work properly in Kubernetes and in AKS. So how was that experience for you guys? Yeah, I mean, we were definitely a little concerned given that we have these 17 microservices and the timeframe we had, if we had to rearchitect, any of the services considerably, we would not have hit our deadlines. And in actuality, once we kind of, we migrated two of them to start and wrote this playbook and then we're able to just apply that playbook across the board. And that playbook really just consisted of configuration changes, some certificate changes. And then like how we initialize the service of how we read in that config, how we install certs. And that was really the only change. I mean, in the end, we are still a full .NET Framework service running in IAS on these Windows machines. And so not much else changed down the line, which is what allowed us to move so fast. I see. So it was more like making sure that you were adopting all the microservices and cloud native approach of running an application rather than changing the code of the application itself, essentially. Yeah, I mean, there's no business logic. I mean, even how it was hosted in IAS didn't change. It's really just how it started up and how we deployed it. That's awesome. If you had like one recommendation for customers, like what was the main thing that you and the team learned during the process of this Migrate into IAS? What would be that you would recommend people looking to be aware of what was the major learning for you during this process? Hmm. So we ran into a couple of issues along the way. So one of them was previously our build agents in ADO. We used Microsoft hosted once. We didn't have to worry about it. And it would just do our new get restores and everything would work just fine. Being in the Docker world, every build is on a brand new container. And we ended up having to move to our own hosted agents in that scenario. So that way we could actually use the Docker cache. Without that Docker cache, we were downloading 10 gigabytes of the Windows image each time and our builds would take too long. And so by having our own hosted agents, we're able to reuse the Docker cache and not redownload all of those images, not do new get restores. And there's some great best practices on creating those Docker files to make sure you're able to reuse as many of those layers of the image as possible. So that was key. Using our keep alive, sorry, not keep alive, our health checks and our startup probes. Just so if a pod goes bad or fails to start up, we don't actually put it into rotation. So effectively utilizing those and making sure the service is healthy before it gets put into rotation. Yeah, keeping the image smaller is one of the issues that we talk to customers a lot. It's interesting to see how you approach that. Docker caches for sure something that customers leverage as well. Yep, and now that we are on AKS and we're on a supported platform and everything's working great, it kind of opens up some more possibilities for us in the future. We could move to .NET 6 and .NET Core and move to Windows Nano servers or even move to Linux to get smaller images and pick up some performance gains. So it's not something that we feel like we actually need to do because we're pretty happy with where we're at right now. But it kind of just unlocks that possibility for the future. Yeah, the fact that you have that possibility open, it's already positive, yes. Yep. That's awesome. So anything final for our viewers, anything that they should know about you or the team or fours of five, maybe your avatar so they can play with you. Tyler Hen is my gamer tag. Feel free to friend me and I try and play a lot of Forza Horizon 5 whenever I can. It's one other thing, actually it was a huge improvement in our dev environment. In the old world of being on cloud services, every single deployment had two VMs that had to spin up for it just for redundancy. And so across 17 microservices, there's 34 VMs already. Whereas now in containers and Docker, we're able to actually stuff all those 34 pods onto just like three or four VMs. So it was actually a great cost reduction in our test environment. And then in our production environment, we did actually play it safe and go with one pod per node to start with, just to keep it very similar to how our old architecture was as we were still learning the environment. And now we're actually able to pack some more pods onto nodes and get even cost reduction in our production environment, which has been great. That's awesome. At this plane, what is the configuration you're gonna use and the sizing of the VMs that are going to run your pods and how many pods you're gonna use is something that customers are also interested in learning more about. And usually the recommendation, there's some guidance of course, but usually the recommendation is played safe, make sure you have enough for supporting at least the load that you think it's gonna have, but then keep an eye on all this care as well. And yeah, I think that approach makes a lot of sense and it's good to hear how you play with that. All right, so thanks a lot for your time. I think that's it for today. Thanks for joining us again. Thanks Tyler for the time and we'll see you on the next video. Thanks a lot.