 Welcome to Ops 105 virtualized and hybrid backup deep dive. Let's get started. Today's presenter is Ben Armstrong, and he's a principal program manager lead for AKS HCI. He's also been insanely responsible for most of Microsoft's virtualization efforts over the last decade and maybe a half, almost two. Anyway, Ben, take it away. And I do not like being told I'm responsible for things. Good to be here today. Super excited to be able to chat about some some architecture, some details. So hi everyone, Ben Armstrong here. As Aaron mentioned, I currently working on AKS HCI, so I am hoping that towards the end of this conversation, we'll be able to chat about containers and where things are going in this space. But yeah, I have been working on virtualization for forever. I was telling Aaron at the beginning of this, almost 20 years, 20 years and nine months, two days, nine months, two days I will have been working on virtualization for 20 years, not that I'm counting. I think people forget that virtualization has been around that long. Like this is not a new thing. No, the really like the really fun thing is like when I started working on virtualization, working at a startup company in Silicon Valley, like we had a couple of guys on the team who were like old IBM and Amdahl guys who were like, I've been doing this for 40 years. They're all retired now. But yeah, it's still lots of fun. It's still exciting. And today we're going to be talking about backup, but we're also like largely talking about like data and how to manage data in kind of this virtualized world. So with that, shall I jump in? Awesome. So talking about like 20 years working on this and the change and so on. And one of the things that I like to check people about is I like to say that like, for all the work that was done in virtualization at the end of the day, there are only two things that matter to our customers and I'm going to start throwing stuff on my whiteboard now. And there are two things that matter to people. The first one is their apps. I've got this nice little bowl here, which for this talk is going to be your app. You have a server app and you need to run it somewhere. And the second thing that you have that you care about is the data for that app. So I have an app, it does some stuff. Most of the time there is data associated with it. And you know, when I look at everything that's that's gone on in the world of virtualization and now with containers. All we're really about is trying to make it easier for you to deploy your apps and manage your data. You know, and if you go back, you know, 25 years ago, you know, and I do I this is a conversation I have a lot because often when I'm talking to people about like, you know, moving to containers containerizing their apps. And we're talking through all the details. People are like, Oh, this this seems complicated, like there's a lot of work here. And I like to point out that like, yeah, 25 years ago, if you want to deploy a new server application, the first thing that you had to do was write out a business justification to explain why you're about to go spend money buying a new physical server. Don't tell me this is complicated. Like, it used to be a lot worse. But you know, for a lot of folk and for a lot of apps, that's the world where we started. You know, you have your app, you have a physical server, and you have physical disks attached to it. And you know, even way back then, you know, we were very aware about the fact that like, okay. You know, like, you can play an app, you can run an app. That's the easy part. The hard part is the data. You know, the hard part is the like, how do I make sure that my daughter is where it needs to be for apps. How do I make sure that if something goes wrong, I can get my data back. And you know, so it would go back, you know, 15, 20 years ago. The answer to everyone's problems was like, go out and get a really nice expensive sand. They're magic. And you know, the interesting thing is, you know, when we, when we think about where we are today with like hybrid cloud and so on. In a lot of ways, I actually look at the sand industry and go like, they were ahead of their times in one aspect and that when I went out and called customers about like, hey, you're spending a lot of money on your sands like why, why are you doing this. On one hand customers would come back and be like, well, it has these features in this week and this week and this week. But on the other hand, you chat to people and they'd be like, well, why do I pay for the same because it's a managed service. Okay, they didn't manage service, but that's what it was. Like you paid someone money for your sand and then like you didn't have to manage it anymore because if there was a further failure, someone would arrive in a car and fix it for you. I remember hearing the most expensive thing in a lot of data centers at one point in time with storage and it's sort of like the moment you were the most expensive thing in a data center, you put a big target on your back. I remember like around around 2010 2012 when we were starting to work on storage spaces direction starting to work on scale out policy and all of that I used to chat to people and I used to say like whatever size deployment you're at, whether you're like a fortune 100 company or whether you're a mom and pop running servers under under your stairs or in the back room. There are two facts that I can tell you about school. The first factor has there is a perfect solution out there for you. And the second factor is it is more than you want to pay for. That is so true and then we start talking about backing up all that data my husband used to manage the robotic tape silo because all that data had to go on to take and then all the tapes had to go somewhere else off site and that was how we used to do backups. Yeah, no absolutely my one of not my first it job but one of my early it jobs I was actually server admin for a division of the University of Queensland local people. And one of my jobs was maintaining the tape backups. And God bless we had one secretary who at least months a week with the lead up business critical file. So we got to we got to test our tape backup system quite thoroughly because at least once a week we'd get a call and be like chaos. But so that's that's where a lot of us started. But then virtual machines came along. And let's kind of step through the journey of that so virtual machines just take this diagram and just make it more comfortable. You know, one of the things one of the jokes I often like to make is I work with the team of engineers where we believe that any problem can be solved by just adding a couple of extra operating systems. I'll make things better. So with virtual machines let's let's spread out this diagram, because we now have, we have our app, which is running on a virtual machine which is running on physical computer. But this virtual machine now has a virtual hard drive that's running on the physical hard drive. So I've now got all these extra layers in here. And this raises like a real question about excellent center. This raises a real question about, okay, what does what does backup. What does it mean in the in this world. And when we first started working on virtualization when I was a part of the team that built virtual server, the precursor to hyper day. Our answer was quite simple. We just looked at that and we're like, that's that's complicated and that is hard customers if you want to back up. So far inside your virtual machine and ignore that this world exists. Which as engineers who are like problem so peace out. But we what we didn't contend for was one that our customers wouldn't be happy with that answer. Two, we saw a fascinating thing happening with our customers as they will bring their applications from physical machines over to virtual machine. And what we discovered was that anyone who's working in the server industry, like tried and true best practice that we talk about is one application per server. One function per server. Like you want a file server. Single server you want a single server you want a web server single server like that is that is textbook that is being textbook. Since the beginning of time. But fun thing about industry best practices and textbook stuff is that often we don't follow them. And we just don't tell anyone that we're not following. Like you go to you go to like user groups and you go to conferences and people are like, well of course everyone does it this way and everyone says there goes like, yeah, and then they go home and they don't do that. So it turns out that like pre virtualization. We had a lot of people who are saying. Oh yeah one one function per server. But then in reality we're gonna I got a big server I'm gonna put a bunch of apps on it. If I had a dollar every time I saw SQL server installed on the same server that exchange was installed on. I'd have a lot of dollars. Yeah. So, but what happened was when we started bringing out virtualization and people could it enabled people to simultaneously follow this best practice. I have one app per server while still using their hardware efficiently. And so what we actually saw was that as people migrated physical service to virtual machines, there was like this explosion of the number of systems. You know, and we would see customers where like that 20 physical servers, and they would virtualize them into 120 virtual service. On one hand, like that was good. They were now they were following best practices. They had like more isolation between these workloads like failure scenarios were better like patch managed like a whole bunch of things became better. But on the other hand, you had this problem that where like if you thought about the traditional backup approach, you went from I have 20 servers I need to back up to I have a hundred and 20. And so customers came to us very strongly and they were like, we need a way to to back up these systems in a more efficient manner. And that actually that started the beginning of a fairly complicated journey as we tried to figure out how to do this. And so the first thing I'll take a pause at this moment to just kind of bring people up to speed on a piece of esoteric technology that you may be well versed in, but you may not be well versed in. And so jumping over to my diagram. The technology is a lovely thing that people in the windows will know, which is VSS. VSS is my favorite three letter acronym for a four word title. I'm going to copy services. Yeah, give the man a try. Volume shadow copy services. VSS the copy. Just got dropped along the way. So VSS is infrastructure that Microsoft built back in the day of sands, where we had all these different manufacturers producing different ways to handle backup efficiently. We had all these different backup vendors who want to interact with them. And we wanted to create a platform that would enable this ecosystem. And so we came up with VSS. And VSS under the covers can be broken down into three different categories. So let's get rid of this. You have the VSS or cluster, the VSS writer and the VSS provider. So what, what are these things and how do they fit in into this picture. So the VSS provider is the thing that sits on top of your storage and says, I know how to take a snapshot of my state. So I have a VSS provider down here that windows can call into and say take a snapshot of my state. The writer is the thing that is attached to the app that you're running. And it is allows your app to get ready to have a backup. So if you're running SQL, it has a VSS writer. If you're running exchange, it has a VSS writer. All these and different parts of windows have VSS writers where you can call into them and say, Hey, we're about snapshot storage. Get ready. And then we have the VSS request or more commonly known as your backup software of choice. So your backup software of your choice, whether it's the inbuilt backup, whether it's DPM or whether it's a third party offering as the VSS request. And the VSS request or when it decides that it's time to take a backup. It goes to your app, goes to all your apps and says, Hey, get ready for backup. And when all the apps come back and say, we're ready, goes to the storage and says, Okay, take a snapshot. We're good. So how did we start thinking about this with virtual machines? Well, the first approach that we took was we said, Okay, what we're going to do is, you know, because we have this problem that you have your VSS request or your backup software running here. And it can't see the VSS writers inside your virtual machine just sees these virtual machines. And it's like those things are weird, but I don't see any apps here. I'll do a backup call into the VSS provider, take a snapshot and all the apps inside the virtual machines were not ready. It's a train. And so what we did was we're like, Okay, what we will do is we'll do two things. The first thing we'll do is we are going to make a Hyper-V VSS writing and it's going to sit here. So that it can tell the backup software, hey, VMs are apps too. Like if you're backing up the system, you need to talk to us so that we can make sure that your VMs are right. Then the second thing that we did was we actually created a little itty bitty VSS request or basically a stub backup program, which we call our backup integration component that runs inside the virtual machine. So we have that. So now what happens and what I'm describing here is actually the technology kind of as it was in Windows Server 2008, 2008 R2. So now what happens is your VSS request or your backup software comes to Windows and it's like, Hey, I want to do backups like what apps are here that I need to back up. And Windows come back and lists its apps and among it says, Oh, there's a bunch of VMs here that you need to back up. And that's this writer and goes great. So now when it goes to backup, it calls in and it says, Hey, back up your VMs hyper V and turn like shunts this message over to the backup. I see which inside the virtual machine repeats this person goes through calls into all the VSS writers. So like, get ready. And they'll get ready. We return back and we say, we're ready. And then VSS calls in and says, Great, take a snapshot of the heart. So that all sounds good, right? Problem solved. Well, not quite, because things get a little bit weird here. Because the way VSS is designed with these three components. It's actually designed with a very tight like window for taking the snapshot. I'm specifically when, when the VSS requester calls into the writers and say, Get ready for a snapshot. The writers come back and say, Okay, I'm ready. And I am waiting for 10 seconds. You got 10 seconds to take your snapshot. And then I'm often running. And so what would actually happen in this scenario is that often when we took the snapshot on the physical hardware, the virtual machine had already resumed running. We missed the moment. The like, we had a clean backup moment, but it was actually gone. So what we actually did to handle that is, and I'm going to jump over to another page here. So let's imagine I have this machine and had a bunch of virtual machines on it, which had a bunch of hard disks. And I, they're all running on my storage. And I took a snapshot. But unfortunately, the snapshot happened 30 seconds after they were ready for a backup. So what we actually did was we implemented some code in our, in our VSS writer in the hyperb VSS writer, where we said, Okay, when the snapshot is done, you need to go into that snapshot. And you need to go through every virtual hard drive that's in that snapshot. And one by one, you need to mount it connected to the host and roll it back to that clean moment. And then once it's clean, return it to the back to the back. So we would go through and we would do that processing. It was a post snapshot processing event where we would go in and we would clean up all those virtual hard drives. And on one hand, that kind of worked. But on the other hand, as we're getting towards the end of 2008 or two and getting towards 2012. So what we were seeing was that computers were getting bigger and bigger and systems were getting bigger and bigger. And people were running more and more virtual machines. And as they were running more and more virtual machines, the number of virtual hard drives that we were having to clean up on a backup was getting larger and larger and larger and larger. It was a problem because we were now seeing systems where as part of the post operation on backup, we were having to mount and clean up hundreds of virtual hard drives. And we were having all sorts of issues come up. And I mean, just to highlight, Aaron Sonja. If you took a server and hot plugged in 200 hard drives, and then hot removed 200 hard drives. No problems, right? That's going to work perfectly every time. No. Yeah, all the issues we found. So many fun and unusual parts. So as we were coming into 2012, we were like, okay, we need to do something to get rid of this post backup cleanup phase is causing us problems. And we came up with this great idea where we were like, we know what we can do. We're going to change the Hyper-V virtual hard drive to stop being a hard drive and start being a sand. And what we actually did is a really fun thing. If you actually go digging through the device IDs in the change actually happened in 2012 R2 in 2012 R2 virtual hard drive inside virtual machines, stopped presenting themselves as SATA devices, and they started presenting themselves as SAS devices. And what we actually did was we said, okay, the virtual controller for the virtual hard drive for the virtual machine is now better. And it knows how to do snapshots. And we actually implemented a Hyper-V VSS provider. Side note, this made the VSS team super excited because when we did this we became the first software team to ever implement all three components of the VSS stack. And we also gave them a perfect test framework where they could now very easily try out every bed and so on. So now in 2012 R2, now what happens is VSS request, go back up software. I want to back up. Windows comes back and says I got a bunch of virtual machines. It says, okay, back up virtual machine. That message comes over here. Our mini fake backup software says, okay, all your apps back up. And when the apps are doing their like 10 second holding breath, this guy actually would pull down into the Hyper-V VSS provider and say, okay, take a snapshot of just the virtual hard drives with this virtual machine. And once it had done that, it would come back to here and say, okay, here are the files that you need. This was a massive change in 2012 R2. It took something that was a very intense and very, you know, like lockstep sync process and made it hugely scalable and it greatly improved the reliability of backup. So that was a huge one. So in that case then, was that all seamless to the backup vendors? Because you put it down. It sounds like they wouldn't have had to have made any changes because they're just still requesting the same thing from the Hyper-VSS right now. Absolutely. From the backup vendor point of view, all they saw was that things just became a lot more reliable. What you want to do as a platform provider. That kind of got us to 2012 R2. But around that time, was also when, you know, a go back in your wayback machine flashback, was also around the time that people were starting to get serious about cloud computing. And when hybrid cloud was starting to be a conversation, it was also like really early days of the big data conversation. And one of the things that was starting to come up in these conversations was, oh my gosh, our backups are huge. Like, have you seen how much data our backups take? Have you seen like how long it takes to make a backup because like there's so much data to transfer? I remember a time when a gigabyte was an unconscionably large amount of data. And when you had systems where the backups made a noticeable performance impact on the system when the backups were running, regardless of whether or not that system was initiating the backup or was just subject to being backed up, you literally started running out of that time window where there's only 24 hours in the day and the business were about to walk back through the door again. Yeah, absolutely. So like coming out of 2012 or two, as we were like thinking about 2016, that was kind of top of mind for us. What we were really seeing at that stage was we had a bunch of our partners and backup vendors who were trying to solve this problem themselves. And some were successful, some were struggling, people were having different approaches. And so in 2016, we introduced a new technology for Hyper-V, which is RCT. And RCT stands for Resilient Change Tracking. Now, I am not going to do justice to RCT today. I will say of all of this diagram, RCT is actually one thing, one of the few bits where there is an amazing TechEd session on RCT. It was TechEd Europe, we had one of the guys who was working on this Taylor Brown do a session. If you want to dig in deep on RCT, you can go look that up. And if you look up Hyper-V, backup, brand bag, there's a couple of them out there. Check out those links and add them to the video. But RCT, to give the simple version of RCT, RCT provided an API for backup vendors. So we still have this VSS flow, we still have all this going. But with RCT, we provided backup vendors the ability that when they would call in and they would say, hey, Hyper-V, take a backup of this virtual machine. And we would go off and do our stuff and we would come back and we would say, OK, we've done a backup of this virtual machine. Here's the backup. RCT allowed vendors to come in and say, OK, you gave me that backup, which is lovely. But can you tell me what are the specific bits of the hard drive that have changed since the last time I took a backup? And that's RCT stands for resilient change track. It is a log that runs where we don't track the data that has changed. We just track which sectors of the hard drive have changed since the last time someone took a backup. And so backup vendors can be like, Hyper-V, like I asked for a backup and you just gave me a 30 gig VHD. Which bits of this have changed? And with RCT, we can come back and be like, oh, 200 meg, this 200 meg. This 200 meg is the different 200 meg. And that's then allowed people to build efficient backup solutions where they can go like, OK, like we still have our last backup sitting over there in our backup service. So rather than copying the 30 gig, let's go in, get the data that's changed. And ship that data so that we can be a lot more efficient. So that kind of gets us to like 2016 for backup. And we're now getting close to the modern day world. And so we're now going to switch over to a bit of a more loud oriented conversation. So one of the fun things with all this conversation that I'm always talking to people about, and it shocks me that this message doesn't seem to land. In fact, let me, I'm going to turn tables. Hey, Aaron, Sonya, what, what, what, like people say that public cloud is just other people's computers, right? Like when I run something in public cloud, I'm running it on a bunch of other people's computers. Yeah. So and Azure is our public cloud. Yes. So, so PubQuiz. What is the operating system running on the computers in our public cloud? A skewer of Windows. Windows server. Windows server, baby. What, when you start a virtual machine, what's the software that that virtual machine's running it? Hi, baby. Hi, baby. Yay. And not only is it Windows server and a Hyper-V, it is literally the same Hyper-V. Like the number of times I have to tell people that it's like, no, we don't have like a special Hyper-V that runs in Azure. Like you get, like if you go out and get like the Windows server insiders build and get like the ways, like those are the bits that we run in Azure. So, you know, it's really funny when I talk to people and they're like, so how does this work in Azure? And it's like, well, we have virtual machines and we attach virtual hard drives. And like, as we come over to Azure, a lot of this technology looks the same. You know, Azure uses virtual machines, it uses virtual hard drives. We do have, hopefully it comes as a surprise to no one, we don't have a bunch of sands in Azure. We do have our own cloud storage solutions that we run in Azure. Now, and one of the things, and once again, I'm not going to dig into cloud storage in Azure. They're actually, if you're curious about how we do storage, Mark Pracenovic has done an awesome set of presentations over the years at different events, talking about the architecture of Azure. But, and he talks about storage there. And so I would really encourage people to look into that. One of the things that's interesting about Azure storage, I'm going to clean up this diagram a bit is I often get people asking about like, hey, does Azure use VHD or VHDX files or like, what's the file format that's used inside Azure? The interesting thing to understand here is that if you think about a sand, like most people have a thorough understanding of a sand. A sand is actually a fancy server that exposes objects to the network that look like physical disks. Like that's their, their object of virtualization. So Azure storage exposes objects to the network that look like virtual disks. And so to the question of like, does Azure use VHD or VHDX, the answer is kind of neither in both. It has a class of virtualization that exposes virtual disks. But a lot of the concepts are the same. And we've been working in Azure to expose a lot of the same capabilities. So you can now go into Azure, you can have a virtual machine, you can ask for it to be backed up. We efficiently take the snapshot and we store the snapshot. We've also been using our investments and stuff like RCT to enable efficient backup of your virtual machines running in your data center up to Azure. Because at the end of the day, it's all about that efficient data transfer. But at this stage, I do want to switch over to talking about containers. But before I do, I'll pause and see if Arna or Sonya have anything they want to poke me about before I move on to containers. No, not really. I think that you're, you're covering it pretty well in terms of at least understanding. And I'm assuming that, as you said, that the backup infrastructure, the way that it's backed up in Azure, is basically sort of like a scaled up version of how it's being backed up on a server 2019 instance, for example. Yeah, absolutely. The fun thing of like, everything gets a bit strange at massive scale. You know, so the interesting thing about when you think about backup in Azure, as opposed to the way that most people think about backup, is that there are a lot of places in Azure where the way we handle availability and the way we handle backup is by just keeping a million copies of things laying around. Because you can keep a million copies of things laying around. I will not name names because I don't like to publicly shame and also it's a company secret. I'm not allowed to talk about that. But there was a case a number of years ago that I was involved in where a team building an Azure service accidentally checked in code that was just randomly deleting their own data. But Azure Storage was doing such a good job of keeping redundant copies that it actually kept ahead of their own code deleting data. And so the only thing that the service team noticed was they were like, we're not seeing the performance we think we should see. Like we're seeing much lower IO than we would expect. And when they dug in, it's like, oh my gosh, like our IO is low because we write something and then like 30 seconds later we delete it and then Azure covers for us and restores it. And we didn't notice. And that's why we don't have silver like as a service. Not I'm not naming names. So they're like there are different things at scale, but from a fundamental concepts point of view is very simple. So now, let me jump over to a new page. So let's let's talk about containers. And I didn't get a handy diagram ready for us. I'm just enjoying the trip down memory lane and it's really interesting to see that once we stepped into the world of virtualization and you started to uncover the problems at scale as this got bigger. It's not it doesn't seem like it was a huge jump from that once that problem was solved to a cloud based and I think everybody assumes that we've invented the cloud from scratch where it seems like from a backup and storage perspective. It's more of an evolution of what we've already had in place for virtualization. Oh yeah 100% you know 100% it's you know, it's like a lot of people ask me about like then have like you've been doing the same job for 20 years like how is that even possible. And, you know, the, the, there are a couple of answers that I give to that. The, the first one is because like, you know, I've been doing the same job 20 years, like the world is changing the industry is changing, like everything like is changing around me and you know, the real point in time could I have predicted what problems we would be solving in two years time. And often, you know what we're solving today if you went back five years ago and asked me if we could solve that I would have said no that's an awesome. You know, one of my favorite examples of this in the space of storage is. If you, if you had talked to me and like while we're working on Hyper B and Windows over 2008 or two. And you said hey Ben like, we got this customer escalation where customers like got a deployment where they have some flaky storage and the storage falls over occasionally. And the storage falls over the entire system goes down. Like what can we do to help them. I would have deadpan looked at you and said get better storage. But then as we started scaling up and scaling up and scaling up. What we realized is this is like that happens and that's unavoidable. And we need to think of ways to survive that. And so in Windows, Windows Server 2016. We put a bunch of work in there so that and I demoed this on stage and I still think it's kind of black magic. And where we made it so that virtual machines can survive massive storage finger. Which is really cool like we did like we did all this work we're like okay, like if the storage goes away like we will detect it and we'll pause the virtual machine and we'll hold the storage IO, and we'll put it in like suspended critical so that you know an admin. And when the storage returns the virtual machine resumes running. But it's like, yeah, it's, it's been a constant learning, but it's also yeah building on top of, you know, all the stuff that we built previously. So, moving over to container, my favorite topic right now. One of the greatest misnomers. And I have talked to many people about this as they go containers, then why are you talking about containers when we're talking about hybrid backup and data, because containers are stateless right. Any, any app worth running has state. But what is has really happened with containers, which is lovely is that containers have their state somewhere else. And so, for most people what this looks like is, I have a container, it runs my app. And the data for that is over here in the SQL database. And so, yeah, my container is stateless I can restart it as often as I want. The database still exists. And it needs to be to be backed up and managed. Now, that's a very simple diagram, but the real world is a lot more complicated. So let's start talking about how we view this with AKS HCI. So with AKS HCI, you know, first we're going to start with our foundation of we have right now. So we have a handful of physical computers that we're going to install Azure Stack HCI on. Got the latest greatest, you know, Microsoft on-prem cloud solution. Each of these physical computers got to have a bunch of local disks. And Azure Stack HCI is going to use storage spaces direct to put them into one big pool of virtual storage that I can use. So we have Azure Stack HCI. We have our storage spaces direct storage running there and like this guy. So you're going to come along and you're going to say, I want to install AKS HCI on this. And for AKS HCI, as you create Kubernetes clusters, we're going to make sets of virtual machines that we're going to use to run containers. Now, these will either be Windows virtual machines or Linux virtual machines, depending on what you tell us. And above this, and I still forgot to put my container. Let me quickly fix this. There we go. And you're going to run your containers. So as we already alluded to, the containers, yes, technically the container itself or stateless is just that your state is somewhere else. So what does somewhere else look like? Well, there are three models that we see customers doing. The first model, certainly the easiest one is that maybe your data is running off on a completely different system. You have a different system that is your SQL database. That's nice and simple. You have your containers running over here. You have your SQL database over here that certainly makes like easy from a backup and data management point of view because you just back up and manage this. What it does not make simple at all is the amount of network traffic that you're going to have coming out of this AKS HCI cluster and going to that SQL database. And that is actually what pushes a lot of people away from this model. It's just the like, oh my gosh, every piece of data that any of these apps made is now going over a network connection to this. So what can we do about that? Well, the first thing that we see people doing, I'm going to put these closer together, these four virtual machines are my AKS HCI deployment. One of the great things about Azure Stack HCI is not only can it run AKS HCI, it can run VMs. So they go fine. I'm going to put, I'm going to create, I'm going to put a pair of virtual machines on here. I'm going to create a virtualized SQL cluster and I'm going to have my containers access that. This is perfectly viable solution. And this is actually what we see a lot of our customers doing. And now, you know, I use the backup model I talked about to back up these virtual machines and my containers access the data. But my hope is that this isn't where most people stop. Because they're really, there are two other ways to handle this. The first one, the next one is my favorite way is to go, you know what? I am going to play AKS HCI on my system. I'm going to have a bunch of apps that I'm going to run. But I am also on top of AKS HCI, I am going to deploy Arc-enabled data services. Because what does Arc-enabled data services give me in this picture? It gives you a containerized database. You can use Arc-enabled data services to spin up SQL managed instances on top of this cluster. So now you've got SQL running here. So on one hand, this is really cool because now I can have, you know, application specific SQL databases. I can like tweak and tune them. It's all managed in my environment. But this might have you going, but wait a second. Where's the data? Where's the data? That is a good place. So let me answer the where's the data? Like when I do Arc-enabled data services, hint, it's not up in Azure because you don't have that thing. So what's actually happening is we have four each of these virtual machines. They each have a system drive that AKS HCI manages for you. We keep it patched, we keep it up to date. You never see that. But they each have a system drive that's a virtual hug. But what we also do is when we're running Arc-enabled data services and they're wanting to spin up these SQL containers. As part, you know, when you go through Arc-enabled data services and you say, I want a SQL managed instance. What actually happens is Arc-enabled data services comes into AKS HCI and says, excuse me. I need some persistent storage. I'm going to do something in a container where I need persistent storage. And being Microsoft, we go, you know what's good at persistent storage? VHDX. Like we got you. So we provision a VHDX file for each instance, for each SQL managed instance. And then make my numbers match a little bit OCD about the accuracy of my diagrams. Now I have each of these instances of SQL running in a container has its virtual hug drive where its daughter is going to get persistent. Now this virtual hug drive is being stored on the storage spaces directory. So like it's got redundancy protection, it's got high performance, it's awesome. And then what happens is, as we start the individual SQL instance, first, we hot plug the VHD to whichever VM where that container is run. Now handling Hyper-V support, you'll help plug in remote. So we help plug the VHD. And then through AKS HCI, we call in and we wire this all up. We tell this container, oh, by the way, your storage is over there. And by doing this, we get to leverage all of the infrastructure over here that I talked about. And so when you need to take a backup of the system, well, it's just backing up virtual machines and attach virtual hug drives. So you can now come in here and say, like, I need to back up the system and we have all the technology. And bringing it back to where you started the conversation, of course, with the problems of detaching and reattaching virtual hard drives, we've solved the problem with attaching and reattaching virtual hard drives at scale in this end of the technology where we had a challenge with it back in the 2008 days. Yeah, it's actually, it's really one of the, and Sonia, you touched on this as kind of the journey to public cloud and then how that has benefited our hybrid cloud and our on-prem users. It's one of the things that I find particularly fascinating is that, and the work with containers is a great example. Often what has happened is we will have built a solution initially, you know, in 2012, you know, for small private clouds. And then we'll bring that across to Azure. And there's a huge amount of engineering work to scale up, you know, and get the reliability and so on so that we can do things that Azure scale. And then we find new scenarios on-prem like this where we're like, oh, we like, we have a problem solved. And handily, we now have this great tool in the toolbox where we know we can solve that problem with it. And so technology like goes between those two environments all the time. And it's one of the things that I think makes Microsoft's hybrid story so darn impressive is we own the operating system and we own the Azure cloud. So we have this advantage of being able to, you know, look at advantages between those two technologies and merge them seamlessly so that both of our cloud customers and our on-prem customers and our hybrid customers will benefit. I know that, you know, talking to a lot of customers who maybe thought that the cloud story was, you know, I have to lift and shift or I have to refactor all my apps and I have to go fully cloud. And to see that that's not the case and to remind people that there are so many things that we are doing and innovating that we then bring back into hybrid and on-prem scenarios back into Windows Server with all the updates that it has had as well. I think a few people are starting to turn the lights on now about being able to dip their toe in the cloud waters and get a whole bunch of benefits for their on-prem environments that being hybrid gives them. I know I'm going to echo a lot of what you said. You know, it's the fact that, you know, Microsoft, like Microsoft has always had a huge heart and passion for enterprises running servers in their environments, solving problems locally. The fact that we've been able to marry that with, we're also doing hyperscale cloud. And we're using the same software in both places. It gives us a huge amount of flexibility here. And it's, you know, there's been a number of folks I talked to who are like old comensions like me who are starting to go bold on top where it's like, we're all there when it's like, do you remember when, like, thin workstations were going to be the thing? And we're like centralizing everything and thin workstations. And then now it's like, oh no. It's like, you know, anyone who's been in the industry for a period of time knows that like, yeah, though, like, the world changes, like our business expectations change, and the right architectural solution today might not be the right architectural solution in three, four years time. I'll always remember. I, like, I spend, I spend a lot of time chatting with customers, but also chatting with the engineering team. And, you know, one of the things that the engineering team is always looking for for me is to get some insight into like, hey, Ben, like, we go on stage and we talk about this stuff. Like, what's landing with customers? Like, what, what makes sense? What doesn't? And like, there was a good chunk of time when I was out there, we were chatting about cloud computing, and I was coming back to the engineering team and I was like, and no one's getting like, no, like no one's getting it. Like, we got work to do here. But there was a pivotal moment that I remember where, and this was before we started talking about hybrid cloud, you know, when we were very much talking about like, got your private cloud and got your public cloud and we hadn't yet figured out the need for this hybrid cloud. I remember talking to my engineers and saying, it's really interesting right now when I talk to like CTOs and CIOs and, you know, data center architects. Everyone has a cloud strategy. And everyone's worried that they chose the wrong one. You know, like, I would talk to people who are like, going 100% private cloud, and you can see that they're kind of going like, am I doing the right thing for my company? Like, am I, like, am I being foolish with my money? Like, like, is this the right long term pop? And conversely, I go talk to people who are like, yeah, we're all public cloud. And you could see that they were kind of thinking like, what if, what if business needs to change? Like, what if like, what then? And that's really kind of the power of our platform is that we can say like, yeah, like, do what makes sense for your business today. And if your business needs or your environment changes, because pro tip, they will. Then we can help you figure out how to get there. Congratulations for both getting through that entire presentation and that commentary without using the word synergy as well, because. And you've heard me do bad marketing. I can do bad marketing. Hey, we didn't say empower it either. And I do, I like, I disclaimer, I'm going to say I never said this. So like, this instance doesn't count. Not one super excited. Well, that's breaking the paradigm. And with that, thank you very much, Ben, for a very awesome and informative presentation. Yeah, always, always happy to chat about the technology man. Amazing. I just felt like I've had this big trip down memory lane. It's been great. If you enjoyed this presentation and you want to come and chat about it some more, come and join us at aka.ms. Once again, thank you very much for your time, Ben. Thank you.