 Hello, everyone. Thank you for joining us on our session on hot patching today. And thank you for joining us on the server summit. I'm Vishal. I've been a Microsoft employee for more than 15 years trying to work with customers and help them solve the challenges that they face. And I also have Viraj. Viraj, you want to go next? Hey, everyone. I'm Viraj and I've been working at Microsoft as a principal and engineer for over 16 years now. And I've been part of the hot patching program from the beginning. And hi, I'm Tim Drilling. I have been with Xbox supporting the Xbox network for over 10 years now. Thank you Viraj and Tim for joining us on the session. So we have an agenda packed agenda for you. We'll be talking about what is hot patching. We'll also have a quick demo on hot patching. We'll meet our Xbox team who will tell you more about their experience with hot patching and we'll quickly touch on the future of roadmap on hot patching. So let's go. So let's talk about today's update experience. We have been shipping security updates for the last 20 years. Every patch Tuesday you will receive a security update from Microsoft that you would install on your machines and that would cause you to do a reboot. And there is a need for providing all the security updates for your machines because there's a rise in cyber crime today. We are seeing cyber threat to be the number one cause of potential threat that is affecting companies. There's a lot of pressure on our IT teams today. They continue to be under the pressure of making sure that the system is up and running. At the same time they have to make sure they are secure. Users don't want disruption. They do not want any reboot of the machine. They are always working and any disruption causes them a little drop in their productivity. We're thinking about users but think about factory floors where you have machines that are running and that's running the whole factory floor. Any disruption or any reboot or any downtime of those machines will cause the whole factory to have a downtime and that's not acceptable in today's day and age when business is running 24-7. Let's talk about a few feedback that we have actually received from customers. I picked a few for you to maybe some of them will resonate with you as well. Some of the customers that we met have said that, hey, patching is a lot more involved than you think. We have to back up machines. We have to make sure there's extra space. We need to make sure that the users are not using at the same time and this is a monthly effort. It is a lot of effort on a monthly basis for us. Then there are customers who are actually saying, hey, our systems are running very critical compute or actually are running a very critical data that needs to be accessed 24-7 and there is no downtime. How do you expect us to bring the system down and be able to do a security update on these machines? Then there are also customers who are saying, hey, we deal with it every month. Yes, we understand but it is actually taking a toll on our personal lives. We are actually having to miss personal events that are dear to us. These are pain points that may resonate with you as well. This is what we are thinking. We are thinking about a world where you do not need to do reboot or have less number of reboots when you have to deal with security updates. How do we do that? We have a servicing called hot patching. Hot patching is the ability to install your security updates on window devices without the need of a reboot. This is accomplished by modifying the in-memory code without restarting the whole process. Imagine that you have an application running and you are now having to do at runtime, you are patching the machine in a way that the app doesn't have to come down and it keeps running. What are the benefits of hot patching? One of the things that you would notice is the biggest benefit is that it accelerates security. Instead of having to wait for a few weeks on a weekend when you decide to actually do the update, you can now have these updates run on your machine at runtime on a weekday on working times without the machine coming down. That means you have improved the, accelerate the adoption of security for your machines and make them secure at the get go. You also improve productivity because now you're not bringing down the machines completely. You are just making sure the updates are installed, the machines keep running, the users are using it and you are improving the productivity of the entire organization. There is reduced cost, you don't have to be working on weekends to do this work, you don't have to have backup servers where you're backing up the data, you don't have to stand up new servers to take on the load while the security updates are installed. So it it nets out to be a reduction in the cost of your operations. Now let's understand how hot patching works from a release cycle perspective. Today you all are used to having a regular patch Tuesday come up where every second Tuesday of the month Microsoft releases security updates which are cumulative security updates for your machine and it not only contains security updates but also non-security updates for your OS. It's the big package that gets downloaded and installed on your machine and it gets released every month for the 12 months in a year and when you install the update it would need a reboot. Now think about the hot patching servicing, right? In the hot patch servicing you only would be getting security updates that are necessary to keep your server secure. We do not release hot patches every month of the year but we think of it in terms of cycles. We think of one month when you receive the regular servicing package and two months following that you receive a hot patch package which will keep your machine secure but will not require you to reboot and so if you do opt-in if you do opt-in to hot patching service then what would happen is you at the beginning of the year in January you get your regular update and following that you will in February you will get your hot patch update right and in the hot patch update that you receive on February the data that's contained in this package is only the security updates that were not part of January but additional security updates that keeps your machine secure. It will not contain non-security updates. This is a very important thing to remember that non-security updates are not part of hot patches and that's why the hot patch package itself will be very thin. It will contain very less number of binaries but remember it will still be 100% parity in terms of security from your cold patch which was released in February on a patch Tuesday. So the full lcu that's released in Feb has the same security fixes as the ones that are shipped in your hot patch package so there is no security difference between the two but the only thing is it will not have non-security updates. It is released at the same time and date as your regular service package right so every patch Tuesday at 10 o'clock you will still get your hot patch package and you'll get your regular lcu as well. Following that in March you again will be getting another hot patch package and so that means again you do not need to do a reboot and say you missed the Feb update and then you directly install the March update it will be cumulative of the Feb update but not of January update so that means you will need to do a January update before you install any of the hot patch packages that we have for that cycle. Now this is the three month cycle that I was talking about it starts with a cold patch which is a January and then followed by two hot patches and then we go back to a baseline month where you need a reboot. So the reboots that are needed are called baseline months and the way you don't need a reboot they are hot patch months from a calendar perspective. Following that again we follow the same cycle that May and June becomes hot patch months you don't need a reboot and July you would need a reboot again. Now these baseline months where you do need to take a reboot allows you to also be able to patch your application that may require reboot any firmware updates or even any other updates that you think that would cause a reboot you can plan to be aligned with your baseline months. Of course security is first so you need to ensure that if you do need to install security updates then please do install security updates even if it means reboot right so we do not ask you not to do that right but from our perspective we do prioritize security over the rebootlessness experience. What does that mean? That means if in a pep when you are expecting your machine is not to reboot and there is a security vulnerability that would cause your machine to reboot and cannot be hot patched we would be causing the machines to take a reboot in order for them to be secure and so you would not have a rebootless experience for that particular month and but the next following month we will again provide you the rebootless experience. Dotnet updates are not part of the hot patch experience today that means if there is a dotnet security update that comes out in a hot patch month then that if you install it on your machine it will cause a reboot. We are working with the dotnet team to ensure that the dotnet updates are aligned more with the baseline months wherever possible so that we can provide that best experience to customers who adopt hot patch servicing and another thing just to again a reminder that hot patch always begins with the baseline month so that means you have to have a reboot and then you get two months of rebootless updates so potentially you would have three months without having to restart your machine and so if you do opt in you need to be on the right baseline before you get a hot patch so if you say hey I just installed my pep lcu and I want to now opt into hot patch you can opt into hot patch but then you will have to wait all the way up to mail for your first hot patch to be installed on your machine so that was all the cycle that we talked about hot patching now I'll hand it over to Viraj to give us a quick demo on how this works hey thank you Vishal for the great hot patch overview and it's really cycle so before we jump on the demo I would like to brief you on what the demo is how it is set up so we have two VMs created using the Windows Server 2022 data center azure edition both of the VMs have been wrapped with the January baseline image one VM is set up as a chatbot server and the other one is set up as a customer application through which we are mimicking the scenario where the customer will be talking to the chatbot and while their communication is going on we will be updating both of these VMs with a February hot patch with that let's just jump on the demo let us now kick off the demo by starting our chatbot server we observe the chatbot is now online by clicking the monitor server we're playing a scenario where an ideal user is having an online conversation with bot and the bot is providing response both of these VMs should be on January baseline release we can confirm that they are on the desired os version 2022-27 we're now going to kick off windows update scan on the client application VM we've got the February hot patch update as a kb article 503 4860 a hot patch update is getting installed and should finish in just few seconds by running the ver command from the windows command prompt we observe the os has now been patched to the newer version 2277 which is our February hot patch release version note there was no interruption on customer's conversation with chatbot for our chatbot server VM we're going to initiate patch update from azure portal the February hot patch update is available for this VM let's now start a one-time update process select VM and click next expending the preview of selected updates to be installed we see the hot patch update is showing up click next here customer can follow the prompt and choose options for post installation task and select next and then install we have successfully submitted install update request let's now look at the history tab from here we can track the update installation progress the VM has been patched within a minute and 15 seconds end to end from the azure portal we can confirm that the install update task is completed there aren't any more recommended updates pending for this VM let's now check and see if the chatbot server is patched to the latest version or not it is indeed been updated to the latest version one can also check the active hot patched binaries loaded in system memory from process explorer tool in process explorer click on the find handle or dll icon and type underscore hot patch dot then click on search button we now see all the active hot patch binaries their full directory paths along with their respective process ids we can see at least the dnsapa underscore hot patch dot dll and device metadata retrieval client underscore hot patch dot dll binaries are active clicking on the explore button from the properties windows of device metadata retrieval client underscore hot patch dot dll we find all the binaries that are part of february hot patch update in our install on the system this concludes our demo of hot patching windows server 2022 data center azure edition without any customer workload being impacted by a reboot great so that completes our demo and we observed that customer application was communicating with the chatbot server without any interruptions and the hot patch package was downloaded and installed on the system quite quickly now let's hear from our customer who has been using hot patch servicing from the beginning with that i invite team to share his experience on hot patching program thanks raj that xbox network has been involved with the hot patch team since preview and we currently are running almost 1000 sql servers on windows 2022 hot patch edition they are configured in availability groups but as a dba i have to say that hot patching is without question the best thing to happen to us since sql did introduce those availability groups back in 2012 while failovers during patching months are fairly short hot patching has been able to reduce our reboots by over 60 percent as variety or as vishal mentioned there are still occasional months where we have to apply out of band security patches or sql server patches but in generally speaking the availability has increased significantly for all the services that we host it's actually gotten good enough now that we are able to look at consolidation to help reduce costs and and still maintain that level of availability our dba team now spends much less time patching and is able to concentrate on improving the services for the gaming community uh in addition to the sql servers that we run xbox network also hosts a private windows active directory domain and its own dns hot patch has been able to reduce issues that we as the dba team have seen that were often related to authentication during patching and reboot of active active directory hosts and other dns issues the info team is also happy enough with the hot patching uh increased availability that they are looking to reduce the number of active directory infrastructure machines in the network which reduces complexity and also eases their their level of management the the sql server team in xbox network moved from 29 sql server 2019 on windows 2019 we initially took some of our hosts to the preview edition of windows 22 hot patch edition and then later moved those to the ga version and then the rest of our hosts to the ga version over the last three or four months we've been able to migrate its scale using power shell scripts and the overall process has been virtually seamless both from a sql server perspective and a windows perspective we also have been able to leverage the streamlined update management and orchestration that azure offers so overall our experience has been incredible for this my team is very happy with the way it's working and best of all the gaming community is experiencing greater greater availability and less downtime i can hand it back to vishal and he can explain how everyone else in the world can get these services thanks vishal thank you so much tim for the kind words so now that you know a little bit about hot patching you saw a demo now how do you actually start with creating hot patch images now if you're used to test using if you're used to creating VMs using your azure portal then simply go to your azure portal and when you are creating a VM you can actually select the windows server 22 data center azure edition hot patch directly from your azure portal and what you will notice under the management is you will see that the hot patch is enabled by default for these particular machines and the patch orchestration type is azure orchestrated what does that mean from azure orchestrated it means that azure will be managing your updates for that VM you don't have to worry about managing the updates for your machine azure will automatically install the updates for your VM on a regular basis there is no other orchestration type that you can select at the moment for the hot patch VMs so if you're used to schedule patching or any other method they are not yet available for hot patch VMs if you're using PowerShell then you can use this configuration for the publisher offer and this queue to create your PowerShell script to create your hot patch image if you do not provide the hot patch parameter then the default setting is hot patches enabled by default now for those who want to have a shortcut here's a shortcut to get to the portal so that you can go to this link aka ms get hot patch you will be landing on the portal site where you will be able to create this particular image immediately now let's talk a little bit about the roadmap on our journey and where we are going and looking forward to so we started with Windows Server 222 data center azure edition it's available today on azure only it's only available for gen2 VMs because of security reasons and there is no additional cost to your organization if you're using the azure edition VM to get hot patches moving forward we will have hot patching also available on Windows Server 225 data center azure edition again it's only available on azure because azure edition is only available on azure and it's again only on gen2 VMs and there is no additional cost that you pay for these VMs when hosted on azure now there have been customers who have also asked us like hey vishal you have been actually talking about all these machines being on azure but what about my on-prem devices on-prem machines that run business critical data that we cannot migrate to cloud or hey we have this VM Windows VM that's sitting on a third party cloud that's not on azure how do i get this benefit of hot patching on those machines so we have heard you loud and clear and we're actually coming to on-prem now with Windows Server 225 that are azure arc connected so if you have a data center or standard Windows Server 225 that you plan to deploy and you connect it via azure arc then you will be able to get the same benefits of hot patching as we do on azure VMs it will be available for third party clouds as well so if you have a third party cloud where you're hosting your windows VM you can also benefit using that you can also install now your data center and standard on bare metal machines on your on-prem and those also can be connected via azure arc and get the benefits of hot patching now there is a small monthly fee that would incur if you subscribe to the hot patch service via azure arc so that rounds it up with a quick overview of where we started the journey about start talking about the need for rebootless update we talked about the benefits of hot patching overall and what it actually means we saw a quick demo from Viraj and we heard our customer xbox who has been using hot patching so thank you for joining us on this journey and we hope that you can actually benefit from the hot patch service for your organization