 Yeah. Hello and welcome to a new learning life module, managing Azure Stack HCI clusters. I'm Kasten from Germany, 11-year MVP and with me is Bernard. Bernard, say hello. Hi guys. This is Bernard Frank. I'm working for Microsoft within the FastTrack team as customer engineer. Glad to be here. Yeah, Bernard. This is not the first thing we do together, right? We have a long-standing history teaching people Microsoft technologies. Right. I think it's more or less our 10th anniversary. However, I think we did more than 100 events together as I would say, and it was always a pleasure. Recently, we did not do so many, so it's a renewal. I'm looking forward to this one. Yeah, that's true. I had also a lot of fun in the good old days where we did a lot of on-premise stuff together. I remember Hyper-V system center, and you told me a while ago, even something like migrating from, was it Windows Server 2008 R2 to 2012 or something? Something like that, right? There were these migration seminars. But anyway, let's start with our learning life session. So with us, for all our viewers, if you have questions, we have help from FlowFox. FlowFox is also a German Microsoft employee. I know him also for a long time. He was formerly also an MVP, and Flow will help you in the chat if you have questions to answer them and maybe one or two questions he will forward to us that we can answer them. So in this module, we are talking about managing Azure Stack HCI clusters, and if you want to follow the module online, there is a QR code that you can scan with your device with your phone, for example, and then you will get to the website. But we have also the URL here. So if you type it in in your browser, you can follow the module while we're going through the module, and we will encourage you to do that, or at least afterwards doing the module and do the tests, and then you have another batch on your learning life profile, right? If you like batches. Yeah, and we'll encourage you to ask questions, at least in the chat, so feel free to do so. We are monitoring this and also have Flow on the line, and I'll try to answer them by myself as well. Yeah, feel free. So if you are a bit shy, please say where you are from in the chat. That's interesting for us, where you're from. We are now broadcasting in the European time frame, so here it is 10.30 or 10.33 to be precise, but you can watch these also afterwards at the Learn Life module, but it's not the same fun to be live and may ask, well, let's see if this is fun to watch. Let's see. So what do we have? Yeah, we have first, we describe the most common Azure Stack HCI cluster management tasks. I will do this part and do some demos because the slides are great, but looking at a live environment is always more fun, at least for me, especially if we have some problems. I love that. Then we are talking about updating a cluster operating system and the software. So an Azure Stack HCI cluster, usually we have a lot of virtual machines in there, and when we have to update the system, one of the advantages of a cluster is you can update it live while the VMs are running. So we will look through that. Then we will do a little bit monitoring of the Azure Stack HCI cluster. We will talk about Windows Admin Center here, where there are different screens where you can see all the good stuff in the Azure Stack HCI cluster and we will configure an Azure Stack HCI cluster quorum. We will talk about the quorum later, but in essence we need to avoid the split brain scenario so where we have, for example, four cluster nodes and then two nodes fail and the other two nodes don't know should they start the virtual machines or should they power them down? So for that we need a quorum in the cluster and we will configure that. And then we will implement or we will talk about the implementation of a stretched Azure Stack HCI cluster. This is a great new feature that is only available in Azure Stack HCI, not in the Windows Server, Hyper-V or Storage-Based Direct Cluster. I've done multiple of those so it's fun to talk about that scenario because it's especially very popular in Germany. I got the impression in Germany everything is stretched. I don't know why, but it's very popular in the IT environments in Germany. Then we will talk about maintaining a cluster node. There will be maybe a time where you have to shut down a node because you want to add memory or replace a network card because it's defect or whatever. And we go through this scenario and then something special because we use CRED SSP as an authentication protocol. We are not using Kaboros or if you decide to use CRED SSP to be more specific, there can be some challenges with CRED SSP and we will give you some advice what you can do. So that's an overview and now let's do an introduction of our scenario. Microsoft Azure Stack HCI offers streamlined management with Windows Admin Center closely integration with Windows Admin Center and we will see a lot of Windows Admin Center in this module because all the demos are done through Windows Admin Center. As in all learning life sessions, our scenario is the virtual Contoso company, Contoso LTD and we will talk about some requirements from Contoso to why they want to have an Azure Stack HCI cluster or why they are evaluating an Azure Stack HCI cluster. So I'm talking about that, how is your, the customers that you are talking to that are looking into Azure Stack HCI, where are they coming from? Like taking this before Contoso, our virtualized made up company, what's the reality look like? The reality. So the HCI scenario, so hyperconverged infrastructure is very popular. So it's, we had that in previous learning life modules, but if you didn't attend HCI and HCI scenario means we have our storage and our virtual machines in the same cluster. So you don't have to have separate sound storage for your high available VMs. And this scenario is more and more popular and Microsoft has a great offer in Windows Server. It's called a storage spaces direct and I've done that for the last, I would say seven years, but now Microsoft has a new offering Azure Stack HCI that is specialized to these HCI hyperconverged infrastructure scenario. A Windows Server is not. So in Windows Server, we have a lot of possibilities. You know, domain controllers, PKIs, RDS, DNS, whatever file server Azure Stack HCI is only specialized on the virtual machines running in a high available scenario. So I do some implementations where people had storage spaces direct before. So the Microsoft Windows servers HCI offering and they are changing because of the great features that are in Azure Stack HCI to Azure Stack HCI. But I also see customers who are moving from other virtualization platforms. For example, VMware with a zone, they moved to HCI because Azure Stack HCI has a great integration with the Microsoft Cloud Azure. And you can't get that with, let's say VMware. You can integrate it with Arc, also great technology, but Azure Stack HCI is from the same company than that Azure is. And we have some more deep integrations here. And for some customers, this is really very important if they have an all cloud step strategy and they still have some resources they have to run on premises. Because Azure Stack HCI is not some cloud offering what is that is running in a Microsoft data center like the other Azure services. It's in your side. So on premises, you have it in your data center in your sphere where you can touch the servers and everything. So great solution. And I see more and more people from other virtualization technologies or also Microsoft moving to Azure Stack HCI. So answer your questions. Yeah, that was a long one. So if you answer every question so long, we end up in having time problems. Don't ask questions, right? Yeah, sorry about that. But yeah, it's a good one. So yeah, so let's keep on. It was too long, you're right. So let's look at managing Azure Stack HCI clusters. So in Contoso, they want to transition to a new operation model that uses the capabilities of Azure Stack HCI clusters. You want to identify the optimal way to perform the most common cluster management tasks. So let's see what we can do or what we have to do in an Azure Stack HCI cluster. The most common Azure Stack HCI management tasks include changing cluster configuration. So if we have a cluster, we might want to change the access point. That's a cluster name. We maybe want to change the note shutdown behavior. We will talk about that. Maybe you will have a special security security demands in your company. So you want to encrypt all your traffic. We have to define a cluster witness so that our cluster has a quorum. And we can influence how a cluster will balance our workloads. So we will look in all those scenarios. And there's something special. Maybe you have virtual machines that should not run on the same nodes. For example, an active directory. You have maybe two active directory controllers and you want to separate them. There's also possibilities for that. I think we don't talk about that. Yeah, it's in there. And there is the diagnostic data level. So that are some of the cluster configuration tasks that you don't do every day. Normally you decide how to do it and then it's there forever. Then we can change the storage caching configuration such as the persistent in memory cache. We will look at that shortly and talk about it, what it is. And then we have, of course, the possibility to change a lot of Hyper-V settings. So we will look there. And the most important part, an Azure Stack HCI cluster has to be registered in Azure. If it's not registered, we can't really use it for our workload. So we will look into that. So let's go. Yeah, to be honest. I mean, you will figure out that really soon. So if you can install, you can set up the system. However, if you want to, you know, place a virtual machine into your cluster, it will say, you're like, hey, this. Can't do it. Can't do it. Register first. And don't be afraid of it. I mean, you do get a spare time. Let's say if you're registered against Azure, it won't produce costs right away. So you have 60 days, I think. Yeah, it's 60 days. Before the counter starts. So don't be afraid of it. Yeah, Azure Stack HCI is an Azure service. So it's built through Azure, but Microsoft doesn't, as you say, doesn't build the first 60 days. So the first 60 days are free. And after that, you have to pay for Azure Stack HCI. And very important, you can use Windows Admin Center for everything. And I prefer that, but you can also do every task we do here with PowerShell. And to be honest, Windows Admin Center is like a PowerShell translator. Everything you do in Windows Admin Center, you can do with PowerShell, or it uses, it leverages PowerShell. And there are even the scripts. Most of the scripts are available and you can have a look into that. So let's switch over to my demo scenario. Change the cluster settings. So I will, let me see, where is it? Here it is. So I'm now live on an Azure Stack HCI cluster. I have a four node cluster here. And now I have to go to the slide so that I know what we do. We are in the cluster settings. So this is a dashboard of the cluster and we will come back to that. But now we want to change something in our cluster settings. And therefore we have this button down here, settings and the cluster settings are here. So first we want to have a look at the node shutdown behavior that is here. And the cluster usually, if you shut down a node, so imagine you go to the console of a host and you type in shutdown or you use any management tool and say shut down the host and you forget that this is a cluster member. And there are virtual machines running on those nodes. So usually you want that these virtual machines are live migrated to another cluster member. That would be the normal way. And this is when you have this checkbox set, the virtual machines are moved to another host and the shutdown is delayed until the virtual machines are moved over or I think after 50 minutes it's shutting down the cluster anyway. So you shouldn't normally just shut it down but if you do it by accident it will move the nodes. There are rare occasions where you don't want that because if you have to shut down all nodes at once you may be because your UPS is signaling that the power is off in 10 minutes. You maybe want to shut down all the nodes at once and don't care about the VM. So that would be something where you disable this behavior. Normally I would not do that. So let's see. Or in a test environment where you want to save some money, right? So might be, so where would that run this test environment? Then we have a lot of possibilities here with our storage spaces and pools. So in Azure Stack HCI we use storage spaces, storage spaces direct and if you want to you can change a lot of settings here. To be honest, you should be very careful what you change. For example, if you change the always retired failed drives to no that could be very problematic because the cluster notice if a drive fails it should retire it and then rebuild processes or repair processes will occur. If you turn that off, it doesn't occur and that can give you trouble afterwards if multiple drives would fail. Other things we have here and this is the actual Azure Stack HCI scenario. The learning life module was done with Azure Stack HCI 20 H2. We have now 21 H2. So there are more capabilities here in Azure Stack HCI 21 H2 than in 20 H2. For example, we can now provision thin volume. So they will only take up the space for the data that is in the volume. In your scenario or when you go through the learn modules on the screenshot, we have still not the possibility here to change that it's only fixed. And you have here a Windows Server 2019 thing. So we can change a lot of things here. Really think about it. If you change something here you should have the knowledge what it does because otherwise you can get in trouble. So let's have a look at the Hyper-V settings. There are many. So here under the settings we see multiple Hyper-V settings. And the first one we look into is the enhanced session mode. And the enhanced session mode is a great feature. Usually it's turned off. I have turned it on here. So normally it's turned off. You get the capability when you connect with the VM console. That's a console where you can connect to a virtual machine. Usually it's a very simple console. So you don't have a clipboard there. You can't attach drives to it and so on. But if you want that. So if you want the RDP experience that you have when you connect to a server or a server through remote desktop services or RDP, remote desktop protocol. If you enable this you get the same experience. So you have a great clipboard. You have printers connected. You have even, you can even connect drives but be aware that there is a security risk. So if you connect your drives from your local machine into the VM. The VM can access those drives. And if there is something evil in the VM running, malware for example, there would be theoretically a possibility to access your local drives. Werner, do you want to add something to that? Well, I usually turn it on as well for getting copy and paste like functionality. Sometimes it's a bit tedious to copy stuff into a isolated virtual machine. So yeah, it's very convenient. I turn it, you have seen it. I have turned it on also, but you have to be aware of the security implications. That's pretty good. Yeah, and the screen sizing is also sometimes very handy. You could do full screen just by the move of a slider. And that's, yeah, for demos, it's really good. Yeah, it's really cool. You're right. So let's go to one last thing. That's the Azure Stack HCI registration. And this cluster is registered to Azure. You mentioned before, if you don't register it to Azure, you can't really run VMs in it, at least not high available. So you have to do that. I've done it. You see here the cluster is registered. And it's also ARC enabled. That's a new feature with Azure Stack HCI 21 H2. Then you can also use Azure ARC to do great things with your cluster. Here we see the subscription. I shouldn't have shown that. But anyway, it's a demo subscription. That's not too bad. Well, I hope not everyone has access to your subscription though. Bear in mind that, you know, if you have any proxy-like scenarios that you need outbound internet access to some resources here for that. So if you run into any issues here, it's either naming resolution problems, right? Or firewall or proxy issues that happen here. So usually be aware that this is, you know, an area of problems that could arise. Yeah. Yeah, Ben, I have a question for you. So if you said you have to have internet access, you have to have the possibility to communicate with Azure. So do I need a public IP address for that or how does it work? No, you don't need a public IP address. However, it's outbound HTTPS traffic to specific URLs in order to register it against Azure, at least as far as I know. And you don't need to have it persistent, right? So you can't be, you know, completely on an island without any internet access that won't work, right? You need, you know, as you could see at the bottom here with the sync button, you need to have at least once in a while access to Azure in order for the cluster to report, hey, I'm alive, I'm still there. And that's, you know, that's the thing that should go online. Yeah, you don't, yeah, you have to have access to internet even from private IP addresses or through proxies. And you don't, it doesn't have to have a permanent internet connection. I think at least every 30 days, right? So let's come to some the storage settings we had already. So we, let's go to the knowledge check and we talked about some here. So let's first talk about the scenario. Contoso information security requires that all the network traffic for mission critical workloads is encrypted. You need to ensure that all SMB traffic between cluster nodes is encrypted. Which type of setting should you configure? So Contoso has an information security department that requires encryption in the cluster traffic. And the question is, where can you configure that? We were not specifically on that point in this on the screen, but it should be possible for you to find it out. So is it A, a cluster setting? Is it B, a storage setting? Or is it C, a Hyper-V setting? And you can vote at HTTPS colon slash slash aka.ms polls and we give you some seconds to do that and then we show you the answer. So Bennett, do you have any ideas? What's the right answer? Yeah, I have the answer right beside me, to be honest. Well, but let's think about it is if it would be a Hyper-V setting, that would be Hyper-V managing SMB storage kind of things. Well, could be, but a little bit far away. Storage, well, this would be a little bit maybe too generic, right? So it's all about the cluster, having to handle the traffic, replicating it from one node to another. So my answer would be, ta-ta. Hey, let's show it. I think we had enough time to get it right. So the cluster settings are right. Maybe you can show it in the, I can show it here. So in the old admin center, we had a security part under cluster, here's cluster traffic encryption, but now it links us to a new tab here, the security. So if you have the old Windows admin center, everything is involving at Microsoft, even also Windows admin center. So here's the extra traffic encryption, but there is a new tool here, security with the actual Windows admin center version. So this is 21, 10.2. And here we have the cluster security. And here we can say we want all our storage traffic encrypted. So let's go back to- We are keeping things moving in there, in Windows admin center to keep your brain working, right? Of course. So let's talk about updating an Azure Stack HCI cluster. Contoso IT currently dedicates a significant amount of time and effort to maintain their computing environment. And we will learn later that Contoso is still on Windows Server 2012 R2. So they have to move to be in a supported operating system because Windows Server 2012 R2 is out of support next year. So if you are still on Windows Server 2012 R2, please think about moving your workloads to a newer server because after 10 years, and yes, it's already 10 years for 2012 R2 next year. And for 2012, Microsoft supports the operating system 10 years and after that, there are other options, but you should go to an up-to-date operation. Well, if you raise that point regarding the supportability or supported metrics for Windows Server, how about Azure Stack HCI? Are you going to talk a little bit about the supported? Yes, and we will update parts about Azure Stack HCI in the slides. In the slides, it's in the slide. We will talk about it. So with Azure Stack HCI, we have a shorter cadence. So when you look at Windows Server, usually every two to three years, we get a new version. The actual version is Windows Server 2022. We are now in Windows, not in Windows 2022, we are in the year 2022. Last year, I think in August or September, a new server version came out, Windows Server 2022. Before that, it was a Windows Server 2019 that came out late 2018. So that was three years. And I think the next Windows Server will also be available in two to three years. In Azure Stack HCI, that's much shorter because we have a more dedicated setup. Azure Stack HCI only addresses virtualization, software defined storage, software defined networking, and we get a new version, frankly, every year. So in this, we expect another version of Azure Stack HCI this year, and that is supported then not 10 years. So Azure Stack HCI has a much shorter support cycle, and we will talk about that. So Azure Stack HCI consists of multiple Microsoft-validated third-party physical servers running the specialized Microsoft operating system, optimized for Hyper-V Converge infrastructure. We talked about that. And important in this sentence is third-party physical servers, Microsoft-validated. So an Azure Stack HCI, you should only run, at least for production, on a certified Microsoft-validated certified Azure Stack HCI node. And we have multiple vendors a lot who provide those servers. So that's the first for a supported environment. And Microsoft here is what I was talking about before. Here is Microsoft support each version of Azure Stack HCI operating system for two years by providing annual feature updates that customers must deploy within a year following their release. This is important. Azure Stack HCI, you have a subscription. And the operating system will be updated with feature releases every year. And you have to install them at least after a year because after two years, the Azure Stack HCI version is not supported anymore. So when it comes out, then a year, the next version comes out and the support ends a year afterward. So to be in a supported scenario, you have to update your Azure Stack HCI environment, let's say every year. And how do you do that? And you don't only, hopefully only install updates if there is a new version coming out. It's also done through the update mechanism. So you update your Azure Stack HCI environment to the next version also through, excuse me, through the update mechanism. But hopefully you will also install cumulative updates that come out every month. So how do we do that? If we do that by hand and there is a mechanism for that, it's called cow or cluster where updating. It goes this way. We will place a local node into maintenance mode. We will pause it and all the VMs or we will pause it. We move all the clustered roles from this node that we want to shut down or patch or whatever to another nodes. And that is true for all Azure Stack HCI environments except the very new one. We have now the support built 2022 announced a single node Azure Stack HCI node or a single node Azure Stack HCI installation. There we can't move the resources somewhere else. So that's a problem. We have to shut them down. But usually in an Azure Stack HCI cluster you have two or more nodes. So the workload is moved to another node. Then we install the updates. And normally we have to restart the server. For most of the updates, we have to restart the server. And then when the server is starting again, it will join the cluster again. We will terminate the maintenance mode and we will move back the cluster roles that were running on this node. And then we go to the next one. That would be the manual, how you do it as an administrator. If you don't use cow. And as you maybe think cow is doing exactly the same. So cow is standing forward. I don't know if you have mentioned it already. It's a cluster aware updating mechanism. So it's a cluster role that is installed and it can update the cluster. All a human would do. And one more thing that is very important. Most human don't know that. One more thing is done by cow. It's automatically running and it will install all the patches on every node. So you don't have to do it by yourself. You can do that, of course, but the cluster aware updating mechanism is much more convenient. You can watch it, what it's doing, but it's doing all the heavy lifting by itself. And don't be afraid. This feature is not brand new. It's out there for quite a while. So it was many years usable for Windows Server in previous times. However, it's being more prominent here in Azure Stack HCI because of the shorter update cycle. And that could be of help. Okay, let's see. What else do we have? Let me see, what else? So this is the important thing. Cluster aware update is not only for an Azure Stack HCI cluster. We also have, at least when we have a Windows cluster, we can do file server clusters, SQL clusters and everything of the good stuff, but in storage spaces direct. So the storage part of our Azure Stack HCI cluster, it's very important that you don't reboot a node when still repair jobs are running. So imagine if you shut down one node and you have your virtual machines running and the data is stored on multiple nodes in an Azure Stack HCI cluster. So now if you shut down one node and a VM writes some data, it can't be updated on the devices on that node. So Azure Stack HCI remembers, oh, I have to do that afterwards if the node is coming up again. I have to update the data there on the devices. And depending on how long your cluster node was not available in an update, it's a short time. Usually it's five to 15 minutes that a node is not usable and then it's up again. But imagine you have a hardware problem with a cluster node. It maybe takes 14, let's say one day, two days unless the hardware is repaired. There could be a lot of data that couldn't be installed on the devices in that node because it was not running for two days. So the storage jobs, they will run when the cluster node comes up and you shouldn't not start another node when this is running because then you risk more data. So Cow is also aware of that and waits for the storage jobs that they finish or wait that everything is healthy on the cluster. That's very important. And many people who patch their storage bases directly environment with Windows Server or even with Azure Stack HCI who do that manually don't know that or don't think of that and they can get in trouble. So Cow is the better choice. It knows all those things. So here we have a screenshot and I can show the same in my demo environment. So if we go here to Windows Admin Center, well, what did I do? I clicked wrong. Here we have updates. And when we choose this update window and of course I have to lock in. So it will check if the Cow role is already installed and if it's not, it wants to install it. You see it here, add cluster where updating role to the cluster. And I can do that now. If you choose to move your cluster to an OU, I will click that and I will show you what I mean. So in my environment, my cluster is in a special organizational unit. It's not in the computer directory. So there is a requirement. If you move your cluster in an OU and the cluster, when it installs the cluster where updating role, it needs a computer account. It creates a computer account and with the rights of this computer account, the cluster where updating is running. So not I'm creating this computer account, the cluster is. And that the cluster can do that in this OU, it has to have the rights to do that. So if I go to property and to security, you have to add your cluster to the OU, the organizational unit, and give it the, at least the right to create child objects and delete child objects. And there are more specific rights you can do. So you can even do it more granular. But if you do that, now this cluster, this computer or the cluster has the right to create a new account, a new computer account in this OU. And I see that often. Yeah, this is specific. If you use that feature in a demo environment where you may not have organizational unit and you put all everything in the computer group, then you don't run into this issue. But as you said, this is maybe not the best practice, right? So because you wanna have certain group policies, especially for your Azure Stack HCI environment, and then you want, if you enable cow, you would need to have these permissions, right? And that's a very important point you make, Bernard, group policies. Another advice, when I'm in, so an Azure Stack HCI cluster has to be member of an active directory. And many people use group policies for many great things in their active directory. So what I see sometimes at customers that group policies are, how you call, for urban inherited, thank you, Bernard, inherited through all the OUs and you get some weird settings on the Azure Stack HCI cluster. So please do a special OU for your infrastructure machines like Azure Stack HCI machines and disable inheritance from the OUs and create your own group policies there. Because I've seen many weird things, especially with group policies that are inherited and there are many settings, maybe for years in there, they're developed over years and that can cause some trouble. Or at least be aware of that. And if you do a proof of concept and try things out, maybe it's a good thing not to start with a computer's group in your own active directory, but do maybe a special OU and see how that works with your Azure Stack HCI then. Great advice. So here now we see, we installed the cow role and now we should have, here it is, you see it, oh, we get another one. Yeah, there is another one. This was created here. And if we go back to Windows Admin Center, it already checked the cluster nodes, what updates it wants to install. So it's on the actual updates and you see we only are offered Microsoft Defender anti-virus updates. So that you can do on a daily basis. If you want to, the nodes don't have to reboot for that and cow is aware if an update needs a reboot or not. Okay. Time-wise, I think we should move a little bit faster on. Yeah, let me see. So again, we have a knowledge check here. Yeah. And I think, let's Carson, because you already showed the answer sort of like, you need to enable credit SSP for that. Sorry for the spoiler. If you want to do the questionnaires, feel free to do the online training. We are saving some time here because we still get a lot of things to cover. Yeah. And I'm slow at talking, right? I mean, that's good. That's useful information. So however, the slide tag is pretty. It's pretty. It's pretty thick. Yeah, you can say that. But we have still 49 minutes, right? So the next chapter would be, we skipped the knowledge check. The next chapter would be monitoring an Azure Tech HCI cluster. And I think that's still mine, right? Yeah. Go for it. And monitoring is quite something that is really important. Of course, an Azure Tech HCI cluster, it cares about the virtual machines and everything, but there can still things happen. We have hardware involved. So our Contoso guys struggle with different problems. For example, fragmented computer environment with lack of unified monitoring and management solutions. They don't have a management solution so far. So they are looking in their POC. They are looking at the monitoring capabilities of Azure Tech HCI. And we will talk about that. So Azure Tech HCI cluster monitoring, what do we have to monitor? Because the Azure Tech HCI architecture is not the easiest one. There are many things moving parts and so on. So we have to have a look at, for example, the cluster. Cluster or the clusters, if we have multiple of them, those. We have to check the cluster nodes if they are okay. We have volumes for our virtual machines. They are stored in volumes. Volumes, the data of the volumes are in drives. So we have SSDs, NVMEs, HDDs, and those drives can have problems. They don't work forever. So there will be maybe a defect drive. We have to check those. And our virtual machines, we want to monitor our virtual machines because at the end of the day, the most important part in the Azure Tech HCI cluster is the virtual machine, the workload. People are working with them. Your company contoses only working if the applications in those virtual machines are available. That's all the reason why we are thinking about the cluster to get more availability for our virtual machines. So we can use Windows Admin Center. We have a lot of dashboards in Windows Admin Center where we can look at the status of all those different things and even that something I like very much is the performance. Because you don't only see if something is okay or not okay, you only see how fast it is or how slow it is. So could we say that Windows Admin Center is some sort of like the single pane of glass of all these different things to look at? Yeah, you can say that. That's really true. You can create Azure Stack HCI, an Azure Stack HCI cluster with Admin Center. You can manage it and you can also have a bit of a monitoring capability. But it's not the usual monitor solution. So if you are at a big company, you have something like a system center operations manager or other solutions or even a great one is Azure Monitoring. When you use Windows Admin Center, you have to be proactive. You have to open it. You have to check all the things by hand. But with a good monitoring solution, you have agents that are installed and the monitor solution will inform you if a disk fails, for example. With Azure Stack with Windows Admin Center, you have to look into Windows Admin Center if it's okay. So you mean the proactive alerting things that either you mentioned SCOM like and there is as far as I know, and I have tried it out. There is a management pack for Azure Stack HCI that you could use in combination with Microsoft Operations Manager to monitor your Azure Stack HCI cluster and your nodes and your health of your storage subsystem and whatnot. Yeah, and you can use Azure Monitoring because your Azure Stack HCI cluster is connected to Azure anyway. So why not use a modern monitoring solution from Azure? But you are right. You can do it with SCOM. You can do it with different under different other third-party vendor solutions. So it's quite open. Like Windows Server, we have the same management interface. So let's have a look at some scenarios in Windows Admin Center for monitoring. So let me see where we have a lot of things here. Cluster related alerts, list of cluster nodes, list of disk and volumes available on the cluster, list of VMs hosted on the cluster, total cluster CPU usage that I really like. You see how much CPU do the VMs and the workloads use in the whole cluster? How much memory the next point do they use? We can deep dive into single things like VMs and see all these information. The cluster storage usage, total number of IOs. So how many IO operations are your VMs doing? Are they increasing? Are they decreasing over time? We have also some nice statistics here. Average latency of disks in milliseconds. So all great stuff. Let's see some of those. Let's look at the servers. And I do that live, of course. We go back to our live system and here we have the server servers button and we see the server scenario. We have a summary. Now it's collecting all the information I was talking about. We are even loading some charts and I hope that works because I had a little bit problems with, yeah, here we are. So you see we have, in the moment, there's not much running on the nodes. We have only 2%. But what I like a lot is here, you have these statistics over the last hour. So we see the last 60 minutes, one hour, but we can also switch to the day. And I started the cluster in the morning, so not much here, but you can imagine. Now we would see our CPU usage over a day and even our memory. And over a week, over a month, over a year. So we have some great statistics in the Azure Stack HCI cluster. We can also see the statistics per node. You'll see here one is in maintenance mode. We did that before, so I will resume it. Let's resume it and we go back to servers. And we see the information, how long is the uptime, the manufacturer model and so on. Here you see the CPU usage and the memory. So a lot of great information here. And a tip, most of the time, you have a column picker in those overviews and you can choose more information or less information. So we can add even more things. So let's go to the next part. If that's, let me see, cluster. All right, the nodes. If that's not enough for you, so if you want to deep dive in counters that are not available here in the graph, there's even the performance monitor here. It's a graphical version or a Windows Atman Center version of Puffmon and you can create workspaces here. You can add counters, you get your graphs, you get your, everything you can do with performance monitor or nearly everything you can do. And then you can collect your counters and see in detail how all the stuff is performing. Let's go to volumes. We talked about volumes. So we have an overview over our volumes. We see the IOPS, we see the latency that we have, the average latency of all our volumes and we have the throughput, you see here up until here, we had a lot of workload running and now it's not running anymore. So I will start it again. Let's see here. Let's just start them, okay. And then we will see more statistics here. And with the volumes, we have a lot of interesting things. If we choose inventory, we see all the volumes we have. We see the name of course, the status. Are they okay? Are they healthy? Or is maybe a disk not working and we see some impact at the status. We see the file system, we see the resiliency. So here we have some volumes with a three-way mirror, some volumes with a mirror accelerated parity. We see the size, we see the usage, we see the storage pool. And as I mentioned, we had also the IOPS but maybe you want to add the latency, average latency. That's also a very important data point. So now we see the IOPS. In this volume, we have 23,000 IOPS and we have an average latency of 500 microseconds. So 0.5 milliseconds. If I go into the volumes, we have all these great data here. It will load the statistics, the hourly, the daily, the week. As you see here, I started the VMs and now we have more latency than before. We have more IOPS than before and we have more throughput. So I really love this about Windows Admin Center. We can really see what's happening there. And another thing, a little bit more deeper, we have also the drive. So the volumes are an object where our data lives. They are unusually formatted with ReFS. We can look into that, there are all our VMs but our storage-based direct system, Azure Stack HCI system consists of multiple drives. And you'll see here in the drive overview, we have 24 drives, they are all healthy. Nothing is critical or has warnings. That's a good thing. But if there would be a drive that had a problem, we would see it here in the overview and we can drill down in the inventory and we'll do that in a second. But here also you see the capacity. So we have only NVMe storage here. And we have, lucky me, you know I do a lot of Azure Stack HCI and I always think you have to have those stuff yourself if you implement at customers and train to trainings, right? So you see here we have 34 terabytes of devices and depending on how we use our volumes, if it's a three-way mirror, it will take up for a terabyte usable space. We need three terabytes in the storage pool. And you see here, I have, let's say, 23 terabytes already used. And they are still 11 or 12 available. But here's also this area that has a, how you call that. It's not the normal gray part. It has this special color. And if we leave this unused and the drive fails, the system can repair itself because it has still space to restore all the data that was on the defect drive. What's happening now with my monitor? It's back, okay. So that's very important, but we don't have the time for that. So let's look into the statistics of the drive. You see here a lot of informations. We see the model. What is it? We have the serial number here. We see the status of the drive. It's okay. We see the site, what it is, where it is exactly in the system, which slot it is. And what I recognize is that you have, like you should, right? The same firmware on every disk. Now I have a purple screen. Do you see also a purple screen? No. You see the screen, but I don't see it anymore. That's if you do it live. So let's see how I can move it to another. What's happening with my monitor? Check your cables. Unbelievable. So let me see if I can move it. So what, well, Carson is moving his screen and trying to fix that. Yeah, it's a good practice, or at least you should take care that the firmware of every disk that you use is consistent, right? And even with the hardware that you have in your cluster system, make sure that you don't have a colorful bunch of everything mixed into one cluster, right? So you should make sure that you be consistent so that the cluster can perform well, right? So, okay, so that was the monitoring part that is within the Windows Admin Center, right? Maybe you can take the presentation and I try to fix it, but we are nearly done with the monitoring. There was only the VMs, right? But I don't know what with my monitor is. It's really me. So you should be able to see my screen. Hold on, right? Okay, so now I'm have a look at the presentation where we are right now. So sort of going through the screens and pushing a little bit forward in the presentation. So there is also a health service that is dedicated also, giving you insights on the storage space is direct subsystem, which is important because storage is one of the basics that must be in place. Otherwise your workload will be corrupted and maybe not booting up. So it's good to have a health service around that. And even though these, what comes back from this health service is reflected into the Admin Center that you could get a feeling about the health of your storage subsystem. So let's push it a little bit further. And we do have a knowledge check on this, right? So I read it through, give you some time then and then you are asked to do the poll. As part of evaluating the functionality of Azure Stack HCI for your company, you decide to implement alerting and in response to performance or stability issues affect your clustered workloads, which technology you should choose to provide the alerting functionality. Your solution must minimize the administrative effort. So, and we talked about that Admin Center is proactive, meaning, no. Reactive. It's not proactive, it's reactive. Yeah, it's reactive. You have to do the things, right? Other than that, I don't give you any more hints. So if you want to have active alerting, right? Where should you go? Now it's your time for the poll. Yeah, I think PowerShell would be not... You have to type PowerShell, right? PowerShell is always the right answer, however. At least if you ask me. I'm a big fan of PowerShell, as you can tell. Me too. But yeah, in this case, it should be an easy solution, right? So maybe even if you could do something like this with PowerShell, the right answer is Azure Monitor, right? So you do have the Azure Monitoring pieces built into the Admin Center portal, meaning, and I'm right here in this section. So I'm showing you my cluster, which is, yeah, much smaller than the one that Carson did. At least my one is working. No, my monitor is working. My cluster is working, my monitor is not. Let me see, let me look behind my monitor. No, no. Right, okay, so yeah. Go on, sorry. I go on. Okay, so what I configured is I was configuring my Azure Stick HCI cluster to use Azure Monitoring. When you never have done that, right, and you hit the selection here, Azure Monitoring, it will ask you for, hey, I found these two nodes, do you want to hook it up to Azure Monitoring? And then you are handed over to Azure in order to do that. I've done that so far. So I have the green buttons here, meaning these two guys here are getting the agents that are required to gather the information from these nodes, send it to the Azure workspace, to the log analytics workspace, and put it there in the data source so that you can run your log analytics queries onto these data that has been forwarded to Azure, right? Or Azure Monitor has also the alerting functionality, which once the data is in Azure, you can sort of automatically trigger alerts based on certain queries that you define, right? If you are new to these Kusto queries, it's kind of a little bit difficult to start with. They are a mixture of PowerShell and SQL, T-SQL language for accessing data, like, hey, if the CPU usage of these two nodes is higher than 80% within the last five minutes, then do give a result back. And this is something that has been done already for you. So if you click on the alerts and actions and say, yes, take the ones, you get this list already prefilled for you. The only thing that you need to do is in your Azure portal, give it an email address where these alerts are sent to and then you're up and free to go. So let's have a look at my Azure portal if I can show this to you, right? So in the Azure Stack HCI, in the Azure Monitoring, let's go first over there to have to look in my cluster view, which is this cluster here we were just looking at. I see these two nodes, right? And if I select the capabilities, I can see that my logging and my monitoring has been configured and sliding up green, right? That's done via the admin sender portal. And then I get, you know, after some run or after some minutes, I get some data back that I could query. If I go into the Azure portal into the monitoring section here, there's a brand new section regarding Azure Stack HCI, right? And it gives you a so-called workbook, which is, you know, a dashboard that has been prebuilt for you with meaningful data coming from the product team. And it gives you, for example, the nodes that are or the clusters that are configured, but may not have shown up recently. This is, for example, the other cluster I have, which is shut down for saving power and consumption, Azure consumption. So, but it does give me a warning. Hey, this cluster is offline. It doesn't report back, right? The one that is reporting back is this one. And it gives you, you know, sort of like a view. Is it healthy? Does it have a failure or, you know, like disks, for example, or a storage job that has been run in the background because of a recovery, you would see it here. You also get a view on the servers. So, which are the nodes that are, you know, building up this cluster? Are they up and running? What's the CPU usage, memory usage? So it's a good way of looking at things here. And with that, you can even do further things, like go into your log analytics workspace, for example, and do the alerting section, and maybe, you know, trigger out or find your own alerts or build your own alerts, right? Okay, time-wise. Yeah, great segue to Azure Monitoring. You can do much, much more than with Windows Admin Center. But as you mentioned, time-wise, we have to go on. We have still, I think, three chapters. There are shorter ones, but let's, so I could present if you want to. I've moved everything to my other monitor. Great, thank you. So, we talked a little bit about a quorum. A cluster has to have a quorum. And how do we configure an Azure Stack HCI cluster for a quorum? So, the Contoso Scenarios, they had failures in recent times and they exposed flaws in their POC. So, they want to have more high availability. And we have two kinds of quarry. I think it's right, quarry or quarrums, I don't know, in Azure Stack HCI. There is the cluster quorum and there is a pool quorum. So, the cluster quorum, which operates at the cluster level and is based on the votes from nodes and the witness. You can implement such a witness, either as a file share witness or as a blob in an Azure storage account. Important that's on the next slide, a disk witness is not supported. So, with Azure Stack HCI, file share witness or Azure or a cloud witness. And we will see a video about the cloud witness, how you set it up. The other- But could you tell in a couple of words why we need that quorum? Yes, so imagine you have a four node Azure Stack HCI cluster and it happens that two nodes, so we have four votes. Every node has a vote, usually. So, we have four votes. So, now if two of those nodes fail, two votes are gone. So, the remaining two nodes have two of four votes or a better example is, let's imagine the network connection between the four nodes is not working. So, we have two, two, exactly. So, these two are running, they only see their two nodes but not the other two. So, two of four votes and the others are also running, they also see two of four votes. So, now how should they behave? They don't know are we the only ones running or should we start the VMs from the other ones because they are dead or should we shut down our VMs? So, it's the tie break scenario. So, they can't really decide. So, for that we need a fifth vote. So, that we have an uneven number, so an odd number of votes. Now, when we have another witness somewhere, let's say here, these C3 votes do not and the other way around these C2 nodes, two votes and they shut down the VMs and the others with the quorum they start the VMs. Sorry for the long explanations but I can't do it shorter. And there is a pool quorum that's also important which operates on the storage pool level and that's based on votes from nodes and storage resiliency. So, that's also important but we can't really influence that one. It's done automatically but the cluster quorum or a cluster quorum we have to provide. So, and for that here is a very important node Azure Stack HCI does not support a disk witness. So, if you manage your Azure Stack HCI with the failover cluster manager you can still create a disk witness because failover cluster manager is not aware about Azure Stack HCI. If you use Windows Admin Center you only have the possibility to do a file share witness and a cloud witness. And here you see how we do it with, should we show the video because it's another three minutes and I think we are short on time. Yeah, I think it's not rocket science. Like if you know the Azure portal it's like creating a storage account and then you copy the access key to that storage account that you have created and put this one into your Windows Admin Center at the right place where it says cloud witness and then you are hooked up to your storage account in Azure. However, bear in mind it is important then that you have the connectivity to Azure, right? So this one wouldn't make sense if your network is not stable enough, right? So, I have here in this cluster I have a file share witness. Here you see we can reconfigure it we can use a cloud witness. If we do a cloud witness we have to add some information from Azure and then we have our cloud witness. I have a file share witness. That's the other way we have a local share somehow. Here you see it's on a domain controller directory that I use as a witness. Okay, so go back to the slides. Here's the video. And now we do a small knowledge check or... Oh, let's skip the one. Let's skip the one. We skip the one and the right answer would be we have a resilient connectivity to Azure. So what do we choose? Of course, the cloud witness. Okay. And don't be afraid. This is not for Microsoft making a lot of money because you're using Azure resources. No, that's not, you know, a lot of consumption that is being produced. That's containing a couple of files. It sends a per month. So it's a very cheap offer and you can really use it. So next part is implementing an Azure Stack HCI stretch cluster. And I mentioned in Germany at least, stretching is very popular. So we have a lot of companies who have two rooms on the same campus, for example, and they want to do a part of the cluster in one room and another part in another room. So when would you say stretched is really stretched then? And I know that, you know, that this is not an easy one, right? So what does that mean? One cluster or is it two clusters that are just a part or what is it? I get this question quite often so far and maybe that will change, but so far if it's not in one rack, it's a stretch cluster. Or you can say it another way. Why do you stretch? You stretch because you put it in two sides and it's not depending on how far they are away. You put it into two sides to, if you have a fire or a disaster in one side, in one room, your cluster is still running. And if you do that, you have a stretch cluster. Yeah, so if you go that way, put it in different rooms, different size, you have a stretch cluster. Okay, so they identify the quorum quantification has, that's a quorum still. So with a stretch cluster, we use a synchronous replication between the sites, if there are rooms or if the sites are more apart. And Microsoft says you should not have a longer round trip time between the sites and five milliseconds because every wide right that you do, of the M is running in one side, it writes to the storage. The data is moved over to the other side. It's written there and then it's acknowledged and that's the round trip time. So every right is or it will be this time longer. So imagine you write to a disk, usually it's a one millisecond and now you have need five milliseconds to write it to the other side. So your right is not one millisecond, it's six milliseconds now. And if you go further apart, it's maybe even longer. So your applications are getting slower and slower. And this is why Microsoft says if you have replication synchronous replication between the sites, you shouldn't go have more than five milliseconds. And usually that's 20 to 30 miles or in kilometers, let's say 30 to 45 kilometers. So stretching a cluster doesn't make it faster. No, usually not because we need additional rights, we need additional resiliency on the other side. And if you want to be sure that your data is on the other side, you do a synchronous replication and that takes time, right? You're right, stretching doesn't make it faster. No, it doesn't. So if you create a stretch cluster with Windows Admin Center, there's a great wizard for that. And you can do that. I've done it multiple times. You can also create a cluster with PowerShell, of course, but it's very good in Windows Admin Center. So if you said stretching doesn't make it faster, does it make it cheaper then? No. Why not? Certainly not, certainly not. The same argument, you need additional hardware that you usually won't need and you need much more storage. So that's a good segue into this blue part. When we create a volume for our virtual machines and we don't have a stretch cluster, we create one volume, there is the data. If we stretch it, we need a copy of this volume on in the other side. So we have already two volumes, double the data. And for the replication mechanism, we need two more volumes, lock volumes, one on the site where the VM lives, where the volume is active and the lock, and one lock volume on the site where you replicate to. So this is much, much more storage, so more expensive, but you get more security. You are prepared for a disaster that your site is gone and you will still have all your workloads. And that's maybe the reason why this feature is so prominent in Germany because we are pretty afraid people living here. So this feature comes with a price, as I would say. So be aware, don't sell it too easily. You as administrator, it's not easier for you as well because you have to take care a bit more about what you're doing, right? Yeah, but it's great that it's now available like you do it with the SAN, you have two SAN systems, they are synchronizing. You have the same now with Azure Stack HCI, great feature. I don't wanna talk it small, right? It's a cool feature, however, with a lot of power, there comes a lot of responsibility. So we will skip the knowledge check because we have only 15 minutes to go and we have still two parts of the session, right? So will you take the presentation, Bernard? Yes, so let me go through some of the slides that I have here and we are digging down or going down a little bit into the maintenance kind of thing. So for example, we are talking a little bit about adding and removing cluster nodes, shutting down nodes, what to do here and how to use the Windows Admin Center for that. So basically, when would you, for example, remove a cluster node, Carsten? I mean, you do a lot of more productive environments than I do. So when is the time to remove a cluster node? Usually if one node is defect and you can't repair it, so maybe we have our stretch scenario and one side burns. So you don't have anything anymore in that side, so you buy two nodes, two new nodes, and then you have to remove the old ones and get the new ones into the cluster. Otherwise removing a node is not the usual case. Yeah, as I would say, it might be happening a little bit more frequently if you're playing with the things, if you have a proof of concept scenario, but in a day to day life, this is not something that you may do that often. If you do, then there's something else wrong, right? Yeah, so okay, let's look a little bit into the stretch cluster scenario that you were talking about because if I would remove or add a node to a stretch cluster scenario, would one node be enough? No, you have to add on both sides a node because we have to have the same number of nodes in every side. So you can't do three node in one side and two node in the other. It has to be three nodes and three nodes. So if you add the node in one side to four nodes, you have to add also a node on the other side, same model, same storage and so on. So if you were to put a mirror between these two sides, A and B, be aware that you should see at least more or less the same on one or the other side, right? So if there is a big gap, it's not good. Yeah. Okay, good. Let's go ahead and also let me mention this one. In order to get to receive support from our people here at Microsoft, you need to have a valid cluster report, meaning if you have done clusters with failover cluster, before you already know this, there's a cluster validation which gives you a report that you can run time to time if things changes. It's a good thing to have. Otherwise you may have problems opening a support ticket when the support guy asks you for a cluster report and this doesn't look too good. You're maybe alone with your problems. So running this cluster validation from time to time is a good practice, right? Okay, adding and removing nodes using the Windows Admin Center and let me jump over to my demo environment here. So hopefully you should see that. It's not rocket science. If you go into your cluster section here, there is the option for the servers. You need to look for the inventory stuff and then you select the node, right? So you do the checkbox and then these options here on the right, get active. So if I would, for example, tick this server here, right? I would be able to remove that node from the cluster, which is obviously doable, but not the thing that I want right now to demonstrate. However, you get your idea. Likewise, it is the possibility to add other servers to the cluster. Bear in mind, take the example with the mirror for the stretch cluster scenario. If the new node looks completely different hardware-wise, from that what you already have and you're trying to add this one to the cluster, it may work, right? But would this be a good practice? No, so make sure if you're adding new servers to a cluster, it should be same size, same CPUs, same disks, and more or less like a clone of the existing ones. Okay, so from your experience, Karsten, what do people do? I mean, if they are thinking about extending their clusters that they already have, is it always an easy task for them to buy the same hardware that they already have in stock? No, it's not, that's usually the problem. So when they add a node or need more, why are they motivated to add something to the cluster? Usually it's a storage, they need more storage. So firstly, I would add drives to the existing servers if I can, if I have still spare slots there, and if I can't do that, add another node. But in my opinion, it's very important, or it's important to get the same one. If you can't get the same one, because after three years, you maybe need more space, you can add another, not the same one. And in Azure Stack HCI, we have a great feature that the cluster checks out how the CPUs are and it uses a compatibility mode that the VMs can run on every node, yeah? But it's usually not the best scenario. Bernard, I would propose that we skip the extending of a stretch cluster because it's not too often and it's in the learning module, and let's skip to the next part because we don't have too much time left and I think the next part is also very important. Regarding the extension of these nodes, there's one thing that I want to mention because I have seen that and I really found that this is a cool feature. So we do have some sort of forecasting functionality within the Azure Stack HCI that uses an Azure service for doing the forecasting. So if you turn this on, basically what it does, it looks like onto your Stack HCI cluster for a longer period of time, evaluates the storage use and the CPU usage and its growth forecasting you, like prognosing you when you run out of capacity. So that could be a cool feature in order to add and also make for more sense in order to, for this hybrid story, like having on-premise hardware and using with Azure like services for calculating that out. Okay, and also just a couple of words as you do it to have it in the models, extending a stretch cluster is also possible. However, it's more complicated, but you have to do it with PowerShell. And it's comes with PowerShell, right? Okay, so let's jump a little bit forward. We skipped the knowledge check here and perform some node maintenance task, which what is a node maintenance task? Like updates, well, sort of is, but it was handled with the cluster we're updating, right? What cluster we're updating does in underneath, it's sometimes you need to do it manually, right? Maybe it's not an update thing, but it's a hardware replacement stuff or like a networking card is defect or something else went or went wrong, right? And you need to do a node maintenance here. Yeah, you can, what's the process of doing this? You need to make sure that there are not any storage related jobs on the run while you want to shut down a node, right? Because sometimes storage space is direct, needs to replicate out and reconstruct the health of your data that is written on all disks within the cluster, right? And if you are shutting down a node that is currently in the rebuild phase, is not a good idea. So make sure that you don't have any storage jobs running on this specific node, on this disks of this node, before you are doing node maintenance and removing even further resources from the cluster, making it vulnerable to resiliency stuff. Hopefully, sorry for my not native English here, I hope that you got the point. Yeah, so what you do is usually you drain the node. That means you remove all the roles that are running or being executed on that cluster that could be either a virtual machine or anything else that's cluster related, right? And then after these roles have been shifted over to the other remaining workers or to the other remaining nodes, then you can shut down the node and do your things and then reboot maybe the host and then get the other nodes or get this node back into the system again. So how does that look like from the portal? Well, maybe you have seen it already. So you have here this option here, that's the pause option. What it does, it does drain the roles. Let me just open up failover cluster manager so that you have a view that in order to switch things here, I do have a virtual machine running on this node, right? You could see it here. The on-prem virtual machine is running on that node here. And if I drain that or pause that node, what we should see is that it is live migrate or migrating this virtual machine or the workload that's been running on that node over to the other existing one. Let's go back and see. Well, that was fast. So this guy here is now running on the remaining node and you could even see this one, this guy here is paused. It still has a vote, right? However, if you shut down this node or restart it, the other cluster node could still pursue or do its job and run the workload successfully. So this could be done all using the Windows Admin Center tool and there we do. Yeah, that was the short demo that I would like wanted to do. Anything else that you wanted to add to this, Carsten? No, then you shut down your node, you repair, you change your network card or add memory or whatever you do when you have to do your maintenance. You power it up again and then don't forget to unpause it. That I've seen that in at customers, the nodes were in pause for quite a long time and that could be problematic. So we are nearing the end of this Learn Live module. So we will skip the questionnaire again and there is one last session about Cret SSP, but unfortunately, I don't think we have the time for that in the last two minutes, but it's in the module. So please read through the module. I would say it's not so important anymore. I had some problems with Cret SSP when Windows Admin Center came out and there were some problems, but nowadays it works quite fine. And if you have problems, look into this module, there are some troubleshooting tips. Yeah, that's regarding if you get asked what Cret SSP is like, because let's use one minute on this. No, maybe we should close. Okay, yeah, it's an authentication thing making lives easier for administration. People use it out there. So you could do it using Kabros, however, that's a bit more complicated. So Cret SSP is cool, it's doing its thing and it's used for remote administration. Let's switch to the summary, what we talked about. Yeah, we described the most common Azure Stack HCI cluster management tasks. We updated, talked about update cluster operating system and software. We talked a bit about monitoring an Azure Stack HCI cluster. We talked about configuring an Azure Stack HCI cluster quorum and then on the next side, implementing an Azure Stack HCI stretch cluster. We talked a bit about that, then maintain cluster nodes and the last chapter troubleshooting Cret SSP. We didn't have the time for that. So here's a reference to the learning module and that's it, right? Bernard, that was a lot, that was a lot of stuff we talked about. Yeah, sorry for the rush through and maybe a bit too much explanations on some parts and being too short on some other parts. However, you have it in the docs. Hopefully this was sort of over entertaining. Feel free to comment. Yeah, I want to mention something, Bernard. There is a fast track for Azure Stack HCI. So if you are thinking about Azure Stack HCI, contact your Microsoft contact person and ask them about the fast track program if that's maybe suitable for you. And with that, bye from me. And bye from me. Have a great one. Take care, bye-bye.