 Hi, my name is Chris Bloom and I'm a senior architect in the storage BU. Today I'm going to talk about using local disk on VMware with OCS4. On VMware, we have multiple choices of using local disks. For one, we can use regular VMD case, which are VM disks that are directly attached to VMs. They can be based on pretty much anything. Then we have the choice of using RDM or raw device mapping, which maps certain RAID devices directly into a VM. And we can also use VM Direct Path IO, which can map local devices directly to VMs. This is actually what we're going to look at today. We will try to map local NVMe devices into a VM. This is probably one of the hardest steps that will involve many things to do. If you just want to use VMD case, then you just do not do the first couple of steps in the beginning. So there are some requirements that you need to satisfy to do that. You install the OCS4 operator, you install a local storage operator. You need to have local disks of some kind, at least three hypervisors. Like I said, we will be using local NVMe devices for the direct path IO. And you also need the ability to create three 10-gag VMD case on the same hypervisors that you dedicate for OCS for the SEP monitors. So how do you set up the local disk as PVs? You assign the OCS label to the notes with the local storage, install the local storage operator in its namespace. You can use the demon set to figure out the disk IDs for the local disk and the VMD case, either by using a demon set that's provided at this link here. Or you can do that yourself either by SSHing into the host or using OCDBuckNode. And then you create two local volumes in the local storage operator's namespace. One with the volume mode file system for the SEP monitors and one with the volume mode block for the local disk. And finally, to use these local disk PVs, with OCS, you create a storage cluster. So this is the full VM that you need to create. I've split it up into two sections. So basically you put in here the storage class name for your local PVs with the file system mode. And over here, you put your storage class name for the PVs with the volume mode block. And that's basically it. So now let's look at how this actually works with a little demo. So first, I have prepared this OpenShift cluster here. It's freshly installed with OpenShift 435 on VMware. And this is the VMware cluster. So we're on Perth 7. These are all the nodes that make up the cluster. We have three masters and six compute nodes. We will be using the first three compute nodes today to attach the local disk, especially the VMDKs, especially the VMDirectPass IO. We need to shut down these workers because I know that nothing is working on the cluster. I can just tell them to shut down. If you would be doing this on a running cluster, then obviously you first drain those nodes and ensure that the parts are properly migrated off of these nodes before shutting them down. So when they're shut down, we ensure that these are running on the three hypervisors. So you see compute 0 is running on host, right Perth 3. So I want to ensure that they are all running on different hypervisors. So you see that compute 1 is running on 2, compute 2 is running on 3. So I'm going to move compute 0 over to the first hypervisor. Each of these hypervisors have one NVMe disk that I'm going to attach. And now they should all be on different hypervisors. So this is fine now. All right, so let's attach our local NVMe. For this, I'm going to add an NVMe controller and I'm going to add a PCI device. And you see over here, this is my PCI device. And I need to reserve all memory for VMware to forward that PCI device. So I'm going to do that. And then I'm also going to add the 10 gigabyte VMDK for the SAF monitor. So that and I'm going to do the same thing for the other two workers. Also note that when you're attaching a PCI device that's a pass through PCI device, then obviously there are some limitations on that VM. So not only do we reserve all memory and hard advocate that to the VM, but you can also not suspend my grade with the emotion or take a snapshot of that virtual machine. But that shouldn't be too bad for you. All right, now that we're done this, we power them back on. And they should eventually be ready again in the cluster. You can either monitor their state in the web UI or you can do that from the terminal when you're locked in. If you don't see that your NVMe PCI devices in the VM options, then you probably haven't configured them for forwarding. To forward them, they need to be configured for pass through. So you do that when you select one of your hypervisors you go to configure on a hardware PCI devices. Your NVMe device needs to be listed here as pass through enabled device. You can configure PCI devices for pass through by clicking this button up here. Then selecting your NVMe device like this. Do note that when you have your NVMe configured for pass through, it cannot serve as a data store anymore. So it's going to be dedicated just for that single use case. So let's hop over to the terminal briefly. I've already locked into the cluster so I can list the notes and we can see that the workers are already back online and ready for new workloads. And for this demo, we have four files. So first, let's look at the requirements. We first create the namespaces for the two operators that we're going to install and then we are all actually going to install them. So let's do that. So obviously the OpenShift Storage namespace is for the OCS operator and the local storage namespace is for the local storage operator. We label the OpenShift Storage namespace for monitoring so that Prometheus knows it should be monitoring. Let's hop back over to our web UI. Over here we go to operators, operator hub. We go to storage. We select the container storage. Install that. We install it to our OpenShift Storage namespace and we leave automatic. And then we go, while this is installing a background, we can go back to operator hub. We go to storage and now install the local storage operator and this will take a second. All right. And now that this is installed, we can go back to our terminal. And in here we do have two files, the local file and the local walk YAML. And we need to feed it the device path. And as a best practice, we're using the BIID device path because the disk IDs could change between rebooks. And so that we can easily find out the disk IDs and all the nodes, we can use the demon sets. To do that, we first need to label our nodes. So we get the node names and we want to, we know that the first three working nodes have the local disk. So we will just do oc label node compute zero and then the ocs label. And we do that for the first three workers. This is important so that the demon set knows onto which nodes to look for local disk. Now that we have that, we can verify that the label was successfully applied. By just asking for the nodes with the labels. So we see these first workers have the label. This is fine. So now we deploy the demon set. The demon set looks like this. Looks like this. And you see it's nothing too fancy. It's just running some bash to figure out the disk by these. Going back to the terminal. Let's apply this. The link to the demon set YAML is in the presentation slides. So let's check out the parts. And we see immediately that parts spin up on those nodes. Since we have three workers with the label. We have eventually three of those demon set parts. And they're running now. This is fine. So now we can go and gather the log output of those parts. And this looks like this. So we have three workers, compute one, compute zero, compute two. And they each have three disks. So I know that SDA is my root disk. So I'm not going to use that. SDB is the VMDK disk. And NVMe zero and one is the NVMe disk. And they are fortunately named the same on each node. So what I can do is I can now use these disk IDs in my files. So let's do that. I'm just going to copy all this output. And then we're using them to go to the local file and delete this. And then I just paste here. And I can basically remove all the lines that I don't need. So this is for the VMDK. So I'm just going to remove the SDA and the NVMe's. And then obviously I have to uncomment this. We can leave the commons if you want to, but they will be ignored by the OC command. So this is fine. Now let's head over to the local block YAML. And we do the very same thing. So this time, because we want the NVMe's, we're just going to remove SDA and SDB. Comment this fine. Let's have a last look at our files. So the local volume with the volume mode file system has the SDB disk. And they're mentioned by IV. And the local volume with the volume mode block has the device pass for the NVMe's. And for all three nodes. Very good. So let's apply this. So what this should do is we should now eventually see this. So now we see the local PV's slowly appearing. So the 10 gigabyte capacity PV's are used, are based on the VMDK's. And the 1.5 terabytes PV's are based on our local NVMe's. And as you can see that the 10 gigabytes PV's are using the local file storage glass. And the NVMe devices are using the local block storage glass, just as we want it. And we're going to use these storage classes now to deploy OCS on top of these devices. So this is going to look like, this is going to look like this. So we are going to apply an OCS storage cluster. And we're going to tell OCS to use the local file storage glass with the volume mode file system for the monitors. And we're going to tell OCS to use the local block storage glass with the volume mode block for the data parts. So the parts with the OSD's and we want three of them. Obviously these devices are not portable. So we tell OCS to not try to move parts between nodes. So let's first look at the parts in the open-share storage space. They're all running. This all looks fine. And now we apply this file. And what we're going to see is that this will spawn parts and create PVC's that match those NVMe's. Now that all parts are in state running, the OCS deployment is done. And let's look at this again outside of watch. We have quite a bunch of parts deployed now. And we have the OSD's and we see down here that the OSD's are now also using the NVMe's. So we can see that with the capacity. We can match that PV back to our local disk. And they're using the local block storage glass. And for the monitors, we're using the local file storage glass that gives us 10 gigs. The MDKs are directly attached to the VMs. And obviously OCS is deployed now and we do have access to the OCS storage glasses here. And also the Nuba OVC storage glass for OVCs. And just like always, we can now go back to the web UI and to the dashboard. And you see that everything still looks healthy. So we can go to the new OCS dashboard here. And it behaves just like an OCS deployed with the thin storage glass on VMware. And we have the very same look here. This is probably just unhealthy right now because it's rebuilding your data resiliency was just applied but will normalize in a bit. So that concludes the demo about local storage on VMware. Thanks for watching.