 Hello, my name is John Hurr and I'm a Senior Software Engineer at Red Hat. In this video, I will be demonstrating the PCI pass-through functionality of OpenShift virtualization. I will do this by passing a physical NVIDIA GPU in one of my worker nodes of my OpenShift cluster to a virtual machine running inside the cluster. Once this is done, the virtual machine will have direct access to the underlying hardware. After the GPU is presented to the virtual machine, we will test that the GPU is being seen at the hardware level and that is working as expected. Please note that portions of this video will be edited for time. These edits are done to shorten commands and processes that take a long time to run. I will include links to documentation, software, and other useful URLs at the end of this demonstration. As you can see, the OpenShift virtualization operator has already been installed in the cluster and a hyper-converged operator has been created. I have already created a Red Hat Enterprise Linux 8 virtual machine. We will pass the compute nodes GPU device to this virtual machine. The PCI vendor and device IDs are needed to configure the GPU adapter for PCI pass-through. We will determine the vendor and device IDs by viewing the PCI information on the compute node. We can see this device is an NVIDIA A100 adapter and has a PCI vendor ID of 10DE and a device ID of 20F1. Make note of these IDs as they will be needed later. The device is using the Navo driver. We need to have the device bind to the VFIO PCI driver as this is better suited for PCI pass-through. The configuration of the compute nodes is done using a machine config. We will use the butane utility to generate a machine config YAML file we can apply to the cluster. This machine config will enable IOMMU on the compute nodes and also bind the GPU device to the VFIO PCI driver. Here is the butane configuration file. Variant tells butane to generate a file suitable to be consumed by OpenShift. Name will be the name of the machine config as seen in OpenShift and the OpenShift kernel argument section enables IOMMU on boot. The storage section creates two files. The first file specifies the options used when the VFIO PCI driver module is loaded. The options line specifies the PCI vendor device IDs to bind to. This is the vendor device ID of the GPU device as noted earlier. The second file tells the system to load the VFIO PCI module upon reboot. When we run the butane utility, a YAML file is created. This machine config YAML file can be applied to the cluster. When we apply the machine config YAML file to the cluster, the compute nodes will reboot. After the compute nodes reboot, we can verify IOMMU is enabled on the nodes by logging into them and viewing the kernel boot parameters. We can see that IOMMU is enabled on the compute node. When viewing the PCI information again, we can see that the GPU adapter is now using the VFIO PCI driver. Now that the compute nodes are configured, we need to configure the hyperconverged deployment so that it will pass the GPU device to the virtual machines. Let's edit the hyperconverged deployment YAML file to add a section that tells it to pass the GPU device to the VMs. We will set the PCI device selector to the PCI vendor device ID of the GPU device. We also need to specify a resource name. This is an arbitrary name, but it's useful to make it something meaningful. Checking the virtual machine, we can see from the output of LSPCI that it does not currently see the GPU device. Let's stop the virtual machine so we can more easily edit the YAML configuration. We will edit the virtual machine YAML configuration to add a section that specifies the GPU as a pass-through device. We need to add a host devices section to the YAML file. We will specify a device name as the arbitrary resource name we defined in the hyperconverged YAML file. We will again use an arbitrary string for the name entry. This is not used elsewhere. We will save the configuration and start the virtual machine. Let's check the virtual machine again. We can see that the virtual machine now has a GPU device. This is the physical device on the compute node. However, the virtual machine is using the Nouveau driver, and this driver will not allow the virtual machine to utilize all the features of the GPU device. To utilize all the features, we must use the NVIDIA driver. We will install the CUDA toolkit from NVIDIA as it contains the drivers we need and some utilities and tests we can use. The toolkit needs the GCC compiler, kernel headers, and a few other packages. The CUDA documentation also mentions the need for some other repositories to be enabled, and also some packages from the Apple repositories. Now we can install the toolkit. I already have the URL for the toolkit's packages because I visited NVIDIA's website earlier and downloaded the packages for my operating system and architecture. When installing in your environment, you should also visit NVIDIA's website since it provides instructions for specific versions of CUDA on various operating systems and architectures. The instructions provided for my installation are three steps. First, clean the package manager's cache. Second, install the latest DKMS version of the NVIDIA driver. Finally, install the CUDA toolkit. After the toolkit is installed, the directory search path and library path must be updated to include the directory containing the toolkit's executables and libraries. We will set these in the system profile directory for bash, so they will be set upon login. We want to make sure that the device does not use the devote driver, so we will create a file that will prevent the driver from loading. Rebuilding the initial RAM disk will prevent the driver from being loaded upon boot. Let's restart the virtual machine so the changes will take effect. After we log back into the virtual machine, we can see that it is now using the NVIDIA driver. We can also run the NVIDIA SMI command to make sure the GPU can be seen by the toolkit. Now it is time to see how the GPU compares to the CPU when running a heavy workload. We are going to run an in-body simulation that is included with the toolkit. This type of simulation calculates the gravitational pull between bodies or planets. We will first run the simulation on the CPU, then again on the GPU, and compare the results. The dash benchmark option displays the calculation times without actually rendering anything on the display. The dash numBodies option specifies how many bodies to run the calculation against, and the dash CPU option tells the programmer to run the calculations on the CPU and not the GPU. The grep at the end of the command just cleans up the output a little to make it easier to compare. Running the calculation on the CPU took 6,250 milliseconds, or just over 6 seconds. During this time, it ran 0.168 billion interactions per second, or about 168 million interactions. Now let's run the same command on the GPU. All we need to do is remove the dash CPU option. Wow! Running on the GPU is so much faster. It only took 10 milliseconds to run the same calculation, and it was able to do over 98 billion interactions per second. This makes the GPU about 600 times faster than the CPU for just this one calculation. As you can see, passing a PCI device through to a virtual machine can easily be done. During this demonstration, we passed the GPU adapter through to a virtual machine and utilized it directly to take advantage of its performance. Thank you for viewing this demonstration, and I hope it helps you in your future endeavors with OpenShift Virtualization.