 I work for the product security team at Red Hat and today I am going to talk about a little bit about how the KVM virtualization stack looks like. At Red Hat I will look after the security issues in the virtualization and the kernel projects. In the process I have looked into certain details of the virtualization stacks and I have tried to fix certain issues as well. Today I will share what I have learned over the course of time about how a KVM guest works in the stack. For the overview we will look at the x86 execution environment. Then we will look at the virtualization technology which is there which is used by the KVM module. We will look at the kernel virtual machine how it uses the virtualization technology to provide services to the guest. In the end we will look at the user space emulator which is the QMU which is used to create the guest environment for the users. If we look at the x86 execution environment we have a processor and in the processor we have a lot of subcomponents. There is an instruction execution module, there are certain control registers, there is memory, there is an IO unit which works to machine to the disk. On top of all this hardware layer there is an operating system. This operating system uses the processor instructions to execute the user programs. Let's take an example of a simple program. When you say you want to print something on the console there are many things involved. This program will be converted into a processor instructions like this and then there is a console which is to read and write to that console you need a hardware driver which is used by the operating system. So just to print a certain message on the console a user program writes a statement like say printer which is converted into an operating system call which is say write and that write will communicate with the hardware driver which will eventually put that information on the console. So this is just a basic execution environment on the x86 system. Now let's imagine instead of this simple program we want to virtualize or create an environment where you execute an operating system as it is in a virtual environment. As we saw operating system uses the hardware which is like a processor which has registers, which has memory, it has IO unit. So if we want to execute an operating system in a virtual environment we will have to create the same environment which is provided by the hardware. So that's where the virtualization technology comes into picture. So Intel introduced a virtualization extensions on AMD they call it secure virtual machines on ARM processor mode. So what basically these technologies do is they introduce more instructions in the processor which is used by the kernel or the virtual machine monitor to provide services to the guest environment. So these instructions are called the virtualization extensions and there are instructions like VMX on, VMX off and VM read write. And these are used by the KVM module kernel virtual machine module to create a virtual environment which is used by the logical processor which is in in turn used by the KVM guest. So there are two modes of operation like we saw on the host environment in the operating system. You have a user space instructions or user space functions and in the kernel you have kernel space functions. Kernel space functions work in the supervisor mode or the root mode whereas user space functions are at user level privileges. Similarly in the virtualization instructions these set of instructions are divided into two modes. One is a VMX non-root operation. So when guest wants to execute certain instructions they are executed at the non-root operation mode. And then there are VMX root operations which are used by the KVM module or the virtual machine monitor module or the hypervisor which is like privileged instructions. They are used by KVM module to give services to the thousands of guests that it is running. And this separation is there because at any point you don't want a guest environment or a guest operating system to use or access hardware directly. We want to kind of multiplex the hardware across different guests. So that's why there are two modes of operation. Now let's say there are certain instructions provided by the server or provided by the processor or the hardware. But still to create the logical environment or the virtual environment we need to store data which is used by the logical processor. As we saw in the host system or the bare metal system it has a lot of control registers. It has all that information when it's executing instructions it stores and reads information from this control structures or control registers. So similar control structure or similar environment is created in the memory which is called a control structure or virtual machine control structures. It's like a 4 kilobyte memory area allocated by the KVM module. It stores a lot of control information which is used to execute a guest environment. So it involves when does while executing a guest operating system at which instruction does a guest control move from guest non-route operation mode to the route operation mode. Like in the bare metal host environment when a program calls a system call it goes from user space to kernel space. Similarly in the virtualized guest environment at which point does a guest exits from the VM and enters the route operation mode. So that is called the when the control moves from guest operating system to the VM which is KVM module. Virtual machine monitor module that is called VMXZ. And similarly when the KVM module performs an operation on behalf of the guest and then returns control back to the guest environment or the guest operating system that is called VM entry. And to do this VMXZ and VM entry it stores lot of control information in the control fields of this control structure. It has like a 32 bit vector or 32 bit register which stores information about at which point the guest environment or guest operating system exits to the virtual machine monitor and why does it exit. For example it can exit when there is an interrupt or it can exit when it is trying to access it when there is a memory fault or some sort. So at that point control shifts to the KVM module which is a virtual machine monitor and KVM module will look at the exit reason why did the guest exit from the virtualized mode. And then perform that operation and shift control back to the guest operating system. So now you have in the hardware you have lot of these instructions available. Now we want to use that hardware infrastructure and provide services and that's where the loadable kernel module comes into picture. So this KVM module converts the normal Linux kernel into a hypervisor which creates a hypervisor mode or guest mode or it introduces a guest mode which is used by the guest operating system. It creates a device like the slash type KVM which is used by the user space program which is QMU to create this virtual environment for the guest operating system. It introduces the or it offers the services to the QMU emulator where kernel are called. It allows you to create a virtual CPU, it allows you to create virtual interrupt chips and it allows you to kind of synchronize the operation between the guest operating system and the KVM. What is QMU? Now in the kernel module or in the kernel side you have the kernel module which converts the kernel into a hypervisor but now you want to create multiple guests. So how do you create that? That's where the QMU emulator comes into picture which is at the hardware or in the operating system side you have a virtual processor hardware or processor infrastructure. But you still need a PC hardware to run an operating system. You still need a console, you still need a keyboard, you still need a mouse or you still need many other hardware devices which are available in the normal system and that infrastructure is provided by the QMU emulator. So it allocates memory, bunch of memory in the system memory in the RAM and it creates a guest environment in that allocated memory. And then on top of that it calls the services offered by the KVM module or the kernel virtual machine module to interact with the processor or use the processor to execute the guest environment. So how does it all look like? So you have in the bare metal system you have the hardware or the processor which gives you the control registers and hardware infrastructure. On top of that you have a kernel and KVM which uses that hardware infrastructure to create a logical processor or a virtual guest environment. And that virtual guest environment is in turn used by the QMU to actually create the guest operating system and in that guest environment it eventually executes the guest operating system. So for a quick anatomy of the KVM guest this is how it functions or looks like, that's about it. So are there any questions? I guess it was like quite quick introduction I guess. Is there a lot of differences between the Intel and AMD and ARM? Is there a lot of difference there in terms of performance or capability? Well not much of a difference. At the very superficial level all these processors have multiple instructions which work at a different kernel mode and user space mode like different privileges. And they sort of provide the similar services to create the guest environment. But of course there are different features as in in terms of KVM it uses the basic Linux kernel infrastructure to provide the services. So if you look at the KVM code it's very small as in the footprint of the KVM code is very small because it uses a lot of kernel features like it doesn't do scheduling it doesn't do memory management all those things. But for that it depends on the Linux kernel and because it's part of the kernel you can use or manipulate the guest environment using a normal system commands, which is which is different in say Zen or VMware or other virtualization technologies. Yes? How does this convey the obvious impact of the ARM 64? Well I haven't used the ARM 64. I don't have the first time knowledge about ARM processors. ARM 64 is a bit different from, well it's actually very different from... Yeah, X76. The way virtualization is implemented on ARM 64 is connected to a new level or an execution level called hypervisor modes. And there you can run your operating system through the hypervisor modes and so it's a bit different. The way page tables work is a little bit similar. It's difficult to say it's better or not but better in terms of it has a very small memory footprint. So it's very flexible in that way and it's available by default in the Linux kernel whereas Zen works very different. Zen has a guest zero mode which replaces the operating system in this stack which works on the... So it doesn't have an operating system layer but Zen has something called guest zero which works as the operating system here which directly interacts with the hardware and then guest one and onwards they are called the virtual guest. They use the guest zero mode services to work as the operating system, work as the guest environment. So guest zero is the first guest which creates the virtual environment for the rest of the guests. I guess the Zen path is not truly tooled by a machine here. No, no, no. It's not merged in the Linux kernel as such. It's a very different architecture that way. Another advantage for KBM is there is a lot of active community which is very actively developing and fixing bugs and probably shooting and debugging things in KBM. So that way it's very well-maintained and well-developed as well. Sorry, what's that? The host has only two cores. Then I make a guest that uses two cores. Then I make another one more guest that also uses two cores. Yes, those will be the virtual cores. Yes, yes. No, those will be the virtual processors as well. Yeah, so that should be possible, yes. Yeah, they might but that should be possible, yeah. So you're actually running slightly a 10-minute schedule. Yes. I would say let's take care. Well, yeah, so next talk is scheduled to start at 14.30. Yes, we have 10 minutes. 10 minutes to go. Thank you so much for coming.