 So, I'll start. So, hi. I'm Noob. I also work for Western Digital Systems Software Group along with Atheesh and Afer. So, I'll be talking about these five hyperizers and as the topic suggests, it will be all about the hyperizer spec and the various hyperizers that we have quoted. So, broadly we have three topics to talk today and first is the spec and we'll spend more time in the spec and also even have slides to show Apple to Apple comparison with ARM and other particularly ARM architecture. And then we also look at the status of the KVM RISFI and XYZ RISFI project and based on time we'll try to cover the software part more. So, starting with the RISFI H extension or the hyperizer extension as we call it. So, RISFI is not just one ISA, it's actually a set of ISAs where each ISA is targeting a particular functionality and so, H extension is an ISA which targets the hyperizer functionality for RISFI and it's been in development since like more than two years now. The key contributors for this spec are Andrew Waterman, John Hauser and Paulo Bozini and so, there have been like five draft releases so far and recently there have been three draft releases because of RFS and then there'll be one more draft release that is 0.6 with a minor change left based on the feedback from the HCL L4 hyperizer folks but this change won't impact the other hyperizer that we have ported that is XYZ and KVM RISFI and as per the members meeting which was held in recent RISFI summit in December 2019 so this will be like the last draft release and spec will go in frozen state so we can expect the H extension spec to go in frozen state like in next couple of quarters and probably by year and it will be even ratified. So, Westerner has been doing a lot of co-development for the hyperizer spec or the H extension spec so we are using QMO as our RISFI implementation and we are using XYZ and KVM as the two different hyperizer to validate the spec and this is the timeline shown of the various draft releases and at what point we open source which batch is so over the time we have actually contributed a lot of feedback to the spec community and because of that feedback actually if you see recent seven months there were three draft releases because with every draft release we re-implemented everything and just gave a feedback and most of the feedbacks are around the feasibility of software implementation or actual implementation and then how performance is impacted and the lower small boxes, phones are small, it's not ideal but these are the different patches that were upstream at different points in time and we are expecting a 0 or 6 release soon and it will go in frozen state by that. So, what changes are there in RISFI ISA due to H extension? So, there are some mode changes, privilege mode changes but the overall idea is to make the H extension spec suitable for both type 1 as well as type 2 hyperizers and so we have a new HS mode for hyperizer which is nothing but an S mode with additional hyperizer capabilities so any traditional S mode OS will continue to run in HS mode unmodified you don't need to do any special changes to run in HS mode and then we have two additional modes for the guest that is VS mode and VU mode VS mode is like virtualized S mode for running the Linux inside a guest or VM and then VU mode is for the guest virtualized user space so in HS mode now we have three types of CSRs one is a traditional S mode CSR prefix by just S and then we have H prefix CSR which are for the hyperizer capabilities and then we have VS prefix CSR which allow HS mode to access the state of the guest or VM or the VS mode registers and in VS mode whenever Linux does an access to S mode CSR it will not go to the actual S mode CSR it will actually map to the VS mode CSRs so and this is the layered different privilege modes and the color coding is used for a different color for different privilege mode so hyperizer is like the red one and the VS and VU mode are green and blue ones so these are the list of all the CSRs that are added with HS mode it's pretty small compared to the other modern high-pressures so the left-hand side table is the hyperizer capabilities which HS mode has and in that we have highlighted few CSRs with the red those are the ones we actually were added based on our feedback and based on our efforts to implement in QMO and XYZR and KVM and then the right-side ones are the VS CSRs which allow HS mode to access the VS mode state and all S mode CSRs are not covered in this VS CSRs only the critical ones are covered and the ones which are not covered by this VS CSRs will trap and emulate in hyperizer and just like any modern architecture with virtualization support we also have a two-stage MMU the HS mode and U mode will have a single stage which is the HS mode page table so hyperizer also needs MMU for itself so it will have a HS mode page table which will do a translate hyperizer virtual address to host physical address and it will program by hyperizer itself and then we have two-stage MMU when we are running a guest or VM so the guest or VM will program the stage one or the VS mode page table and that will do a guest virtual address to guest physical address translation and then we have a stage two page table which is programmed by the hyperizer for each VM which will do the guest physical address to the host physical address translation and this is a pictorial to show how the translations work very similar to what is done with Intel EPD and ARM nested page tables and coming to MMIO traps and virtual interrupts so in this file H extension injecting a virtual interrupt to a guest or VM is very simple as just writing some bits in the HIP CSR and then we have three virtual interrupts as of now which you can inject through software from HS mode to VS mode one is for the virtual external interrupt another virtual timer interrupt and virtual interrupt processor interrupt and in general injecting virtual interrupt or virtual interrupt processor interrupt or virtual time programming are through all SBI calls so we have SBI calls defined for those things and guest MMIO emulation is done through the stage two traps just like other architectures it's very pretty standard thing and the two prominent devices that we are right now emulating in software is a PLIC so PLIC is right now totally software emulated in most of the hyperizers that we have ported for this file and we have virtual devices which are like the paraware devices which are again software emulator right now so coming to like since this is in a very good shape now it makes sense to compare the RISC file extension with a well-known modern architecture like ARM64 which is now quite mature and ARM64 is like the most recent architecture which added virtualization support so it will be easier to compare with that we can even do a similar comparison with other RISC architecture as well like MIPS or VHS but like because of our familiarity with ARM64 we chose to compare it with ARM64 so to start the comparison actually like I mentioned we have privileged mode changes in RISC file extension but those privileged mode changes are like adding additional capabilities to S mode we are not adding a totally isolated separate privileged mode for running hyperizers so in case of ARM when they release ARM V8.0 they added a dedicated EL2 exception level with their own separate CSR to run for hyperizers and then the guest or VM were meant to be run or they run in EL1 and EL0 but in case of RISC file they run in VS and VU mode so the approach that RISC file took is actually suitable for both type 1 and type 2 hyperizers but in case of ARM V8.0 the first release of ARM64 it was not that much suitable to type 2 it was suitable to type 1 but then there was a lot of context switching overhead for type 2 hyperizers because the world switch will lie in the EL2 mode and the rest of the host kernel will run in the EL1 so there was a lot of back and forth switching between whenever there was a VM exit particularly for type 2 hyperizers like KVM so they realized that fact and then in ARM V8.1 they improved it and they came up with VHE extension the virtualization host extension in ARM64 this is nothing but a very giant aliasing where you have a host kernel, a Linux kernel which is meant to be run in EL1 and they want to run this in EL2 but they want to fool the kernel so what they will do they will map all the EL1 accesses to EL2 MSR and similar aliasing is also there in case of H extension but this is more at the VS mode level so where all the S mode CSR access from the VS mode will map to the VS CSRs so pretty much this is already taken care in case of RISV so going forward to virtual interrupts so like I mentioned virtual interrupts are just just programming the HIPCR is sufficient to inject virtual interrupts we don't have a defined interrupt controller with virtualization support as of now so that's why interrupt controller is totally software emulator right now in RISV world and for save restore overhead of virtual interrupts we just need to save restore to CSRs HIP and HIE coming to ARM world actually most of the most of the cases in Ocaros hypothesis the virtual interrupt is injected by programming the LR register of Geek V2, V3 and this is good because there is no trapping as such the VM will directly access the CPU interface registers and the other part of the Geek will be trap and emulated but there is a context switch overhead so whenever there is a context switch all the complete hypervisor controller including the LR register will be saved and restored and moving towards the virtual inter processor interrupts they are through SBI calls in case of RISV 5 which will again trap and then in case of ARM 64 these are also trap and emulated because whenever guest or VM access is the ICC, ESG and EL these two registers they will trap and emulate in hypervisor they are not virtualized by the hardware and for virtual timer events in case of RISV 5 we have an SBI calls so since it's an SBI call it will again trap and be emulated by the hypervisor and in case of ARM they have virtualization support in the generic timers itself but this means that hypervisor will have to save restore the virtual timer state on every context switch and lastly nested virtualization so functionally nested virtualization is already supported and considered in the RISV 5H extension spec so everything which is required for hypervisor to implement the nested virtualization is there the only thing which is not there right now in RISV 5H extension is special acceleration to reduce the number of traps in while you are doing nested virtualization but in case of ARM the most recent ARM v8.4 has support for nested virtualization and also the enhancement to accelerate the nested virtualization the only downside with ARM was like nested virtualization was not defined from start itself they realized late and they added with ARM v8.3 but for RISV 5 it's already defined and well specced out actually and so what is the future work on the spec front actually so like I mentioned most of the things functionally required from the spec are there most of the things we can expect now will be on the performance front like optional acceleration for nested virtualization or optional acceleration for stage to dirty page tracking like other model like Intel and other the architectures have and some things will be there in the SBI spec as well we need a per hour time still accounting API then we also need an SBI extension for guess time scaling for migration for across post with different timer frequencies and there are two new specs also being drafted and they'll come up like over the years one is a new interrupt controller with virtualization support and MSI support and then there'll be a new spec with for IMEMU as well and if you have any suggestion please feel it's the community is open for ideas please write so now we'll talk more about the software status of the different hypothesis that reported and I'm sure I'm running out of time but I'll try to catch more so first talking about the KVM-RIS-5 is the type 2 hypervisor that we first ported for the RIS-5 world it's pretty the key aspects of the KVM-RIS-5 that we contributed was like we didn't actually change the KVM-ABI there was no new IOTS added and the world switch is really minimal possible because the CSRs are very less compared to other architectures and we have a VCP load put interface for lazy save restore of most of the full world switch and then we have KVM one range interface for user space to access the VCP state and then for guest or VM the hypervisor eliminate the SBI 0.1 interface and like I mentioned PLEAK emulation is done in user space totally in software we also have huge page support and all the undefined SBI calls will be forward to user space and this is the software layers involved and so where are the patches so like we contributed the patches the first version was like in July 2019 and all the patches were reviewed pretty fast by v6 everything was act and reviewed and by v10 so recently we contributed v10 so it was as re-basic everything to the recent draft release that was 0.5 release and and recent we also started sending out the KVM tool patches and so far we are waiting for more comments it's at RFC v2 we are holding the QMU KVM upstreaming right now we are working in background but we are upstreaming anything because we want the KVM RISC-5 to be merged in kernel first and then here are some of the important links so here is the developer repo or the maintenance repo and then we also created a mailing list because KVM RISC-5 is not just about the kernel support there is also user space pieces like LibWard, QMU KVM tool and other pieces so for that we have a common list where you can see all the patches and then we also have a wiki setup on github as well we right now have one page important page where you can using the steps over there you can just replicate whatever we have done so far and this is like brief tool is it will keep growing but we need to get 30-bit working at SBI0.2 support TracePoint supports, KVM unit test support, virtualized vector extension that's another thing which will be frozen soon so then QMU KVM support, then nested virtualized support, migration support and the last two also is like for completeness like 32-bit guest on 64-bit guest or big Indian guest on little Indian and those two things are already spec'd out they are already there in the spec moving to the other hypervisor that we ported I'm sure we have some time so is the type 1 hypervisor bare metal type 1 hypervisor open source it's much simpler compared to the KVM approach is just one monolithic layer running entirely in HSS mode whether it stands for extensible versatile hypervisor it's a GPLV2 bare metal hypervisor and like it's not new projects it's been there around like more than 9 years now and this is the website and mailing list and it supports all the family of ARM Intel 64-bit and RISPhi now so there was a recent release like start of JAN which has RISPhi H extension support with compliant to 0.5 draft it's able to boot multiple guest with SMP and also able to boot open embedded or federized guest OS and here are some of the documentation there is a documentation text file in the source itself you can follow that and there is a working progress QMO we have hosted which has the emulation support which is over here and please join the mailing list we are interested in this project and same the two rule lists are pretty much same as KVM RISPhi we need to get a 30-bit working add SBI0 to support and then there is one new item like we want to use the un-privileged load store it's a special thing there in RISPhi spike we want to use it with XYZR and to speed up the guest time access and loadable module support is not there it's there in ARM XYZR ARM 62 but it's not there in XYZR RISPhi so we want to add that other things are pretty much same as KVM RISPhi so questions if we have time okay okay the questions either I did very bad okay process for so your question is what is the process for the stuff we do to go into the western region so okay so different organizations will have their own process to take out or manufacture the what I have tried to talk over is the process that RISPhi foundation follows how it defines a spike and freezes it so the first slide yeah so if you look at this so this tries to show a high level skeletal process so any new spike in RISPhi starts with a trap goes through various revisions and then eventually it enters a frozen state and then eventually it is ratified so after frozen state there is no functional changes at all and then it's only the wordings and or ARAT or some or ARAT has to be redefined and then it enters a ratified state so the whole process of the defining a RISPhi extension in RISPhi is designed around that you take a lot of time to define the extension so that it is long lived in