 Hello everyone. Thanks for joining us. My name is Ophir. I work for Google Cloud. Alex from Oracle and I will present address space isolation for HVM. This idea started more than a year ago, was presented also in LPC by several different companies. They were trying to merge work by IBM, Oracle, Google and potentially others into one unified framework for other space isolation, more on that slightly later on. And other space isolation is a defense mechanism against speculative micro-architectural attack. And we discuss more of that in a second. So why do we need other space isolation? So about a couple of years ago, we've seen attacks like L1TF and MVS that are able to get information for micro-architectural buffers. L1TF can get data from the L1 cache. And if you look at this drawing, you see, for example, two VMs running, one of them the attacker VM and the other is the victim VM. And the attacker VM is capable of basically inscribing the L1 cache. And it doesn't even matter who this data belongs to, whether it belongs to the victim VM or maybe to the host, the attacker is capable of getting this data. If we look at the MVS attacks, the attacker is able to get data for micro-architectural buffers. For example, the ride buffer. And this is bad. And the reason this is bad is because obviously we are breaking the VM boundary. One VM can steal data belonging to another VM. And also a victim VM can also steal data that belongs to the host. And potentially the host has credentials to communicate with the rest of the data center. And while it seems initially that those attacks can only work in very specific scenarios, those attacks are actually very powerful because it seems to be more practical than one would think. So the danger is that one VM can steal data from another VM or steal it from the platform. And this is obviously very bad for the cloud environment. And we want to come up with an efficient mitigation that allow us to prevent these attacks in a reasonable performance over there. So let's first see what are the obvious mitigations that one can deploy in order to prevent these attacks. So one obvious mitigation is to disable hyper-threading. Basically saying, you know what, hyper-threading is a bad idea. But that comes with a pretty reliable solution, but it's quite expensive. In theory, we can lose up to 30% if we disable hyper-threading. And that's not what we want. Another option is to basically make sure no two different VMs are running on the same physical core. Again, it creates slightly more complicated scheduling, it's possible. But we still have the problem, what happens when we need to save a VM exit? Then the host needs to run some code, maybe bring some secret data into the cache or into some other micro-architecture buffers. And basically what you need to do then is maybe find a way to stop the other thread. Let's say you have one sibling logical core serving a VM exit. Then before you serve the VM exit, you want to somehow pause the other sibling thread. That's also quite expensive, obviously. And it's a quite complicated mechanism. And in other space isolation, what we want to do is basically deploy defense mechanisms only when we actually meet, meaning only when we actually access what can be sensitive data. Only then we will deploy mitigations. So let's try to give some intuition about ASI. So in the next slide, if you can, yeah. The intuition to an ASI is basically we will try for the most time, for most of the time to not have most memory mapped in the page. And here's a trivial example of Spectre version one. It's a function that will access an array, its location index. And if you look at the if in the beginning, we will check if index is within the bounds of the array. And then we'll access some other array with this index. And what will happen is that if the index is too big and the branch will be predicted accidentally as taken, we will access array location potentially million. And in theory, we can access an arbitrary location memory. And then this memory will be brought into the L1 cache or into some other micro-architecture buffers. And the idea is that if when a program runs, only data that is not sensitive will be mapped in the page table, then any access, whether it was speculative or not speculative, it will cause a page fault. And basically the processor cannot speculate behind a page fault. The processor doesn't know the translation between virtual and physical. It cannot bring data into the micro-architecture buffers. And if you cannot bring the data, then it means that secrets are not in micro-architecture buffers. So that's the key intuition behind ASI. Basically, let's not map data that we don't need to into the page table. So as an overview, the idea is let's say we have, just in the case of VMs, we have two guests running. And most of the time when we are serving a VM exit, whether it was an interrupt or something else, we touch memory that has to do with the current guest that is running. And the idea in ASI is let's have a page table, a limited page table. You can think about it as ASI domains. Guest A will have in this example ASI domain 1 and guest B is ASI domain 2. And when guest A is running, to serve most VM exits, we're only touching data that has to do with guest A. And the idea is that if guest A is trying to steal data that belongs to itself, then we're saying, we're slightly relaxing the security requirements, basically saying if a guest VM steals data that has to do with managing itself, then we're okay. It can try and steal it because there will be no particular secrets that can be exposed. And what happens if we need to serve a VM exit and then touch some other data, then we want to identify it and say, okay, now there's a chance we're actually touching secrets in memory. And we won't be more careful. Maybe now we want to synchronize with the other sibling core and pause it. That's sometimes known as stunning. And then maybe when we touch privileged memory before we get back into the ASI domain, then maybe we want to scrub secrets from buffets. For example, to prevent L1DF, after we touch privileged memory and we want to go back into the ASI domain, we will only then flush the L1 cache. And that's nice because it means that hopefully for most VM exits we're only touching guest A stuff and only every once in a while we touch some other data and only then we flush L1 cache, not on every VM exit. And if we want to defend against MBS attacks, it means that maybe we run the VERW scrub instruction only if we actually touched potentially sensitive data. And with ASI, we will know that it potentially touched sensitive data because when we touch it, we'll get a page fault. And that's something very tangible that we can notice and invent that we can manage. And then there's also a question on how we end the interrupts, of course, but more on that slightly later on. And this idea now we explain is how to manage address space isolation for KBM, the other options as well. And Alex now will explain what other usages we can have for other space isolation. Yeah, thank you. So although ASI was initially designed for an application with KBM to mitigate the some speculative attack, we expect that we will have over usage for ASI and that we can make ASI more generic and not specific to KBM. So for KBM, as explained, obviously, that's to protect against this guest-to-host attack. Main challenge in VERW is what data do we want to include in the ASI, what is really sensitive and secret. What we expect ASI to be useful to is to do what we call a user ASI, which is to reimplement the kernel and user page table, the page table itself and the switch with ASI in order to refactor the kernel page table isolations of the KPTI, which is a mitigation for the specter of V1. And KPTI work almost the same way as ASI, that is that you have one page table for the kernel, one for user-land, and when you enter an exit user-land kernel, you will switch the page table. So this is an area where we can use ASI to have the same implementation. Over possibility would be to have some user-land ASI. This is some idea, we haven't looked into too much detail yet, but the idea would be to provide the ability to use a process to have multiple address space. So currently we use a process as a single address space, that's a single page table, but we can imagine a process having multiple page table and being able to switch between the different page table. We think these may have some useful usage for virtual environment, like the Java virtual machine or the Grow Virtual Machine or containers, where you need to, a single processor is running multiple environments and you want to be secure between these different environments. So this is really something we are only investigating at the moment. Now we are going to go into more specifics about the KVM ASI implementation and how this is used exactly. So the idea is to have an address space with a limited kernel and VM mappings. So basically we need the mapping to enter the VM and then under the VM exit. The goal is really to be able to run the VM and under most VM exits, at least the most we can, without exiting ASI, so that we can loop and stay running safely with ASI. So the idea is that on an ASI we are going only to map the data for a single VM. And that way we can prevent over VM running on the CPU core to steal data from another VM or from the host. And to do that also safely we would need to synchronize on the VM entry if the other sibling is not running. There's no need to synchronize if your sibling CPU thread is also running ASI because we know it itself, we know there is no data to secret data that you can get from there. But if anything else is running in kernel context, not in ASI, then potentially you can steal something. So this synchronization is really needed. And in that area core scheduling is going to help because with core scheduling you are going to be able to schedule on the same CPU core process from the same VM. So you would know that at least that's the same VM and you just have the risk of being in kernel context where you can steal some data. So as I mentioned ASI is more generic, it's not specific to KVM. And the generic life cycle for ASI is pretty simple, basically you create an ASI, each ASI is going to have its own page table. Then you would need to populate the ASI page table with data you know which is not sensitive so anyone can access it, it's not a problem. And then all you have to do is to enter ASI, so when you enter ASI that's an explicit command and it will switch from the kernel page table to the ASI page table. Once we're running with the ASI page table we still need to be able to handle the exception before the context switch. And sometime to do that we need to interrupt the ASI and maybe a resume depending on the context. Once we are all done with that and we don't need ASI anymore, we exit the ASI switch back to the kernel page table. Once we are completely done with ASI we can destroy the ASI on the page table. So that's a fairly simple use basically like yeah page table, use that page table when you are done you exit. If we look at the usage more specific to KVM, the idea is that we will create one ASI per VM or per vCPU. So it depends on the implementation. I don't have to test both, you have prawn cones for both of them. If you create one per VM then you will use the same ASI for all vCPU otherwise you can also have one per vCPU it depends on the implementation you want to do exactly. Then obviously you populate your KVM ASI page table with things which are generic as kind of information you need just to enter and run your ASI and also things which are specific to your VM or your vCPU. And after that the KVM ASI will be used when you are running the VM itself so when the guests are running the vCPU guest. So it will all start with the KVM run IO tour and you enter in KVM the vCPU run loop and during that loop will enter the KVM ASI. And right before entering the VM we need to ensure that the over zippling CPU they are also running with the KVM ASI. This is where we want to ensure that what we are going the VM we are going to run is secure and it's not going to steal our data. So this is where we'll have some synchronization to do. Once we know that the zippling CPU are also running the KVM ASI we can do the VM enter run the VM. Once we are done we exit the VM and once we have exit the VM we can stop this enforcement of having a KVM ASI running on the zippling. So we are back from the VM we are still running with ASI and now we want to process the VM exits. So we'll try to run most of the VM exits with KVM ASI that's really the goal. If we need at some point when running the handler to exit to work and then we'll do that. But because this is a loop we are going to re-enter and we are going before doing the VM enter we are going to re-enter the KVM ASI to be safe on the next run. So first thing as I mentioned is to when you have an ASI is to fill the page table of the ASI. So there is a very basic solution which is just to add each mapping that you need. We have command like ASI map where you specify ASI the address you want to map and the size of the buffer you want to map. And I'm a command when you are done. So this is similar to the memory alloc and free. So instead your map and your unmapped your page table and that works fine. If you have well-known buffer you know when they are located especially if they are pre-allocated they are located at the beginning of the boot or when your module is loaded it's easy to map them before entering your ASI. It's becoming more complicated when you have buffer which are dynamically allocated and frequently relocated especially this is happening while you are running with ASI. So we need some mechanism which are more performance especially for trying to add some mapping more automatically. So one first idea is to track and tag the statically allocated buffer that we need and we know have no sensitive data. So a simple solution is just to put this data into a dedicated section. It can be fairly simply done by just setting a compiler section, a compiler attribute with the section where you want to store this information. So a simple solution is just to define a special tag ASI not sensitive which is an attribute, compiler attributes which this fight but these data is due in the dedicated section. And once we have that when we create an ASI we can just map the entire section into the ASI. We know these data are safe but we need them to run our stuff. For the dynamically allocated buffer what we would need to do is to flag this buffer. So to do that we can use a new tag to specify that the buffer we are locating is going to store non sensitive data. So currently there is a global non sensitive and a local non sensitive. A global non sensitive would mean that the data are not sensitive whatever ASI you are using. So you can map them into any ASI. The local non sensitive is different in that it's sensitive data, but it's not sensitive only for this process for the current process. A good example for KVM is the VMCS. So the VMCS has information about the VM, but it's not sensitive. If you are using and running the VM it's sensitive regarding a different VM but not regarding the current process and the current VM. So when using this tag the buffer are located and they are automatically mapped into the ASI, the current ASI. And they will be also automatically unmapped when you free the buffer. Once you have your page table defined for ASI we need a mechanism to switch the page table when you enter the ASI and when you exit the ASI. So that's very simple, especially on x86 it's just a control register to update, but we see a free control register. So that's basically one assembly instruction, very simple. The complication is that it's not very efficient if you don't consider the TLB. So the TLB is the translation look side buffer. This is a cache of the virtual address to physical address translation. And this cache having avoid having to go avoid having the MME to go through the entire page table to do the translation because the translation is already cached. So when you switch you are required to flush the TLB and this has an impact on performance. If you don't know any special processing. So now the Intel processor they have some optimization which is the user process user context identifier the PCID. PCID that's a facility to associate to TLB entry with page table. And the benefit is that it avoid flashing the entire page table TLB you will just flush the TLB entry for your page table. And so in the implementation this is something we're taking into account so that we have an efficient page table switching mechanism when we enter and exit the ASI. So now we have created the page table. We have a mechanism to switch the page table. And now when we are running ASI we need to be able to handle the interrupt and the exception. Because yeah, you still need to process them. The problem is that you have ASI you have defined so your page table you have defined for your ASI may not necessarily have all the mapping to run the interrupt and exception handler. The simple solution for that is to suspend the ASI when there is an interrupt process the interrupt and then reenter the ASI. So that's why this drawing is showing so you start with some code running with ASI there's an interrupt you switch you exit the ASI switch to the counter page table process the interrupt handler. When you are done, you switch back to the ASI and your code is going to continue. So the simple implementation. We expect we can do better for some interrupt and exception handler because this mechanism require an additional page table switch. So ideally, we would like to be able to run the interrupt and exception handler with the ASI page table. Of course, this may not be always possible for Portugal and draft whites. It's better just to switch switch off to the to the channel page table. But still we can we can put in place a mechanism to do that. So the idea is that the handler when there is an interrupt the handler is going to start, but it's not going to switch the page to the ASI. So it will continue to run with the ASI page table. So basically, it can fail at some point because there is not the require mapping to do to do the processing. In that case, it's going to take a page fault. And the solution there is to handle the fault and to do and switch at that point to the channel page table. To continue the processes, the processing of the interrupt handler. So here, the figure is showing the ideal situation. We've got code running with ASI. We take an interrupt. We don't switch the page table and we continue and are able to process the entire the entire interrupt with ASI. When we are done with the interrupt return and we're using the execution of the code still with ASI. Most of the time, what is going to happen is this situation where we've got some code. We're adding we've got an interrupt. We start processing the the interrupt with ASI, but at some point because we don't have all the mapping we are going to take a page fault in the ASI. So we go to the page fault handler at this point we're switching to the channel page table. And we retry and continue the interrupt handler with the kernel page table. Once we are done, so we have processed the entire interrupt part of it with ASI part of it with the kernel page table. But once we are done, we return to the ASI and the code continues its execution. So it may be a way to to investigate which which handler we can run with ASI and found which mapping we need to do the processing. But that's basically an optimization we're trying to to be more efficient when there is an interrupt or exception while we are running an ASI. Now there is a particular case, as you have seen also with the page fault. So if you have an ASI and you don't have the mapping you're running with ASI there's mapping which is not that you need which is not present in the ASI. Then a basic behavior is to exit the ASI switch. So there's page fault page fault handler is going to exit the ASI. So switch to the kernel page table and it's going to retry the instruction level for Tokyo. So basically we are exiting the ASI on page fault. What we do in that case is that we log some information about the fault so that we can identify after after what is missing in the ASI and what has caused the fault. And eventually after that we can add the missing mapping to the ASI page table to prevent the fault. So this is manual processing we have to look at below figure out what's missing and then add the mapping. We can certainly do better. And there is some automatic way to do this by automatically adding the missing mapping to the page table. The only issue if we add the mapping automatically is that we need to ensure that it does not provide access to a sensitive data. So if we have that we can add the mapping automatically and return to the ASI so retry the faulty instruction and hopefully it will work and we'll continue and it will also avoid faulty again in the later on the same problem. And to do that, we're also making usage of the global non sensitive flag I mentioned earlier about dynamic allocations. If you have done it, if an allocation was done with that flag, you know that this data is not sensitive. And so you can automatically map it into the ASI if there has been a fault on it. So that's a way also to improve the mapping in an automatic way. What can happen also when you are using ASI is that you are being scheduled out. And in that case, the behavior would be fairly simple is that we interrupt the ASI so it's going to exit the ASI, save the ASI information for the task. And once the task is scheduling, then we are going to resume the ASI that is enter the ASI again and with the information which were set for this task. There's a bit of complexity if this secure during an enter up an exception handler because the ASI may have already been exited or interrupted. So this is just thing we need to take into account into the implementation. A more complex mechanism that we are also using is the synchronization across CPU thread. As I mentioned for KVM ASI, when you enter KVM, we need to make sure when you enter VM, you need to make sure that the CPU is also running the KVM ASI. So we need a mechanism to force all CPU thread from a CPU core to use a specified ASI. And to do that, we need the CPU to look at the other sibling CPU if they are running ASI or not. If they are running ASI, then that's fine. We can just continue to run. The only thing is that they should not be allowed to to exit the ASI because we want them to run ASI. If the sibling frame is not running ASI, then we need to force it to run ASI. And to do that, we send a rescue request. And if the next stack is using ASI, then we can let it enter ASI and run. Otherwise, we need to enter ASI but wait in an idle loop. So basically the mechanism is like this. We've got the CPU in VM, which is running KVM. Suppose we have two sibling CPUs. So first the CPU running KVM. At some point before entering the VM, it's going to start this synchronization process and it's going to have a look. So pick for ASI if it is running on the other CPU. In that case, we can see that the T-over CPU, they are running ASI. So this is fine. There's nothing to do. And we can stop the VM. We know the other CPU are running ASI. If one sibling CPU, for example, CPU to try to exit ASI, then we are going to block it. It's going to be blocked in ASI in an idle loop until the main CPU exit the VM and stop the synchronization. At this point, it's free to go into the kernel. The other CPU if it is running ASI will just continue to run. If the other CPU is not running ASI, there's a bit more complication. Because in that case, we still have the same beginning. We stop the synchronization. We look for ASI and the other CPU. In that case, other CPU, they are not running ASI. So we'll send a reschedule request to this CPU. So they will select a new task. For example, CPU one is going to select a task which is using ASI. So that's fine. It can enter ASI and continue to run. Our CPU is picking a task which is not running ASI. So in that case, we cannot run that task immediately. Instead, we switch to ASI and wait. So at this point, when both the sibling CPU have entered ASI, we can run the VM safely. Once we are done with running the VM, we stop the synchronization. So for the CPU already running ASI, there's no impact. But for the CPU which was in the idle loop, we stop this loop and we can run the task which was not using ASI. So that's a bit of complexity to do all this synchronization. And this is something we're still investigating and trying to optimize. Another thing with this CPU synchronization is how do we handle the interrupt and exception, especially if there is an interrupt exception and you need to exit ASI. And in that case, we have to force all sibling CPU to interrupt so that everyone is going to stop and do the processing and interrupt. And so there is a synchronization needed to when we do the processing of interrupt and we are synchronizing the ASI. We are also going to look at what Korska doing is doing because they are the similar mechanism in place when they got an interrupt on the call which is running a tag process. And now, yeah, we'll have a bit more detail about the page table and we'll continue about that. So now we'll give a brief discussion on how page tables look like. There are probably several ways on how to manage page tables, but this is one way that we're doing it right now. And just like as a general background to kind of like ease into like why we design the page table as they are. And in the next slide, we have this is a rough description of what look how it looks like for KPTI, right? When the user space, only user space memory is mapped other than tiny bits of the company memory. When we switch into the kernel, we're switching the page table into a page table that also has everything that the kernel has, which also belongs to all the processes in the system, global variables, obviously also kind of text. And in KPTI, basically the idea is that privilege is based on whether you are in user land or in kernel space. And in ASI, this is slightly more nuanced. So instead of like basically saying you can have space or not, basically what we're saying is, well, we're defining privilege based on the data, not necessarily on the executioner. So we had, for example, on the right hand side, user space page table. And now we have in a simplified, at least scenario, two extra page tables, the unrestricted page table, which is basically what we used to think of the kernel page table, which has everything. And then we have the restricted page table, which is the ASI page table. The restricted page table only has mappings of non-sensitive data. And the question is, how do you define non-sensitive versus sensitive? In general, at least now we are hoping to use kind of like an allow list approach of saying, okay, everything is a secret except for what we think to be a non-secret or non-sensitive. So this can be global non-sensitive data or local non-sensitive data, which we'll talk about in a second. But basically we're saying non-sensitive data is data that we are okay to, if it's stolen, if a malicious virtual machine tries to perform L1TF attack or LDS attack to steal the data, stealing this data will not cause a threat to other virtual machines or to the cloud infrastructure. And for performance reasons, we clearly care about data that is frequently accessed in order to set my VMX. Because if data is being frequently accessed, it means that if we will not put it in, consider it as non-sensitive, it means we will incur many ASI exits and any ASI exit obviously has a performance cost. And now we can kind of like discuss a bit more in detail about that. So as we said, non-sensitive data can be accessed by L1TF or other information. Yes, thank you, Alex. And we can discuss what does it mean to be local versus global data. So local data, as Alex mentioned before, it can be the VMCS, maybe the five script table, anything that belongs to the specific VM. And again, we say if the VM tries to mount an attack to steal it, maybe it's not ideal, but it doesn't necessarily cause a significant threat to other VMs or the infrastructure. And global data is data that can be basically accessed by all virtual machines in the specific machine. And if it's stolen, there is no significant threat. And the nice thing about the global data is that we can share the PGD entries in the page table across all the different ASI page tables. And this gives us a really nice way to manage memory, because if you think about local data and if you need to update the ASI map, we probably need to update it across all the course that run the specific ASI map. And this is specifically a problem if you are deciding to unmap a piece of data from the ASI map. We'll probably need to flush the TLB for all cores who run the same ASI map. And with the global data, if we share PGDs, it means that we can edit the global ASI page table. And we don't need to update even if we have hundreds or thousands of ASI page tables. We don't need to update them. If they will have a page fault, we will basically look to see if the page fault was in the area of the global sensitive data and if it was, we will just add the PGD entry. And that's it. And I think now we can go to the next slide and talk about early results that we have. So we have various implementations we're hoping to kind of like coalesce into one unified one, but this one is one we ran on LSPI, YCSB benchmark. And the first thing we can look at is the ratio of ASI exits versus VM exits. Obviously, ideally we want this ratio to be as low as possible. Let's say we had 10,000 VM exits. Ideally, we want only once or twice to touch something that forced us to do an ASI exit. We're not there yet, but what you can see here in this data on the specific benchmark configuration we ran, we had two VMs and every VM had eight vCPUs. And if you look at what's circled in the red rectangle, you can see the ratio between ASI exits versus VM exits. It's between 20 and 50%, which at least shows that we can implement ASI in a way that we don't have to do an ASI exit for every VM exit. And we believe that basically by adding more and more data into the ASI page table, we can hopefully reduce it to a very low number. And that can be in the lows 2%, 2%, 5%. So the performance impact of ASI will not be too significant. So if we go to the next slide, we can start seeing how can we analyze what caused the VM, the ASI exits, right? One of the big challenges in ASI is kind of like, okay, what exactly do you need to put inside your page table? And as we said before, whenever we have an ASI exit, we can log this information. And if we look briefly at the column, it's like on the left side, we see the instruction pointer, which instruction pointer caused the page fault, caused the ASI exit. We can look at the address, what was accessed, and then we can start and do some analysis. So if you look at the middle column, which is accessible, it's basically an interpretation of the instruction pointer saying, okay, what source line accessed this data. And if we go to the next column called allocator, it's basically our bug framework trying to interpret, okay, who allocated this data. And this is very useful because then if we see which code allocated this data, we can maybe say something smart whether this data is sensitive or nonsensitive. And hopefully we can decide it's nonsensitive data and add the flag like GFP, local nonsensitive or GFP, global nonsensitive to edit. So next time that we really run the benchmark of the experiment, it will already be in the ASI map and it will not cause ASI exits. We can even do a little bit of more analysis if we go to the next slide. So for example, we can look at the allocators and kind of like see which allocators were the most common to cause ASI exit. And okay, the top one is something that we couldn't infer who allocated it. But again, you can look at the data, see which source line accessed it and maybe say something about it. But if we look below, we see for example, the second place is basically allocating PMU context, basically Perth events context, which potentially we can argue that the context of Perth events maybe is not that sensitive and we can try and say okay, those allocations we can consider to be nonsensitive and add them to ASI. We can see, I think it's the fourth line here, you see SDLLC size, that's a global variable that apparently was accessed quite frequent. And we can look at it and say okay, this global variable is probably not that sensitive, just basically says what the LSE size is. So we can add the tag of ASI, not sensitive. And that way we can have an iterative process of saying okay, we want an allow list, we want to only have an ASI data that we need to be done sensitive. And we can use this debug framework to give us some visibility as to what actually causes ASI exits. And hopefully by doing this iterative process, we can add more and more data to get better and better performance for ASI. Now Alex is going to talk about the current status of the ASI project. Thanks, so just a quick story of where we are coming from with ASI. So the idea was suggested after the L1TF speculation I think discovery, this was by the end of 2008, it was first presentation at KVM forum. And it's mostly inspired from a fix that Microsoft has done in their high pervy for their high pervy mitigation. We had more discussion at Linux Pronger in 2019. And also, first development where started at Oracle where we have submitted several RFC. There was a RFC V1 which was very KVM specific and the comment from upstream that was that we needed something more generic. So that's what the version 2 has proposed. Which provided a kernel ASI framework and KVM is just a consumer. V3 was the integration with PTI. This was another comment we have upstream that ASI is similar to what PTI is doing. So we should be using ASI to implement PTI as we have a generic framework. V3 added some page table management function, which may not be the final solution we are going to, but this was the first try. There was a V5 which was supposed to be released after Linux Pronger this year. So this was adding the ASI synchronization in addition of V4. But at Linux Pronger was also this presentation from Google of different implementation. So since then we are working together to go to common solutions instead of developing our own implementation. So really the work we are doing now is some collaboration between Oracle and Google to define a common solution and to convert our implementation. So we have some things to, there was a lot of work to do, especially around how we do the page table management, what would be the solution we are going to to manage the interactive section, the page fault. How do we do the synchronization of the CPU threads with CPU stunning. We need to look at what's happening also on the core scheduling side where they also have a CPU stunning mechanism. And also there is the PTI integration to investigate because of the different implementation. But the goal is really to provide and to upstream a common implementation and not to have different implementation. So there's a lot of work to do. We also preparing some preliminary work, which is to relate it to PTI, which is going to help and simplify the support of ASI, which is to defer some page table switching that PTI is doing currently in assembly code to C code. So we are doing a lot of cooperation and there is, but there's still a lot to do at the moment. So that's it for this presentation. Thanks for your attention if you went that far. And hopefully we'll have some opportunity to get some feedback and questions at the KVM forum. Is there anything you want to add? No, thank you very much Alex. We'll be happy to take questions again. Thank you.