 Thank you very much. So my name is Roger, I work for Citrix on what's called a platform team. We have people that work on the enormous project, and many work on Zen. I'm here today to speak about how we can remove the PCI emulation. I will tell you the description of the PCI class, and we will speak a little bit about the current state of PCI emulation inside the center. We've done it for different kind of areas that we support. Finally, I will speak a little bit about the work I'm doing on the PCI emulation. Let's start with the basics. PCI is a pass that was developed by Intel in 1992 in order to attach devices to a computer. It was originally designed for servers, but it was adopted for desktop also quite early. So basically it was developed by Intel in 1992, but now Mendinga and Kaiser have contributed to the stack. It's a group of several companies called DCMC, which is formed by more than 800 different companies. And they inspired the PCI with their specifications. When it was started, if there were all kinds of pass like this, there were a lot of different kinds of pass, which was for computers, and basically PCI was super simple. And it was the main pass, you know, almost any computer. The main two standards are the first one, which was introduced, which is the PCI motor pass, and then PCI Express was introduced in 2004. In between there were also several other standards like the HDD and PCI-N, which basically started up with the PCI-L of that specification. And it was mainly used by artists, actually, because it was very long, and PCI-N was not initially slotting the model, but it was applied by artists. And then the rest of them would be PCI-N as well. So this is how it looks on a laptop. This is our part of it, and in this case the first four are PCI Express as well. And the last one is a conventional PCI slot. If you still have testers and you open them, you probably find something interesting. Maybe you might have different lengths, different lengths on that kind of slot, but basically you don't have something like this. It's a very popular graphic card, but then you won't have anything like that, so you don't have classes. And here we also have a picture of what we would actually find in tomorrow's classes. This is a USB-3 PCI Express card that provides for a USB-3 port than that. And one of those PCI slots. Now let's move to what's actually interesting to me, which is about the power power of PCI, and more interesting than the software power of PCI, which is what I've been working on. So basically it's called the PCI configuration state. There's a standard that's provided by the PCI and by the PCI class, and basically it defines, or provides, e-devices with a certain amount of space. In this case, it depends on the implementation of the specification, which is usually false. You have 250 files or 400 e-devices or a state e-device that use to store the configuration of e-device that can get set by the device. This is very interesting, because it allows the rest of the standard way to detect devices. Basically, this is a standard mechanism of detecting devices using the PCI configuration state. Each device inside of this configuration state has a unique identifier, which is formed by the device and the function internals. This is usually called BDF, and it's an unique couple of the blocks. So basically it's different devices as we need it here and now. So in the configuration of the device, the first 64 bytes are standard, they are the same across all different kinds of devices. This is called the PCI card, and it's served in all the devices, which allows us to easily identify devices. And the rest of the space is used to store what's called capability, which is per device that usually contains more, let's say, extra functionality of those devices, like the same sign interrupts, the same X, where they have SRIUB, all those things that are stored inside of these capability disclosures. So basically, I hope you know of the PCI that would make discovery devices much more easier, because it was a standard interface for discovery energy. And that's also another, what usually depends on the different root forces kind of all of us, which means that it was the one device in the bus, in case you were out, where you have a black device or not. This is usually called the root forces kind of the bus. And while you can do that, but in general, most people, for example, you should be able to find in order to figure out if the device is a present of the system. And then we can speak to them about the PCI bus. We think we see the way to access the configuration space with either from the IOS page or from the memory space. The IOS page access is called the legacy. It's the first one that was introduced. It's made of PCI conventional or PCI local bus. And then we also have the way of accessing the configuration space from memory, which is called the enhanced. And it was introduced with PCI trust. So this is how PCI, a lot of PCI apps that look commercial or PCI apps that look to be done on the IOS space. So this is how PCI, the IOS space. And it's important to note that in CPC, the IOS space is made of 64 bugs, so it's impossible to fit a whole multi-register space inside the IOS space. So basically, what they did in CPC was to use indirect access to the configuration space. This is done using two different registers, CNA, is the register that contains the others that we are going to use. So in this case, for example, you can see that on the register we will write the last class of registers that we want to access. We write this on the register and then we read the other remark on the CFC board. This is good, but it's not very obvious because for each data that we want to access it provides us with two different instructions. So we have to write to CFA and then we have to write to CFC. This is fine, but we also want to be designed with SMB money because every kind of you access it in some kind of login in order to make sure that no other CPU is also going to access the last. Because if you have, well, let's imagine that you write that the others are going to access to CFA and before reading CFC and obviously when you write to CFC you can access to CFA then you will read the report back. So basically you need some kind of access to the login on that. But to solve some of the problems that CFC has in 2004 this is called the enhanced procuration access and it basically is done by mapping the full procuration and spacing to memory. Memory, as you can see, it's huge. You have 64 bytes of memory so you have a lot of space so you can basically map the full procuration spacing to memory. And then basically when you want to access the specific register you can access that as you want. It's a similar structure and you can require an extra login. Well, in this case the register they have here is just a little bit long but the higher part that access to the SDV was called the base address of the memory procuration space. The firmware can position the access to the procuration space in memory so you are going to have the staff and then you will be starting from that to automatically. It's also important to note that if you see here the resistance here it's 6 bytes of battery actually 8 bytes on but on the enhanced procuration there is a register that needs to be long. That means that you can have 4 kHz data using the legacy one and also on the enhanced one you there are some hardware that supports 60-40 access it means that you can use more in order to read from memory but it's not guaranteed so it's not going to be used from the previous system because you don't really know how to support it. So how does PC800 properly work inside of Zen? For those familiar with Zen you have to follow the previous domain you usually go to dom0 it's the first guest that's started after launching the hypervisor terminal so basically you start then which is a kernel itself which is my order and then you start again this first guest is the previous domain it's called dom0 it has special abilities over the hardware and over the Zen itself it's basically used to control it. So in this case dom0 makes full access to the PCM from the personal space because it means to manage all the hardware under it so basically Zen on that access is to the CFA and to CFC in order to maintain stabilization so basically dom0 makes full access to the PCM from the personal space then when you pass through of PCM devices to guests we have two different kind of devices one is for the BB and the other is an aggregation for parallelization it's a personal that was developed by Zen and in that case if you want to pass through and the guest wants to access the configuration space we use a specific PD protocol in order to access it that's called usually PCI-con and PCI-pack and the specific is a very thin specific way of accessing it but if you're a guest in your hand the guest needs to have full access to the PCI-con because it's easy to evaluate the configuration space for the guest so we pass the we need to have full access this is actually the diagram of how the system works usually so we have the dom0 which has full access to the hardware basically so we can access the PCI-con without any limitations and then I have the PD-S with PCI-pack we can see we don't need the PD-S and PCI-pack speak directly so that means it's faster it's less overhead but it's also using the same specific protocol it means that PCI-con you want to port the Zen and you want to port PCI-con this is fine but sometimes you want to port the Zen because it's a growth source and then for the GMS we actually have accesses to CFA, CSP and to any access configuration space from the guest so basically one of the guest-type classes one of the regions we get without using that and we forward these accesses to the GMS we have an end-to-end with the device is fully emulated it will be handled by the GMS there is the reply to the guest we have to be fast on the other hand sometimes the response is directly to the GMS sometimes the GMS has to be reached from the hardware and all of that to the guest itself so yeah, this is nice because we don't need to modify the guest but it's actually much more slow it involves traveling to the Zen and then we can go to the GMS and then go to the GMS and then go to the hardware we won't try to do this at all so now we have an intersect that's called PMA which is basically an end-to-end guest from the end point of view because it doesn't come from any of the devices so basically it means that it doesn't have the device model it doesn't have the end-to-end system because we don't have the end-to-end devices and also we need to emulate some back-to-end devices with the IOLP, the P, the P the HPED I think and maybe something else but all those devices are no longer a multi-faceted concept so basically beyond the multi-faceted device we are going to do a VHS that's beyond the multi-faceted device this is nice because we can go on this new home that's called the 0-1-1-U because it doesn't provide the end-to-end so we can do it for both 0-1-1-U but at some point we have to act because we have to ask ourselves how do we like VHS to access the configuration of space we would like to use the beginning way we would like to use something like the end-way that is the end-way and we decided that the best thing probably was to do something like this so basically in order for VHS to get some kind of main key access to the PCI configuration of space we decided to do the PCI in which we were very excited we called this vehicle a PCI with PCI in order to argue and basically the idea is that by putting some configuration of space in order to be able to handle a pass-through for PDAs and GMS without having to go to the end for the GMS, I mean that you already required the end so it's not that much for me maybe a little bit because if the end-way doesn't have to access the real power you can, let's say, mark the end inside of a container or something like that you can easily get there because it doesn't need to run out of wood anymore because it doesn't need to access other things and then what we did is lay inside the same PCI of the pass-through for the GMS the initial implementation is targeted at non-zero only we look at basically because on non-zero the speed is much more relaxed because on zero you feel you have access to the power so we don't have that much experience but on zero basically it's hard to watch all the traffic there it's the traffic road that we run so if you want to run at non-zero it's the same as normal and in that sense the stability of the PCI emulation that we have now is not very strong because it's only being targeted for a long time so we look at a medium of the features meaning in order to have some kind of working PCI emulation on zero so basically we need to have access to the PCI here in order to the deck when it gets access to the head and then we also need to have access to the entire site and the MSA extra to do four of them now so yeah the first system we want to implement in the Intel 10 was to actually have access to the registry server that allows access to the descendant version of the and we have to have access to CFA and CFC in order to the deck when they get this kind of access to the configuration of the deck and then from there we can obtain the media from the registry server that makes exactly the kind of access we also have to have access to the actual configuration of the that every single basically what we want to have access we want to have access so then when we get the media we can actually search for the devices that we can obtain and we need device as a link at least of the devices that we are working I think it's easier to see that with a little diamond so basically the theme of the talk would represent the access by the desk so we can obtain both themes and we can operate on the media source and using the media we search the ZMP and we can obtain all the digital devices in order to obtain the digital devices that we have to have access and when we have that we get a list of registers and a list from the desk and we search for the registers that we have to have access to to use this for a few moments so basically if the register is not if the register is not registered so this is a PCI header this is actually what's called a normal PCI header this is an ID and then at the moment we only support a module of devices with a type 0 that 0 is for the normal header PCI and that's also another type of header that's type 2 I think that's the car wash header that has devices normally there's that and then the themes everything green are the ones that we actually track so basically we have the common register on the PCI header we basically need to track the register and we go on to the desk and it's time to enable and normally we go in the bus into the desk the bus the bus is the data this is the bus is basically and what the header is about the distribution of the data we also track builds the data on the gas drives to be able to take that type of memory and finally we also got the access to the normal data to be able to integrate it into the next world problem and then we also have access to the MSI and then we go into structure this is found after the header and this structure is used in order to set up interlocks the gas basically arrives to these fields and then somebody will get interlocks in there the MSI is used to signal that you want to enable or disable an interlock MSI interlocks then we have the message address and the message address we have those few signals on the deck and finally the NAS feeds are used in order to mask the feedback that we want to disable this is also the more complete MSI header because we are using 64 bit addresses and it also has a NAS feed there are people that don't have a structure that only contains 32 bit address fields and don't contain the NAS feed or any of it we also have access to the MSI X capabilities and the MSI X structure the only difference is that instead of having a global message address and message address we have one for example also another of the differences which are the structure is that the part that resides on the configuration space is very small it's at the top of that slide that's the part that actually needs in the configuration space because from the information control which is used in order to disable MSI X and then we have two fields that actually point to a memory address inside of the memory space at the outside of the configuration space at the main disabled usually if you don't use disabled it's quite big so if you will place disabled inside of the disabled configuration space people will probably overflow so how about that so in order to test this it's not very easy to test all those kind of things when the package is inside the configuration with your system and then you can find your way to actually test it so we decided to create what we call a user space that's hard as it basically it's simple but what we basically do is we end up with a very simple program that we want and it does some kind of simple test this is fine because it allows us to test a lot of functionality like adding registers removing registers and things like that but it doesn't allow you to test specifically for the patients like MSI emulation MSI emulation you can actually test that in the actual hardware in the actual hardware in order to do that some of us have been hacked so we didn't like or thought not yet they don't need to send they've got to be found it's the MSI emulation the main reason is that right in my car they also have some pre-executives in order to test this the pre-execute is not in the same we only have to do this we do one of these types that's why it will meet the working progress but the things we work we have to work with as usual we want to test with different tests we can send the test of the party force to send the mail and then we can send it to the city for the infusion of our mail so I'm sure I will it's realising and then the field of work seems to be quite important the residents are working on implementing a special emulation for non-zero super-execute that's our main reason then I will probably focus on the role so we can use the role for deviating media and finally the idea would be to switch the 8pm before the past few months from the end and use that to save all the pre-executives with that that's where the question is there are many people I could imagine that you have heavy loads in your sense and maybe you want to even for each guest to use a magic card as a for your future work do you think of sharing that in some way the question is whether we are planning to share a graphic card so what do you say it's not an indication of the end you just need a graphic card to allow some kind of sharing that thing there are some kinds of problems that already have that license and what should they do for some kind of GPU sharing thing but I'm not sure about the status of using general simpler because crazy media I think so I think it should work with them I think for this it's not a specific card that the GPU sharing will be possible maybe easier because we don't have an editor but the problem is that from the personal you know the performance how much does it change using this new magic card from this card performance in terms of performance I haven't mentioned that yet but I guess the question is how much performance difference there is from using the end quality thing I haven't mentioned that yet I expected some performance gain because we talked about there's less moving around and so on but I haven't talked about that and quite sure that some performance improvement is done because we need it also I think the main focus was to get something working first then we will talk about the performance I think for the GPU we don't have devices so when you create them it's hard to miss the GPU the ECI for which the HVM abbreviates the reply ECM because there are devices that are few with a more limited HVM like the new work card with a more limited HVM that's not something that we can do with this HVM this ECI version is only going to be used by Pascal so when you do Pascal physical devices to a guest but the emulated devices are mostly HVM are currently supported with HVM cases that will work with HVM and won't work with the ECI I don't think so because at the end there's a passion there so you can place it inside the HVM or in a place inside of them the place is not relevant in that case so I think as long as we have the right to it's going to work if it doesn't work it will act thank you very much