 Hi, this is Jun Nakajima from Intel. Today, I'm presenting how we can include IOP devices into a trust execution environment. The first generation of a virtualization-based trusted execution environment, TE, VMs, virtual machine, such as Intel's TDX guest, trusted domain extension guest, do not include devices or accelerators into a trust boundary of the TVMs. So the devices are not allowed to read or write the TVM confidential memory. Because of that limitations, the current techniques used for the TVMs incur significant performance overhead, especially for high performance IO. So this is our agenda. I'll start with existing IOP technologies and implementations for VMs. Then I'll explain the limitation of the current IOP virtualization when used for their TVMs, especially with IO virtualization. I'll summarize TDISP, TE Device Interface Security Protocol. And then I'll summarize the TDISP architecture and then requirements. I also introduce the Intel TDX architecture in support of a TEIO as Intel's implementation that helps allow direct assignment and an establishment of a trust between a TD, TD is a trust domain, and then TDI Device Interface hosted by TDISP compatible device. Then I'll talk about the software changes required for Intel TDX in support of a TEIO. I'll show some high-level software flows for basic operation and then show software touchpoints or new functionality to enable Intel TDX in support of a TEIO, showing a conceptual structure of a VMM with Intel TDX support. So let's revisit the current IO virtualization technologies. There are two types of IO virtualization technologies today. First one is software-based IO virtualization. Those include virtual devices or synthetic devices, such as Vortail. With software-based IO virtualization, the VMM exposes a virtual device, such as a network interface controller, Nick, to a VM. And software device model in a VMM or the host OS emulates the behavior of the virtual device. The device model translates the virtual device command to physical device commands before forwarding the request to the physical device. And modern processors or platform provide features to reduce virtualization overhead. Then maybe you try to buy the VMM to allow VMM to use a direct access to hardware IO devices, especially Intel, for example, supports the following hardware-assisted IO virtualization schemes for direct data movement without needing software assistance. For example, direct device assignment, or SRIV, or a scalable IV, scalable IO virtualization. Here, I would like to define device interface. It's the unit of assignment for IO virtualization capable device. Now, let's take a look at the direct access to hardware IO devices in VM. We need a hardware support, for example, MMIO to access device interfaces, such as the registers. We also need a DMA remapping capability, or the IOMMU support. Also, for the device interrupt isolation, we need interrupt remapping feature. The point I would like to make here is the VMM needs to be trusted, because it set up hardware for the feature above for protection and isolation for both the host and the guest. And for TBM, trusted VM, devices also need to be trusted. Now, take a look at the TBM and IO virtualization. Here, I'll talk about how they base trusted execution VM environment, especially in this presentation, our focus on virtual machine, such as a TDX guest. As I said at the beginning, the first generations of a TBM, such as Intel TDX guest, don't include the physical devices and accelerators, such as FPGA, into the trust computing base of the TBMs. Devices are not allowed to read or write the TBM confidential memory. Because of the limitations, today the VMM exposes a synthetic device interface to a TBM for IO virtualization. The synthetic device interface is defined to be virtualization friendly, to enable efficient virtualization, compared to the overhead of associated with IO emulation or soft emulation. But this, however, requires the TBM to stage the data that needs to be sent out, copy in, copy out, from devices in shared memory, which is designed to hold the contents accessible to the VMM and the hypervisor. For us to protect the confidentiality and the integrity of such data, the TBM is expected to use cryptographic protection on IO data. For some synthetic IO devices like a network storage, TBM may employ software-based cryptographic techniques for data protection. For example, the networking, the TBM may use TLS, transport layer security, or some mechanism to protect the data sent to the NIC. But this cryptographic protection actually disabled the TBM to offload the computation directly to the GPU or PGA, because they need to consume the data. So if it's encrypted, they cannot consume the data for computation. So the question is, if we can trust the GPU or FPGA accelerators, can the TBM use them to offload the computation? So this is the case. The keys for the encryption decryption can be shared with a trusted device and a TBM. For example, we can use MKTME on the interplatforms. However, the issue is, the problem is if the data link is out of a TCB, then, for example, because of the switches in between, the TBM still needs to use a shared memory combined with cryptographic data protection. So as we saw, the following are requirements for TIO. We need some trusted entity outside a TBM, because a TBM is not included by the TBM's TCB today for TBM, trusted VM. And we also need a mechanism to trust the devices or trusted devices that would require device attestation. We discussed this one. We presented this device attestation yesterday, actually. And then we need additional hardware support. For example, we need to make sure the PCIe data links are secure. And also, we need protection or access control for the DMMA mapping and MMIO. And more importantly, we need a standard, because devices are platform-independent. So trusted devices need to be used for vendor agnostic ways. And I believe the answer to that is TDISP, T-Device Interface Security Protocol, which I introduce next. TDISP or TDISP defines an architecture for trusted IO virtualization, which includes establishing a trust relationship between a TBM and TIO device. TIO device interface, called TDI, is a unit of assignment for IO virtualization capable device. For example, a TDI may be an entire physical device or SRV virtual function. And second thing is securing the data path, PCIe interconnect between the host and the device. Transfers must be cryptographically protected to avoid or to provide confidentiality and integrity, and then replay protection to TBM data. Such schemes must also be geared against the violation of a producer-consumer ordering, and then support TDISP assignment and removal of a lifecycle in trusted manner. And then more details are defined in the spec here. And TDISP builds upon the foundation provided by the CMA SBDM. The SBDM, PCIe SBDM, is a security protocol and data model specification. It's about the standard messaging data object sequences for performing messages, message exchanges between devices. The device exchanges include authentication and provisioning hardware identities, measurement for firmware identities, session key exchange protocol to enable confidentiality with integrity-protected data communication, and other related capabilities. The CMA means component measurement and authentication. It basically defines additional optional security feature. And this is used for the software session. I'll talk about that later. The other one is IDE integrity and data protection. IDE provides confidentiality, integrity, and then replay protection for TLS, TLP, the transaction layer packets, which is also PCIe defined by PCIe. It's a loadable hardware layer. And then this provides the implementation for securing the data pass, the interconnect. I just mentioned the previous slide. This is TDISP architecture overview. As you see, there's a hardware interconnect the PCIe fabric here. And the color coding, this light blue is a TCB of TVM accepting the device. And this color is a TCB of all TVMs. And then the yellow stuff here, yellow or orange boxes are not TVM, TCB. And then this is a key element in this architecture, which is TSM, T Security Manager. This runs on the host. The other thing is this is a pro device. It's called the DSM, Device Security Manager. They are a logical entity. And then TSM is I'll talk about more details of the functionality of TSM or DSM later. And then TSM and DSM build trust relationship using SBDM, TD, ISP. And then, obviously, this is TVM. Then we see the two TVMs here. And they have access to the TVM, TDI. This is a T-device interface. This is, again, the unit of a device assignment. And as you see, they have a state. Also, TDI can be assigned to legacy VM. And in that case, it doesn't have the TDSP state. It's just to act like a legacy device. I'll explain the TA here. And before I explain the TA, the standard added definition of TVIT. TVIT is defined in IDE prefix. It essentially allows the originator of a TLP to indicate a TLP is associated with a TVM. And TA in the picture, it's a translation agent. It's permitted to use a TVIT. And that provides access control to TVM assigned memory. And then memory mapped IO registers, like a trusted MMI or a DMA. And it also identified the request originated by TDI in a run state. Run state is one of the TDI state. And then whenever the TDI is ready for use, it should be in a run state. Next, I'll talk about the more detailed TSM, the functionality of TSM. So first, it provides interface to the VVMM to assign memory, CPU, and then TDI resources to TVM. Also, it implements a security mechanism to access control. For example, IOMM translation table or MMIO. They are still a bit vague. I'll explain this with more using implementation. That's fine. And then also, like I mentioned, the TDI has a TDSM state. And then TSM uses a TDSM protocol to manage the security state of TDIs. And then also that it used to establish the ID encryption keys for the host. Now next, I'll talk about TSM's functionality. Again, the TSM is a power physical device. This is a device. This box is. And then you have the TSM power device. This is used function as an authentication of a device identity and then also reports a measurement of the device. This is a device level, not the TDI level. And for the SPDM session, then once the key is there, then it configures the ID encryption key in the device. And then it manages the TDIs, such as the configuration or locking the TDI configuration, also report the TDI configuration, and also attach or detach TDI from a TVM, so forth. Now next, I'll talk about the Intel's TDX for TIO. So this is architecture overview. Notice this is a TIO device. In this picture, in the previous picture, the T device was at the bottom, and this has been moved to the right-hand side. But in terms of architecture, logically, this is consistent with the TDX architecture that I just showed. And the number one, this one, is a T device. In this picture, this device has one physical function and a legacy virtual function, and it has two TDI, the interface, trusted device interface, two. And then the DSM, and then the PCI ID port. OK. And this, in Intel TDX with TIO support, this functions as, this means that the Intel TDX module and then the TPM, TPA is the TDX provisioning agent, functions as a TSM. And then number two is IDE, we just saw in TDIF's architecture. And three, the hardware has trusted MMIO and DMA mapping support, and also have PCI ID root port support. And then number four is the combination of Intel TDX module and TPA. TPA is so-called architectural TD. TD is a trusted domain. It's a trusted guest or TBM in TDX implementation. And between TBM and TDI, the trusted MMIO and DMA mapping are provided in this picture. Like I said, this is a software session between the DSP and TSM. And this is a hardware session. This is a power device, and then this was also a power device session. OK. The next I'll talk about changes required for the Intel TDX in support of TIO. Before I show the changes required to enable TDX, I'd like to show the high level of software flow for basic operation, including initializing how we initialize the hoist machine for TDX support. That's called step A. So let's take a look at some details. So if a TDX module is not loaded, then the host VMM needs to load the TDX module and complete the initialization, and then checking the presence of a TIO support. Then it configures the trusted IOMMU and then the PCX-based root port. Then launch the TPA-TD. Look at how we start SBDM session with a device for secure communication. First, the VMM checks the device capability required for TIO device. I don't talk about the details, but it's about, for example, PCI, DOE capabilities. I'll cover that DOE later. Then it creates SBDM metadata for the TDX module, OK? And it invokes the TPA to start SBDM session with a physical device, injecting the interrupts, basically trigger the TPA action. And PPA uses SBDM to collect the device certificate and the measurement, and then return to the VMM for device attestation later. And TPA passes the SBDM session keys to the Intel TDX module. And then TPA returns the device certificates and the measurement to the VMM. OK, the step C is set up the ID stream for link encryption. First, the VMM creates the ID stream by calling the Intel TDX module. And it queries configures the device ID extended capabilities. And then it programs the keys for all ID substream with both receive and transmit direction. And then it starts all ID substream with both receive and transmit direction. Now, we have a SBDM session and an ID session. They are secure now. Now, what can VMM do when handling TDX messages? TDX messages are integrally protected. Essentially, it's encrypted. And the keys are maintained by TDX module. But the VMM communicates with the device, DSM, and then actually sends, receives the actual TDX messages. So it needs to call the TDX module to prepare, I mean create and then protect, or require a TDX message. And then the TDX module returns a secure payload so that the VMM can actually send or receive the message. Now, talk about the step D, which is start the TDI. Now, talk about step D, which is start the TDI. So the first step is the VMM creates a data structure for the device interface, and then configure TDI. Then the VMM calls a TDX module to move that TDI state to the locked state by generating a TDX message. Then the VMM creates a TD. And VMM assign a TDI to TBM. Now, TD gets the device certificate and a measurement from the VMM. Then TD verify the device based on the TD specific policy. Once the verification passes, then the TD can accept a TDI to the TDX module. Then the TD uses a TD's protocol to send a TD's message, get interface report from the DSM. And TD accept, if it's OK, then TD accept MMIO and then DMA remapping in the report by the function provided by the TDX module. Then TD requests the TDX module and then the VMM to move the TDI into TD's run state. And then finally, it set up, accept MMIO and DMA to pin. Now, we are talking about the software touch point for enabling Intel TDX in support of TIO. Our goal is to take advantage of TD's architecture and implementation as much as possible. We want to define the common layer across the different implementation from different vendors, and then define a vendor-specific operation, including Intel TDX. So this picture shows our conceptual structure when we support TD's and then Intel TDX support. The color coding, the orange means that the subsystem required for TD's support. Like I said, TD's builds upon SBDM and then IDE. And SBDM requires PCI-DOE data object exchange. Essentially, it uses PCI config space. And then yellow boxes are generic TD's support subsystem here. And then light blue shows Intel TDX-specific subsystem or functionality. As you notice, there are a couple of places. One is around TD's for IDE, SBDM, and also the IOMMU. And then also is also in the guest side, TDX support. OK. So let's take a look at more details. So one is associated with PCI-DOE subsystem. To expose a TDI to TD, we need to identify TIO capability of the device. And then once it's identified, then we may need to do additional initialization that are required by the TD's before we assign to the guest. And then TD's specific subsystem, this will be responsible TDI state management. Also request SBDM IDE session support. And then bind TDI to the target TBM. Also for Intel specific operation, for example, we call the TDX module to prepare a required TD's message. Like I mentioned above, right before. Third one, this is around the trusted MMIO or DMS support. Trusted MMIO is managed by the secure EPT. We can use the same page, paging structures for both secure EPT and then IOMMU page table. OK. And the last one is TBM or TD changes. For example, enumeration of a TDI, attestation, also acceptance. And then also extension to the kernel API to support MMIO or DMA. Essentially accept the MMIO or DMA reported by device interface report. We have a white paper published probably during this week. And that should be available in the website where TDX specifications are placed. With that, I'd like to finish my presentation. Thank you for your time. Here.