 Hello, everyone. Changpeng and I are coming from Intel Storage Software team. We are talking about the evolution of SAPK towards secure container storage service. SAPK is a storage performance development kit. It provides a set of tools and libraries for writing high performance scalable user mode storage applications. SAPK Rehost is one SAPK-provided application. As a high performance storage authorization solution with flexibility and robustness, it's already widely deployed by lots of SAP and enterprises in their infrastructure. In this topic, we start at the consideration on VM-based secure container. From recent papers and reports of AWS or Ali Cloud, secure container is the battle rock for their public cloud container service. So it is popular in SAP. And commonly, secure container is machine-based, like Cata containers. Here, the evaluation work we have done is based on Cata containers. Since it is machine-based, can SAPK Rehost directly be applied to Cata containers? Let's take a look at the characteristics of secure container. The first characteristic is high container density, not like a traditional virtualization scenario that only tons of VM are running in a server. Secure container, so more than 1,000 of containers on one single host, which means more than 1,000 of lightweight virtual machines are running on a single host. The second item is resource overprisoning. CPU and library source will not be planned and pre-exempt to specify virtual machines. Probably, these resources will be scheduled to different container dynamically, and they are always tense seriously, since there are a lot of containers. And commonly, containers do not only focus on extremely high performance, but it also focus on flexibility and robustness. The first problem we might with SAPDK and secure container is about the high container density. SAPDK application uses polling mode, which is an end-is-well loop to pull each device, check, and process their request. For warehouse device, SAPDK pulls each word skew in wrongs. Polling mode with this type of application framework is good and efficient for heavy workloads. But for container, their L workloads are not always so heavy. So when containers are in high density, pulling to query massive word kills is not efficient at all. The second problem is about CPU and memory resource overprisoning. When some CPU cores are running SAPDK warehouse, then this course will be occupied or pinned by SAPDK warehouse all the time due to pulling mode, just like our slogan, don't interrupt me. I'm pulling. For memory, due to user space device, DMA operation, SAPDK requires memory to be pre-advocated as a huge page memory. The memory pre-advocation and the CPU occupation required by SAPDK warehouse have barriers to overprison all server resources to container. OK, towards the secure container, can we involve SAPDK applications to be interruptible? Let's first look back on SAPDK internal. One of the primary aims of SAP is to scale linearly with the addition of hardware. To achieve this, SAPDK execution unit must be independent from one another as much as possible and avoid soft logs. Instead of placing shared data in a global location that all threads access after acquiring a log, SAPDK takes a different approach altogether. SAPDK will often assign that data to a single thread. When other threads want to access the data, they must pass a message to the OIN thread to perform the operation on behalf of the threads. This strategy, of course, is not new. For instance, it is one of the core design principle of Ericsson-Lang and is the main currency mechanism in Google language. SAPDK provides several layers to construct the messaging passing in fast infrastructure. The most fundamental libraries in SAPDK, for instance, don't do any message passing on their own. They depend on SAPDK thread abstraction. SAPDK thread abstraction provides a basic message passing framework that defines key primitives. First, SAPDK is an abstraction for a lightweight stack list thread of execution. A low-level framework can be executed. And SAPDK thread for a single time list by calling SAPDK thread pool function. The lightweight thread is the fundamental abstraction for threading in SAPDK. There are also a few additional abstraction layers on top of the SAPDK thread. One is the SAP Polar, which is an abstraction for a function that should be repeatedly called on the given thread. And the other one is the SAPDK message, which is a function pointer and a context pointer that can be sent from a thread to another. The library also defines two additional abstractions. One is SAPDK our device, and the other is SAP our channel. In order to implement a message passing straightly, the code would describe some objective within global state and also some cross-read context associated with that object. This makes SAPDK very portable to a wide-level variety of asynchronous event-based frameworks. SAPDK provides a framework for writing a synchronous pool mode, shared nothing, several applications. The framework defines several concepts, reactors, events, and lightweight thread. Reactor, the event framework spawns one system thread per call. Reactor stands for the system thread. Events, reactor connect to each other with lockless queues. Events can be passed between the reactors. The lightweight thread is a representative for SAPD thread, who is a stackless execution unit, lightweight thread in SAP linked in a list inside a reactor. In this event framework, reactor running a wall loop round and round pulls its event queue process incoming events. Also pulls the SAP thread attached in it. So that is the puller executed. This framework shows how to put SAPDK components together. SAPDK community uses it to build the applications like a rehearsal target. It glues SAPDK components in a polling programming model. So here, for SAP Warehouse target, it's applying related components in SAPDK with use space and the polling techniques. When run via SAPD Warehouse target, first step is when create a warehouse device, SAPDK will create a specific SAP thread. Inside the lightweight SAPDK thread, pullers are registered to tick and process our request in which to queues as a front end block device as a backend. All functions are driven by pulling executions in reactors. Now we want to evolve SAPDK application in interrupt mode. It should be compatible with current SAPDK event framework and SAPDK thread abstraction, which is created for effective message passing, a synchronization, and a run-to-completion characteristics used by applications inside SAPDK. The interrupt abstraction object now are based on Epo and even the FD on Linux platform. It is the core concept to achieve interrupt with notify mechanism. Initially, it is one Epo instance with registered target file descriptors. Different target file descriptors represents different interrupts or even the source. For example, even the FD is used for internal queue notification. Soaked FD is used for network data coming or out notification. Timer FD is used for periodic work. VFL used FD can be used for user space device interrupt. The most important one is cascading the Epo instance for group events. If we map the interrupt abstraction of Epo group, then we can get SAPDK reactor and SAPDK thread to be interruptable. When applied interrupt abstraction to reactor, the reactor itself contains one top-level Epo instance. And one Epo and one even FD represents for its internal event queue is registered to the Epo instance. Or small other registered file descriptors are all sub-level Epo instance, which represents SAPDK thread, which is attached to this reactor. As we just said, SAPDK thread itself also contains one Epo instance as a central role. Then we can register corresponding file descriptors to the SAPDK Epo instance, instead of registering polars to SAPDK thread. For example, at first, message queue polar or message queue process event can be replaced by one event FD for message queue. Polars with interval time can be replaced by time FD. Hardware queue polars for NME device or what help device can be replaced by VFL event FD. NBD polars can be replaced by just registering its FFD to the Epo instance. Epo BDO can use event FD together with Epo BDO. If a SOC group applied the interrupt abstraction for its connections, then SCSI or NVMF polars can be triggered directly by the SOC group interrupt. After applying interrupt abstraction into reactor and SAPDK thread, then one SAPDK application can run in such a methods to get itself running in interrupt mode. The application will do block weight on reactor Epo instance and do non-block weight on SAPDK thread Epo instance. Then the system stress of the reactors will only get blocked at the reactor Epo instance. If the Epo instance of reactor wakes up from its event queue, then it directly come to the event queue and process the event. If it is waking up from one SAPDK thread message, then the reactor will come to the specific SAPDK thread. And then the thread will process the message queue. If there is one DPAO L completion, then the reactor Epo instance will wake up and come to that specific SAPDK thread, and then the thread will come to process the DPAO event. Each time the process is completed, after waking up, the system thread will come back to reactor and block on reactor Epo instance. SAPDK further provides a full block stack as a user space library that performs many of the same operations as a block stack in an operating system kernel. All of them are designed around the message passing instead of locking. And most of SAPDK libraries make several assumptions like message passing, event, coroutine, or lightweight threading framework. So if the underlying threading model is interruptible, then many of the libraries inside SAPDK can run directly in interrupt mode. For example, the basic BDAO modules like readBDAO, speakBDAO, GPT, or mallocBDAO, for SAPDK, Blob Store, and its upper layer, logic volume, they can all run directly in interrupt mode without any modification. For SAPDK, Blob FIs, and its fields module, they can also run directly in interrupt mode without any modification. So here we have prepared a minimal set of interrupt mode. We have targeted the interrupt mode for secure container evaluation based on some interrupt mode patches. It will use Enix ALBDAO as the storage backend. It will export the Enix NBD service to host, and it will provide a warehouse block device to the lightweight machine of secure container. A flexible block stack with logical volume are there for advanced block operations. Users can also manage this evaluation application still by SAPDK RPC methods. The difference of this application with the general SAPDK application is that it won't be pulling or occupying CPU cores anymore. When there is no workload, its CPU percentage is lower than 1%. A set of tools are provided by SAPDK. Some of them are also easy to run in interrupt mode. For example, the BDAO Proof tool. It is a benchmark tool of SAP Blob device for thread and event work coast. A set of tools are provided by SAPDK. Some of them are also easy to run in interrupt mode. For example, for BDAO Proof tool, it is a benchmark. It is a performance. A set of tools are provided by SAPDK. Some of them are also easy to run in interrupt mode. For example, for BDAO Proof tool, it is a performance benchmark tool of SAPDK Blob devices for thread and event framework coast. We have added a parameter to BDAO Proof to set interrupt mode flag. With this simple modification, we got some brief performance evaluation on interrupt mode for you to have a preview. Of course, there is an obvious performance drawdown. When you run BDAO Proof on memory BDAO, it shows about the throughput of interrupt mode is less than 1 third of the throughput of the polling mode. When run BDAO Proof on NVMe-based ALBDAO, it shows about the throughput of interrupt mode is around 5% to 8% worse than polling mode. This is not good for extremely high performance situation, but for the container scenario, it would be enough. OK, the last part. On top of interrupt or interruptable SAP application, we can provide a secure container storage service. At start, the most straightforward step is to providing device volume from SAPDK we host by dark command. Provide volume service to Cata containers where the interrupt or SAP will host the target. And SAP Logic Volume can do same provisioning and snapshot like a logical volume on a device member. SAP Logic Volume provides the basic functionalities to meet a container root FS. Provide root FS service to Cata containers where the interrupt or SAP will host the target and the container D. How users run SAPDK is much more diverse than we expect. From SAP GitHub issue and the Slack channel, we realized that there are users who run SAP inside the dark container. When continuing SAP application, there are no SAPDK specific changes needed. However, generalized SAP application always requires dedicated CPU costs and huge memories. So exclusive results, OQ patient is the main issue for continuing SAPDK application. If SAP application inside the container is running in the interrupt mode, also with not huge memory, the container density will not be impacted by SAPDK container. Currently, only basic interrupt mode is supported if we want to get SAPDK to be more suitable to secure container scenarios from container infrastructure to container instance. So let's give a summary first. Pulling, pinned CPU and huge memory player location can be avoided from non-performance situation. With the interrupt mode, SAPDK host will be a good choice to provide a storage service to secure containers. And for the future in evolution, we may add interrupt support on use with hardware block device back end, such as the MME driver and the block layer with whole device driver and the block device layer. And also, we can add the interrupt support on modules related with the network particles, such as MME or fabrics and the iSCSI. Also, with a running mode switch between the pulling and the interrupt, the last is we can also show non-huge memory support for all the non-DMA-SP application. That's all for today's presentation. Thank you.