 Hi, I'm Kohai Tokunaga from Entity Corporation. I'm a reviewer of ContainerD and a maintainer of Build Kit. I'm joined today by Tao Pen and Group. Today, we will introduce and deep dive into faster Container Image distribution technique discussed in the community, which is called Lazy Plink. First, I'm going to give you a brief introduction to Lazy Plink. Plink container image is known as one of the time-consuming steps in the container lifecycle. This has been affected various kinds of use cases, including batch job execution and building images. There are efforts to minimize the size of container images, but not all images are minimizable. Research shows that Plink packages accounts for 76% of container start time, but only 6.4% of that data is read. The root cause of this issue is the current OCI images. In these images, each root effect layer is formatted as TUR and optionally plus compression. TUR is not seekable and cannot perform parallel extraction. So when we start a container, we first need to download the entire image content to the node, then extract each layer sequentially. If the image is large, starting up the container will take long. For solving this issue, several OCI alternative image formats are discussed in the community. They aim to speed up container call to start, so they are sometimes called as accelerated images. Accelerated images enables a sort of technique of container image distribution called Lazy Plink. This allows container runtimes to start up containers before the entire image content is being locally available. Instead, necessary chunks of content, like files, are downloaded from the registry on demand. In this talk, we put focus on two of image formats and deep dive into how Lazy Plink is achieved. Installed it is also a compatible image format for Lazy Plink. This comes with perfect optimization and content verification. This is proposed as a backward compatible extension to OCI image specification. NIDUS is a Lazy Plinkable image format with perfect chunk data application and E2E data integrity. This is compatible with OCI distribution spec and OCI artifact spec. This is incompatible to OCI image specification, so this is proposed as OCI image spec V2 format. First, let's take a look deeper look at Estergizet image format. Estergizet is an image format for Lazy Plink. This is backward compatible to the current OCI image specification, so this is lazy pullable from OCI compliant registries as shown in the following figure. And even legacy and Lazy Plink agnostic runtimes can run this image as a non-lazy image. Estergizet is no usable on a variety of tools, including Kubernetes, K3S, Konchanadi, Cryo, Portman, Buildkit, KaniCo, Core, Buildpaks.io, Konchanar Registry, etc. We will look at how Estergizet looks like with these tools in the later slides. Estergizet comes with performance optimization and content verification. Prefetch optimization allows important files to be prefetched to avoid network overhead. Estergizet is discussed in Konchanaristargizet Snapshotter project. This is based on Stargizet image format proposed by CRV5 but comes with performance optimization and content verification. As shown in the following figure, Estergizet is compatible to Gzip and usable as a valid OCI or docker image layer. Each file is compressed as a separated Gzip member and the offset to each member in the layer is recorded in TOC file, appended at the tail of this blob. This enables random access to file entries and HTTP range request is used for the random access to the layer blob stored on the registry. Estergizet supports prefetch of files that are likely accessed during Konchanar startup. These files are called prioritized files. Before Konchanar startup, Konchanar runtime prefetches them to mitigate the network related overheads or reading them. Other files are fetched from the registry on demand. They are also fetched in background to make the image locally available as soon as possible. This shows the time to take for startup Python 3.7 image among three types of formats. Legacy images cannot start Konchanar before the entire image contents become locally available so startup takes accordingly. Estergizet noopt is Estergizet image without prefetch optimization. This shows the time to take for startup but there is a runtime performance drawback because of the overhead of on-demand fetching. Estergizet can mitigate this by prefetching. Note that creating Konchanar takes longer than nooptimized Estergizet because prefetch occurs during this step. Glassfish server image also shows performance improvement of pool by Estergizet image. And same as the Python image, optimized Estergizet mitigates runtime performance drawback of on-demand fetching. Lazyplink can be enabled on Kubernetes. As shown in the following figure, plugins are needed for CRI runtimes. Sargizet Snapshotter is used for Konchanar and Sargizet Store is used for CRIO. We will introduce these plugins later. There is a real-world use case at CERN for speeding up analysis pipeline. K3S supports Lazyplink of Estergizet. This patch is already matched to the main branch so will be included in version 1.22. Sargizet Snapshotter provides a kind node image that enables Lazyplink of Estergizet. This is a demo of running Hello World GoLang CI as a workflow on K3S. Here I use the upstream K3S that supports Lazyplink of Estergizet. And run it in K3D, which is K3S in Docker. This workflow builds Hello World GoLang binary. It runs GoLang CI Linter and GoStack, then builds the binary. And finally it executes the binary in Alpine, Ubuntu and Federal Continents. APA shows the workflow that uses Estergizet image for the runner image of each step. Each step runs without waiting for the poor completion of the runner image. It takes about 52 seconds. Lava shows the workflow that uses normal Lazy image. Each step needs to wait for the poor completion. So it takes longer than the workflow with Lazyplink. In this case it took about 94 seconds. For continuity, Sargizet Snapshotter plugin enables Lazyplink. This is implemented as a remote Snapshotter plugin of continuity. As shown in the following figure, Sargizet Snapshotter mounts Estergizet layer on the node using Fuse. Each file contents are downloaded from the registry lazily. Sargizet Snapshotter communicates with continuity via GLPC API and provides Fuse-mounted Estergizet layers to continuity. Using this configuration, you can perform Lazyplink on Kubernetes. Nodectl, Adokka Compatible CLI of continuity also supports Lazyplink, so you can perform Lazyplink on your terminal as well. This is a demo of Lazyplink of Hello World Estergizet image on continuity with Nodectl. The upper console shows Lazyplink of Estergizet and the lower shows normal non-Lazyplink. Estergizet enables continuity to run the container without waiting for poor completion, so it finishes in about 5 seconds. Non-Lazy image needs to wait for the poor completion, so it takes about 32 seconds. For Portman and Cryo, Estergizet store plugin enables Lazyplink. This is implemented as an additional layer store plugin available from Portman version 3.3.0 and Cryo version 1.22.0. Additional layers store allows Portman and Cryo to acquire continental image layers without pulling them from the registry. As shown in the following figure, Sstergizet store mounts extracted Estergizet layer on the host using fields. Each file contents are downloaded from the registry lazily. Sstergizet store communicates with Portman and Cryo via Firebase API and provides field mounted Estergizet layers. By using Cryo with this configuration, you can perform Lazyplink on Kubernetes. If you use Portman with Sstergizet store, you can perform Lazyplink on your terminal as well. This is a demo of Lazyplink of Hello World Estergizet image on Portman. The upper console shows Lazyplink of Estergizet and the lower shows normal non-Lazyplink. Portman finishes running Estergizet in about 5 seconds. Already exist message shows that Portman acquired image layers from the Sstergizet store. Non-Lazy image needs to wait for the pull completion, so it takes about 25 seconds. Buildkit experimentally supports Lazyplink of Estergizet formatted base images since version 0.8.0. In general, every build starts from preparing base images by from instructions of Docker file. When you build an image using large base images that aren't cast on the node, they need to be pulled there and it can take a long time. This Lazyplink feature is trying to solve it. You can use this feature via Docker Build X as shown in the command example in this slide. As shown in the following figure, builds run without waiting for the pull completion of from instruction. Necessary files or chunks of the image are pulled on demand during copy and run instruction. A variety of tools in the community start to support creating Estergizet images. Buildkit and Canonical support building Estergizet image. Buildkit supports building Estergizet as one of the compression modes. You can use this feature through Docker Build X. Compression equal Estergizet option enables this feature. Canonical supports building Estergizet. This is an image builder, runable in containers and Kubernetes. GZCR experiment Estergizet equal one environment-variable enables this feature. There are image converters as well. CTR remote is a CLI command for continuity. This is provided by StarGizet Snapshotter project. So you can fully customize and tune the Estergizet image, including prefetch optimization. NerdCTL is a Docker compatible CLI for continuity. NerdCTL image convert sub-command converts an image to Estergizet. NerdCTL is Docker compatible, so it has NerdCTL build sub-command. So you can build arbitrary Estergizet images using build sub-command and convert sub-command together. GoContina registry and plan also supports image conversion. This is a library and a CLI command to interact with registries. GZCR experiment Estergizet equal one environment-variable enables conversion from an image into Estergizet. Tools like canico, co, and buildbox.io use GoContina registry, so these tools also support Estergizet creation as well. As discussed in this talk, Estergizet is now widely usable on tools in the community. For keeping interoperability of Estergizet among these implementations, we are proposing Estergizet to OCR image specification. Because Estergizet is compatible to the current OCR image spec, we are proposing this as backward compatible extensions to the current OCR spec. Feel free to join the discussion. From the next slide, Tau introduces NIDUS image format. Thank you, Kohei. Hi, everyone. My name is Tau Pan. I'm a staff engineer at Ungroup. I'm going to introduce the NIDUS image service to you. The NIDUS image service is an image acceleration service that we built in-house at Ungroup and deployed largely on our production system. When we start building NIDUS, we make sure that we do support reasonably, but we still want more from just NIDUS. So here's the official list that we have in NIDUS. Firstly, we can support NIDUS with many prefaced policies. While we do poor images from standard registry, we want to make sure that even if the network goes down accidentally, we can still access it data locally. So we want to pre-fetch the data before, even if the application doesn't access it, because it may access it later on. So we support pre-fetched policies, and we have different pre-fetched policies, such as spantically or hintedly or background prefetching. Another thing we support is chunk-level deduplication with native built-cache. With chunk-level deduplication, we can increase the deduplicate ratio of a computer image, not very dramatically, and with native built-cache, it improves image conversion from a standard OCI image to NIDUS image very largely. And we support N2N data integrated. That means we can optionally verify the data contents when we read any data locally or from the registry to make sure that it is actually the data that is being built into the image. So the reason is that sometimes there may be a local disk break or there may be some bugs that accidentally change the local image cache. With OCI image, the application will never know it, but we want to provide the capability to find out such situations so that in NIDUS we can optionally verify it. And let us support the producible image building. That means we can build a NIDUS image from source on any machine. It will not be affected by the local machine environment. Also, we support the NIDUS image format is compatible with OCI distribution spec and artifact spec. So it is the reason that NIDUS is usable with a standard cloud registry. Also, NIDUS has a rich container ecosystem integration. It works with Kubernetes, Docker, Continuity, Buildkit, Haber, and Drinkfly and different container long-time such as transparency and cloud containers. It is very resource efficient and production ready. We have a very large-scale deployment at Ant Group and the service is available for our Alibaba cloud service. So this is the architecture of NIDUS. In the middle, we have a few streaming and it provides both Fuse Endpoint and VertiFS Endpoint. Fuse Endpoint allows the traditional density container to access the image data directly and VertiFS does the same for other containers. And it has shared uncompressed local cache. So the local cache is uncompressed when being pulled from the remote storage and it only needs to be uncompressed once and it will be shared by different NIDUS streamers on the same host. Also, we support single-time board with extremely low memory footprint. We have this and five megabytes per instance for each NIDUS streamer. And we can use Django Fly between a registry and the NIDUS streamers. Also, the image data can be distributed by a tp2p network. So this is the core of the NIDUS image service. We have RAPS registry acceleration file system there. It has a Markov 3 device. Each image into two parts. One Markov 3 metadata layer. It is self-verifiable by Markov 3 definition. And also it has chunk shared data layer. So different images can share the same data chunk on the data layer. Also, to be OCI compatible, we just changed one field in the OCI manifest. So we introduced a new metadata for the NIDUS probe. And the list of the manifest is compatible with the OCI artifacts and it is really big. So that means it is compatible with registry functional entities as well. And it is widely working with many container registers such as in many card registers. And it works with Docker registry and also reverse with Haber. And this is, there is a putting bunch back for, we have done for NIDUS. As we can see, for OCI image, the end-to-end time for OCI image creation, the time increases and the image size increases. But with NIDUS, the time is mostly consistent or below four seconds for different images. And NIDUS works with a variety of tools. We provide a NIDUS step-shorter plug-in for container D. So this is a link for the step-shorter. And with this plug-in, NIDUS works with many, many container-container-ecosystem components, such as Kubernetes. It works with CI console, it works with Nordic console engine, and it works with CI remote or CTR remote tool. Also, we provide tools to manually convert an OCI image to NIDUS image. The tool is called NIDUSify. And NIDUS, what it does is to download the image from remote registry and convert it to NIDUS format and then push it to a remote storage. So it's very easy to use, just call NIDUSify. Convert to density source and density target. And also we provide another tool. We provide built-kit support to manually convert or to manually build NIDUS image. So this is, we have a PR there for built-kit and it can build NIDUS image directly from Docker file and push it to a remote registry. We, you know, even with manual conversion, people will ask for automatic conversion. So we start to work with the hardware team to provide automatic support for NIDUS in the hardware project. We have designed there and we are working on the implementation. So the general idea is when user push an OCI V1 image to hardware, and the hardware will call an image conversion service to convert it to NIDUS. Also the image conversion service is image format aglostic. So right now we are working with this style. We co-hit together on the user GZ support for image conversion service as well. In future we may support more image format for the same component. And on the standardization point, we are working with the OCI community to push OCI artifacts manifest to the community. This functionality is to map an OCI image manifest to an artifact manifest. So we can connect NIDUS manifest to the original OCI image manifest. And it will, this will help the registry to check the relation between different images. And we are, inside the PR we are proposing NIDUS image as an artifact type. It's called cnsf.nidus.v1.dashrc1. This manifest is beneficial to its own signature NIDUS and scan relux etc. It is widely usable for different artifacts. Also as we, also we are discussing with the OCI community, proposing the v2 image spec. So we, there is a brainstorming on what is, what should the v2 OCI image spec look like. And in the discussion folks, the OCI community listed several key features such as reduced duplication, reduced reproducible image building, explicit facism objects and metadata, long-time data integrated, multiple facism format and laser fetch support. We've analyzed many open source projects out there that supports laser fetching or improves data distribution, the image data distribution, such as P2P Projector, Dragonfly and Kraken. All these projects lack, lack part of the requirements for the v2 support, v2 image format. But as we can see that NIDUS already supports both, all these requirements naturally. Okay, let's do some recap. We've introduced both StarGZ and NIDUS in this presentation. And the reason we do it is mutual. We are aware that OCI images are large and small. We need to do the image-less pooling to start containers quickly. And prefetching is a good friend to laser pooling because we can get data even before the container actually uses it. And we acknowledge that container ecosystem adoption is very crucial for this new image format. For StarGZ, its key feature is backward compatibility and it's an extension to the existing OCI image spec. For NIDUS, it's more future-looking and it's a proposal to the next generation of OCI image spec. And here are some information for StarGZ project. It is 100% OCI compatible image format for laser pooling. And it is a sub-project of CNCF-grangered project container D. Here is the ending for resource and snack page. Also, there are pre-reconverted images here. And for the NIDUS project, it is an image acceleration service with various improvements over the existing projects. And it is a CNCF-equipmenter project-granger-flexer project. And here are these links to it, its source, its stack, and tutorial. So thank you everyone for watching this video and for taking questions and we will answer later. Thank you.