 Okay, let's start. Good afternoon. First of all, I'd like to introduce myself. My name is Hu Zuozheng. I come from Alibaba. I'm the founder of Dragonfly. Can I ask you a question? Have you ever known Dragonfly? Or have you heard of Dragonfly? If you do, please raise your hand. Okay, a few of them have heard of Dragonfly. Okay, I will give you a comprehensive introduction of a Dragonfly so that you have a better understanding of this project. And today I'd like to talk about four things related to Dragonfly. Sorry for that technical glitch. Okay, sorry for that. Well, I will talk about Dragonfly. History, status, adoption, and roadmap. First, I will talk about history of Dragonfly at the milestones it have achieved. And next, I'd like to talk about Dragonfly community. It's a current architecture. And I will share with you two cases on how external companies using Dragonfly. And lastly, I'd like to talk about our future roadmap. What is Dragonfly? It is a P2P-based, highly reliable image distribution system at a larger scale. It aims to improve distribution efficiency and lower the bandwidth cost. And why we've developed Dragonfly with the expansion of Alibaba business? The deployment volume increases. Therefore, the software package to be distributed has increased too. And due to the bottlenecks in pulling images, our distribution efficiency is very low. So in order to address this bottleneck, we've used a P2P to increase the distribution efficiency of images and software package with more and more scenarios. We have been proving and adding new features, flexible dynamic compression, et cetera. And we are trying to build a Dragonfly into a comprehensive file and image distribution system for all scenarios. This is important milestones of Dragonfly. Dragonfly was born in June 2015. So the first version was released within the group in June 2015, mainly supporting P2P file distribution and CDN. In September 2015, that is three months later, we were able to support image distribution. We developed a local proxy. So we've downloaded images and put on a local client. In November 2016, Dragonfly covered the whole Alibaba group with 1.5 billion downloads every month. We finally became part of the infrastructure of Alibaba in November 2017, that is one year later, in order to address more problems and further improve distribution efficiency and lower bandwidth cost. We've developed automatic preheating, memory file systems and through dynamic compression, we have lowered the bandwidth cost, especially cross-region and cross-country bandwidth cost. And we've also became open source in November 2017. And in November 2017, we were the first to support single-stay sales campaign of Alibaba group. When we are shopping, all the sales data, you know, nearly 12 p.m., there will be GBs of data need to be pushed to tens of thousands of computers within 30 seconds. We've pushed the wrong data or the pushing failed. It will probably led to the fact that we couldn't shop during the single-stay. That means Dragonfly is a very highly reliable system. It can address various kinds of file and image distribution problems under all scenarios. And in November 2018, Dragonfly joined CNCF sandbox project. At the same time, we integrated with the harbor. I don't know whether you know about harbor. It provides image registry services. We are now integrated with the harbor. We developed with the harbor in developing automatic preheating function. We will sync it up with the node of Dragonfly. Currently, we've completed refactoring with a goal lamp. The version was already released. And we also support HTTPS image registry distribution. In the past, we could only support HTTP. And now we have over 50 adopters. So these are important milestones. Next, let me talk about status of Dragonfly and the overall architecture. Native shows using the native download method that is called pull without using Dragonfly. On the left, we have testing conditions. Two servers, which are Dragonfly servers, 24 and 64 G SSD and 1000 MBS. We've tested on 200 VMs. It's a 4 and AG 100 MB second. And on the right hand side, we have the concurrency downloads and average costs. The green one is native. And the orange one is Dragonfly. With the increase in downloads, you can see that the average cost of native has grown exponentially. Whereas if we use Dragonfly, the average cost remains flat in spite of the increase in number of downloads. And with the expansion of a scale, Dragonfly's advantages became more and more obvious. When reaching certain scale, using native will probably lead to explosion of registry. That means the image downloads would fail. Whereas Dragonfly could address all the problems faced by native. This is a Dragonfly community. We have 3,700 stars and over 50 adopters. These two indicators rank number one in terms of image distribution. And our adopters come from different background, cloud computing, e-commerce, financial services, telecom operators, live broadcasting platforms. We have 6 maintainers. Two are outside of the company. One from eBay, one from May 2. The other four are internal maintainers. And we have 32 contributors. And I hope that more of you could join us to maintain Dragonfly and to help it grow. And currently, we have discussion groups on Dingding and Gita. We also have WeChat discussion groups. Next, let me show you the P2P mechanism. Dragonfly has two main components, Supernote. You can regard it as a P2P Shattering Center. On every physical node, we will deploy a DFGAT, which is a client of Dragonfly. And DFGAT include two small components. One is a DFDemon, which is a smaller version of DFGAT. When we download images, doc code will move to DFGAT and then submit request to Supernote. And Supernote will check whether there is image file locally. Otherwise, you could download images from registry to local. It is synced up by block. When Supernote downloads a block, it will package block and generate an overall block. An overall block will be downloaded by client. If one client downloads a block, the block can be shared with other clients so as to achieve P2P distribution. We know that one image has many layers. When all packages on the single layer are downloaded, container will copy the file from DFGAT to local registry. And every layer is downloaded in this way. If all layers of images are downloaded, then the pod download will be completed. For Supernote, Supernote also supports remittance storage. If download only proceeds 50%, then when connection is ready, Supernote only needs to download the other 50% of the file. In this way, it will improve the download performance. Now let me look at the Supernote architecture. Supernote has several modules. On the top, we have an API gateway which provides REST API. The core modules include P2P scheduler, CDN manager, transmission control, pre-heater, and on the bottom we have storage manager. Next, I will give you detailed introduction of every module for P2P scheduler. It will determine what kind of blocks should client download and which kind of client you need to download blocks from. And what about the download speed? First, let's look at how Dragonfly should determine what kind of blocks should be downloaded by client. We use two strategies. One is sparseness, maximum sparseness principle. We would choose in the P2P the most scarce block to the requesting client so that different blocks in the P2P network can balance dynamically so that we can avoid the scarce block or the sparsity. So with the sparsity theory, we can choose blocks and then for the different blocks. For example, there are a lot but of course every time the operable block is limited, for example, we have two blocks and now the client can only download one block. So how do we choose one from the two? Actually, we use the back-to-back. What do you mean by that? So that means that when the client downloads the different blocks, now we know it's block eight and nine and the client can only choose one. Then we would choose nine because nine is the closest to 10. That means when the client downloads, it's like a sequential IO. We can do the driver IO so that we can have this sequential reading and writing so that we can improve the IO efficiency locally and improve the image distribution. Now the client has already chosen the blocks to download. So for the blocks, which clients should we download them from? So we have the proximity. That means we should choose the closest client that is on the network because the latency is smaller. Then we can reduce the download latency and also we can choose different clients. Then among the target clients, we need to do another screening and now we can use affinity. We can use affinity to define which client has better affinity to my client and also we have the work limiter to choose the lowest load client. For example, now we have two target clients A and B. For A, A is providing three blocks out and two is providing only two. So then we choose B because the load there is the lowest and then the speed to respond to the requesting client is faster. So that is the scheduler strategy. Now we have six strategies of course and they combine with each other so they can also impact each other. Then for CDN manager that includes the synchronizer. We have the multithread continuum so that we can synchronize the image. We have the breakpoint resume so we can generate the seed and then for catcher that is the CDN cache. So it includes the cache testing and also the cache dynamic GC and then compressor that is the dynamic compressor and we would dynamically decide whether this file should be compressed and also the compression strategy is block level. That means for the whole file we want to compress it but we would choose the most worthy part of the file to compress rather than the whole file. So with this kind of method the compression benefit is the largest and also after the compression it will be cached so we just need to compress once and use multiple times and of course according to the super node load we would decide whether to compress or not. Then we can avoid the impact on the super node normal scheduling and the compressing is also multithread and then transmission we have the rate limiter that includes the super node synchronization image and also the client and it's downloading speed for the blocks then the size limiter that is the size of the block for different files the size would defer and then data security it means for different block we would use the dragonfly designed format to pack them then we would avoid any manipulation during the transmission. The fourth module is the preheater of automation it is not just for container image it's also for general files and what do we do in preheating so when we build the image when we push every layer that we push we would have this lab hook request for super node and then that layer of file is synchronized on the cdn local cache so when we push after we pushed the image is synchronized to the local cache of cdn and when we apply and deploy when we pull the image then there's a hit in the cache so we don't need to synchronize once again from the network so that the image distribution efficiency is better so with these strategies dragonfly will try its best to improve the download efficiency and reduce the cost and the bottom layer is the storage manager we have different kinds of data to be stored for example the block data and the metadata the metadata for different block and also the mdu code and also the number and the files then for link data this is the soft linker data why soft because with soft linker we can go cross file system so we can access to different driver on the super node with the soft linker we can access to all of the drivers and the last one is statistics data it includes for example the time of distribution and also the success rate and for each node which client are they downloaded from and so on then it's more convenient for us to analyze the distribution efficiency and also we can provide some data support for the later improvement and on the bottom layer that is the memory file system we have the memory file system as well as the disk file system why do we have the memory file system because we want to solve the problem of large file distribution for example 10g 10 giga so it will occupy part of the IO resource then it will cause impact on the downloading of other files so we can use the memory file system to avoid this kind of situation and we can solve the large file resource issue just now we talked about the allocation or distribution principle and also the architecture i believe that you already have some wholesome view on dragonfly so let me introduce to you the use case first on an adebaba cloud in alibaba cloud there is this acr service the alibaba container registry service so let's take a look how they used dragonfly so it deploys the supernode config api and that component will maintain the basic configuration of supernode for example the list and also the rate control and also the hardware control and so on and then on each node there is this config watcher and periodically it would pull down the configuration and save it locally on different nodes and when the container applies pull then the pull is onto the dfdemon and then according to the config watcher it will pass it on to dfget client then the client according to the config it will request on the relevant supernode and then the process will be like described just now now for acr has this demo version with dragonfly and if you want you can try it and this is another example in zhejiang mobile how they use dragonfly so above there that is the decast system of zhejiang mobile they have this client in charge of uploading the image and triggering the image download command and down there the light blue one that is the dragonfly component so they would upload the image to harbor and push it to harbor and this is the backend storage after the push there is this distribute command to the api gateway then the api gateway would call the corner service kernel service and then it's pushed to the dock server dock server you can understand it like docdemon the api interface and then the docdemon would push it to df client the df client then will interact with the df master which is the supernode and then there's the pdp network and there's the p2p transmission so basically then the process will be just like how i described it and this is their data comparison the blue one is the deployed time average and the red one is after the use of dragonfly clearly you can see that after using dragonfly the application deployment time is largely reduced i think the efficiency improved by three times then let me introduce to you the future planning of dragonfly so i will talk about several things for example the cnf ecosystem cncf ecosystem we would cooperate or integrate deeper with harbor and also we want to support the client d and that is already being carried out then for security we will support the private container image registry and also that's already done and then in encryption we would introduce different encryption algorithm for the customers to choose we can ensure that the transmission is safer then for efficiency of distribution we also explored a lot especially in the intelligent flow control and also the intelligent scheduling algorithm and then in openness first we used the goal line to refactor supernode now it's a better integrated with the cncf and now in the new version we used this kind of plugin based architecture so that it's easier to scale according to the customer demand and then for scalability we would simplify the complexity of scaling of supernode so that with the increase of distribution we can automatically, flexibly scale supernode is already very stable right now i assume one supernode fails well it's nothing to use because all the clients of the supernode will migrate to another normal supernode for continuous downloading so it has high availability otherwise it might not become infrastructure of alibaba of course in order to further improve its stability we are exploring a solution that is decentralization it's not an entire decentralization we just want to weaken the role of a supernode further we minimize or further we try at least our purpose to minimize impact on user when the supernode fails in terms of scenarios we will support more cloud disks the entire alibaba group will move to cloud so back end storage is via cloud disk therefore we will support more cloud disks and in the future we're going to support different repositories which include ledges, maven, git, etc so this is our roadmap okay that's all i have to say and questions what is preheating as i've mentioned when you build image you need to push for every layer you push the image registry will generate a request to supernode after getting the request supernode will sync up the image to its local to locally therefore a supernode don't need to sync up image from remote registry it will save time and further lower bandwidth cost for registry what is the lab the progress regarding integration with paver now it's at the preheating stage paver has a registry when you push the image to hover hover will notify us the request and we will download the image to local supernode so unless when you push it will automatically process yes i'd like to ask you you you refactor supernode with gold land it's based on crt or you did it yourself crt we did ourselves will use gold land in refactoring at the very beginning a supernode was reaching in java and now we are open source and become a member of a cncf so we refactor supernode with a goal and yes almost all cncf project will use a crt we are different so you say that you're going to use disk for system supernode every supernode will use its own file system to store all the files so when you deploy it you will use the master's file system yes yes for question is that that if one supernode fails a df get will go to supernode for further downloading the supernode is a local storage but images if one supernode fails how can you how can you migrate to other supernodes if one supernode fails our client knows how many blocks this file has when i request a new supernode i know how many blocks i need to request for what about other supernodes how can they sync up to the failed supernode wow i'm not sure about your question your supernode fails the entire disk let's assume the disk's file system fails how other supernodes know about this information they don't need to know about their information these are client moves to new supernode you need to read register and the supernode will test whether you have local cache otherwise you need to sync up yeah they need to rethink again but client how how many blocks they need to uh or how many blocks they have downloaded they need to report it to new supernode in this way can supernode knows how to distribute the rest of blocks to the client another question the dragonfly doesn't support k a s yes it's an image container how about persistent storage it's a file system so we use the Ali Claus storage system we use the local storage we use the local file system we are planning to use the Ali Claus os s all the network file system to do that i have a question supernode