 Okay, so I think we can start. Quinton has left me no that he's going to be running a couple of minutes late, but he will be joining us shortly. So as a, just as a bit of context for the other people on the call. So in the last TOC meeting last week, the Dragonfly project presented their annual review and with the aim of potentially moving to incubation stage from sandbox. What we're looking to try and do here is to help the TOC, so the TOC has asked us to provide some help in reviewing the project. And although Project Dragonfly isn't a fully, you know, kind of like a natural storage project that was felt on the call, on the TOC call that it probably was more storage related than anything else. So that's kind of like the context of why we're looking at Dragonfly. So with that, Alan, who's one of the Dragonfly maintainers, is going to be presenting. So do you have a deck that you can share? Yeah, I can share my screen and I have the slides here. Perfect. And just for your knowledge, the session will be recorded and put on the six storage YouTube site. So we will use this as a, we will use this Q&A session as well when to send the TOC as part of this. Okay, I think it's okay for me here. Okay, let's start. Brilliant. And thank you everyone. Thank Alex for arrangement for the meeting of the six storage and Dragonfly team. Actually, almost everyone of Dragonfly team is online. And lots of time we do a Dragonfly incubation state review for the TOC meeting. And the TOC, they found that we need to talk to their secret storage. And that's exactly what is introduced by Alex as a context. And let me introduce myself. I'm Alan Sun from Alibaba Cloud. Actually, I'm a maintainer of Dragonfly for almost two years. And here's the Twitter name of Dragonfly. And we will publish every news of Dragonfly. And we cannot see that. And here is a very brief description of Dragonfly. And Dragonfly, it is born in Alibaba at about June 2015. At the first time, it is focused on the file and image distribution in Alibaba Group. At November of 2017, it is open source. Actually, at the time, Dragonfly has been a fundamental infrastructure in Alibaba Group. Every month, it will distribute about 3.4 pp data. And at about October last year, we tried to redesign the roadmap of Dragonfly. And actually, at that time, cloud native technology goes up very, very hot. So the aim of Dragonfly is becoming to be cloud native. Actually, at that time, we've already been adopted in production users. Lots of production users. The number is up to about more than 20. And we are very lucky to join the CNCF as a sandbox level. After joining the CNCF, Dragonfly has been adopted in a variety of industries. And in the following slides, we will talk about the adoption of Dragonfly. Actually, with the step of joining the CNCF, Dragonfly integrated with Kubernetes ecosystem software very easily. And it is a big help from the community. And we can later talk about the integration of Kubernetes, and Promsys, and Hover, and some other software. And last week, we've already done the CNCF annual review. In the future, we try to adopt Dragonfly to more cloud scenarios, like some IoT scenarios, like some other technology areas. And in two months, we will do the GA release. Everyone knows, at the first place of Dragonfly, it is written in three languages, Java, Python, and Golang. After about half a year's refactoring, Dragonfly has been now only in return in Golang. And in the next stage, we are hopefully we wish Dragonfly could enter the CNCF into a Beijing label. And the aim of Dragonfly is the image and the fail distribution system in cloud native. At the first, we try to tackle the image distribution issues in the Kubernetes, in the CNCF foundation. But actually, Dragonfly can be easily used to distribute fail, generic fails in your production environment. So just a couple of quick questions there. So would it be fair to say that the GA release is sort of scheduled for October time? October last year? No, you said the GA release is due in a couple of months. So it's in October of this year? Yeah, we will GA release will be October this year. Cool. And I might want to understand that Dragonfly is for all types of images, for all types of files, not just images or containers, right? It's basically anything. Yeah, basically anything. Cool. Thank you. Okay, thank you. And later we will talk about how to support the image distribution. It will introduce another component, but without this component, it is also can be used to distribute fail. And here's the features of Dragonfly. Dragonfly, the most important features of Dragonfly can focus on three keywords. The first one is efficiency. You know, we provide the P2P-based image distribution. The P2P features can improve the efficiency at your large scales when you scale growth and your image pooling, the network bottleneck of your image source or file source can exist. And the P2P will take the advantages of every node bandwidth, network bandwidth, so it could help to improve efficiency. In addition, it provides a passive CDM feature to avoid repetitive downloads. So the first one, the first part is efficiency. The second one is low control. Everyone knows that in the container world, every node, maybe every node will bear some application workload. Besides the workload, the image pooling workload still exists on the same node. And how to isolate from the application workload, for example, the disk IO and the network IO and the Dragonfly can intelligently regard this as an element to judge where and when to download the task and write the disk. So the task level and the host level net and the disk speed limit could be implemented in Dragonfly. And it will provide the disk from the high IO. The third part is the security. The security is that Supernode will cast the fails for the pure network. And the fail on the Supernode, we should try to encrypt it so we can transfer the secure data on the network. Only the source has low data and only the node has low data, so it can guarantee the security of it. And the fourth part, it's very easy and it's very simple and easy to use. Dragonfly is non-invasive support. All kinds of container technologies, the most popular one, the most popular one is Docker and another CNCF graduation project container D and also the Ponce container which is open sourced by Alibaba. These container engines can invasively use Dragonfly to pull images. So the users can pull container images as usual. This is the feature of Dragonfly. And here is a very brief introduction of Dragonfly's community. We have an incubation pull request on the GitHub of CNCF TOC repo. And everyone can leave your comments if you have any thoughts on Dragonfly. And last year, we already merged the sandbox pull request. Currently, we are very lucky to see Dragonfly project have more than 4,000 GitHub stars. And up to now, we have 41 direct contributors of the ecosystem. We released 11 formal releases and the GA release will be at October. And for the maintenance part, we have seven maintenance across four independent companies. And the initial company is Alibaba Group. And currently, there are three more companies which produce maintenance. These three companies are eBay. eBay they are using Dragonfly to distribute images. And May 2. And White Dance. Both the two companies are in China. And Dragonfly has more than 50 public adopters which will be introduced later. And we did not include the private ones. We have more than 300 members in the team of developer groups and the CI best practice passing. If you are not familiar with Dragonfly, here is a very brief workflow of Dragonfly to illustrate that. We can see the right part of these slides. We have a super node. Super node. We can regard it as a control part of the Dragonfly system. It will try to cast images from the registry. It will try to cast the files from the source. And it will schedule the downloading request among the peer networks. Every node, every node, we have a container engine. Every container engine, besides the container engine, we have agent which is a DF daemon. DF daemon is used to intercept the image pooling request. And we are restricted request to send it to the DF gate. DF gate is a generic agent on the node for the file distribution among the peer network. So the component of Dragonfly could consist of three parts. The most important one is super node to control, to manage the system. And the DF gate is a generic proxy agent on the node. And the DF daemon is used to intercept the image pooling request. So these three components. Here is a very brief workflow. We can regard it as six steps. The first one is that the container engine wishes to pull image from the DF gate. And it will send a request to the DF daemon. The DF daemon will intercept the request and send another one to the DF gate. And the DF gate will send the pooling request to the super node. Oh, please remember that the P2P network wishes that all the nodes at the same time trigger the same image pooling request at the same time. At the second part, the DF gate will send the pooling request to super node. The super node will check if the image has already existed in the cache. If it's not, it will try to use the CDN feature to cache the image from the outer side image registry like Harbor. After it's finished to cache the images, it will try to reply to the requesting node. I have already had the file. Then the host, the agent will know, oh, the super node has already cached the images. And it will try to get one block of the image from the super node to the node itself. And for the other peers, it will do the same thing. Then afterwards, and the whole network, the whole P2P network will finish the whole pooling when other blocks have downloaded it to each node. So it is a very brief workflow of Dragonfly. And here, when we are a little bit familiar with the Dragonfly's workflow, here is the Dragonfly's architecture. Hi. Sorry to interrupt. Just a quick question. So if I understand this correctly, effectively, the Docker daemon or the patch D is pulling through the various proxies to effectively get the images. Is it just Docker or patch D? I think you mentioned container D as well. Is that supported too? Yeah, of course. Container D is supported as well. Here, we have an ecosystem integration. This picture, we have that container D can be supported. It is very easily. So we can configure the proxy configuration of the container D to make it point to the daemon address. Then the daemon can get the image pulling request and it will intercept it to the dfget. Then if the request has been sent to the dfget, it will be a generic file pulling request. Understood. And the top level registry, so in these diagrams, it's shown as being Harbor, but I'm assuming it doesn't have to be Harbor. It could be any registry, right? Yeah. It could be any registry. Yeah. Cool. All right. Thank you. Okay. Thank you. I have a question. Does the C&Caching service operate at the granularity of files or blocks? All right. Just a moment. Yeah. I'm wondering if I understand your question clearly. You mean that the supernode, which policy is used in the supernode to cast images, block or the files, right? So, yeah, like, does the caching system, does it cache images at the granularity of files or is it actually does it integrate to blocks and blocks can be shared in multiple images so you get more savings? Okay. Okay. Actually, supernode is only used to cache files. So, image can be regarded as one specific kind of file. I see. Okay. Thank you. Thank you. Okay. So, if anyone have questions, have sought some Dragonfly, you can feel free to to interrupt me to ask the questions. So, here's the architecture of Dragonfly. Actually, we have a cluster. In this cluster, we can deploy supernode in a high availability mode. So, in this picture, you can see two supernodes. And the supernode provides the CDN features and the P2P scheduling features. Under the supernode, you can see the agent will construct a peer network to pose the files at the same time. So, we can see a P2P network. So, this is a very brief architecture of Dragonfly. And we can talk about it later. And here is a much more specific architecture of a Dragonfly supernode. Actually, supernode provides the API abilities for callers to control which kind of work to do. And above the API, we can see the P2P schedulers. In P2P schedulers, we provide lots of scheduling policies. These policies will be introduced later. Here's the sparseness, the proximity, back-to-back affinity, and the stiff isolation. Maybe you are not familiar with that, but we can share that later. The CDN manager. The CDN manager will do three parts. And it will download the files from the source from the registry. And in addition, it will cache the things. It will try to compress the file to improve the efficiency, to reduce the storage size. And it will also encrypt the data, encrypt the files. Besides the CDN manager, here is the transmission part. The transmission part will try to try to config the rate limit of each task, pulling task. It will control the uploading tasks on the supernode. Because lots of requests are coming from the agent to pulling requests. But the supernode still needs to guarantee it can provide the fixed ability for other services. And it provides the set determination for the files. When every task comes to the supernode, supernode will try to cut the file into many of blocks. But what's the size of the block? It should be it is decided by this part. And it will encrypt the data as well. For the preheater part, preheater part is used for the fail prevention or preheating. Just takes the harbor integration as an example. When harbor sends a request to the supernode's preheat API, the supernode will will reflect or will react to download the files from the harbor, download the images from the harbor to catch them into the CDN manager. Then when every node tries to send the same image pulling request to the supernode, there is no need for the supernode to download the file or images from our sources. It can directly to to to to schedule the image pulling request among the peer networks. So it can reduce some time for the users. This is the usage of the preheater. And at the bottom of this picture is the storage manager. Here is some element which we needed to store. The first one is block data. Block, it's very easy because in P2P network, we try to distribute smaller blocks. So for a single file, if it is very large, so we need to divide it, divide it, we to cut it into many blocks. The block data should be should be stored and should be managed by the storage manager. And the metadata, metadata is some kind of assistant data, like the MD5 data, like the checksum or something like that. And the file link data, file link data is, yeah, someone has a question. I was just going to ask a quick question there. That block data, how is it determined to be sort of in sync between the different P2P nodes? Do the blocks have some sort of have some sort of checksums and do something like a merkle tree or something like that to diff the files between the nodes? Okay. So you mean what's the policy to judge the size of the blocks, right? No, not the policy to judge the size of the blocks. So the question is more along the lines of how do you compare, how do you determine if the different P2P nodes actually have the correct file or if the file has any changes? Are the blocks checked some in some way? Can they be compared between the source and the destination? Okay. So different nodes, if the different nodes want to pose the same file, but the file have changed, right? So what's the... So for example, just as an example, if a P2P node crashes, for example, while it's in the middle of downloading some blocks, how does it know whether it's got valid blocks or corrupted blocks or something like that, for example? Okay. Okay. Okay. Okay. If a node crashes, we provide some policies to make the mechanism still work. If a node crashes and there will be some mistake in the cluster, there will be some mistake data in the cluster. The first one is that if the node has already downloaded some blocks, these blocks cannot provide the uploading services for other agents. This is the first part. The second part is that the blocks, the data, they have already existed in the agent, but the super node has a scheduling data of these blocks. These blocks should be corrected. Then for the second part, the data will be corrected because when the node crashes, then the hard bit of the super node and the super node and the agent fails, then it will... The super node will try to regard the agent as a failure one, then it will map the data, the blocks that exist. This node should be mapped as unavailable. As a piece, we will try to... As a super node, we will try to schedule another node which has corresponding blocks on the value node for other agents to serve. Yes, this is the first part. This is the second part. The first part is that when it fails, another peer tries to access the block data from this node and it will fail. It will fail when the failure times will increase up to the fixed number. It will try to report to the situation to the super node. Then the super node will reschedule this pulling image block task for this node. This is for the first part. Okay, so that seems like a fairly sophisticated retry and rescheduling policy for when there's a crash, but ultimately, is the data on disk protected by some sort of checksum or some hash or something so that you know that you've got the correct data and it hasn't been sort of tampered or it hasn't been corrupted in some way? Okay. Yes, actually, in our plan, in our roadmap, we will try to make the... Actually, this is our mechanism. When we use a CDM feature to catch a fail, when we have finished downloading the fail, we will try to encrypt it. Then after we encrypted this fail, we will cut it into blocks. So when we try to distribute the blocks, actually, the block is one part of the encrypted fail, right? Then when we distribute it to the node, actually, when the agent, when the node receives the block, the block is also encrypted. When the node will try to reorg the fail, it will reorg all the blocks into encrypted fail. Then it will try to unencrypted this fail. I see. I see. Okay, so effectively, the encryption is used as a way of validating that the file is correct? Yeah, I think so. All right. Well, I think if you decrypt a corrupted encrypted fail, you just get a decrypted fail. It doesn't tell you it's corrupted necessarily. So I think we still... If I understand Alex's question, I believe, and I don't think it's been answered yet, so maybe we can take that offline. I had another question, and my apologies, I joined the call late. There's talk of a peer-to-peer network, but then it sounds like everything actually gets downloaded through the supernodes. Is that right? So does all... If a client is downloading an image, is all the data flowing through the supernode? Actually, no. Actually, not all the data is from the supernode. When the node tries to... For example, when 10 nodes want to download the same image, then the supernode will cast an image, and it will first schedule one block to one node, then it will record that one block... In the peer network, there are already two nodes that has a block, the node one and the supernode, and it will try to schedule another node pulling requests to download the block from the node one. Then after seeing this mechanism, it will try to use the peer network to distribute the blocks. Okay, so the clients ultimately pull the data off the P2P network nodes, not off the supernode. Yeah, yeah, yeah. Okay, thank you. That's correct. Okay. And here, at first, an Alex introduced maybe Dragonfly... Maybe Sieg storage is the best Sieg group for to relate to that we use Dragonfly. I had some slides to introduce some storage part of Dragonfly. We can divide into... Dragonfly's ability into two parts, the CDN part and the P2P part. The CDN part, first, we will try to download the file, and after we download it, we try to transform it. In this part, in this phase, we will compress the file and encrypt it. The compressed phase is used to improve the distribution efficiency, and the encrypt part is to guarantee security. After we transformed it, we try to stop it. When we stop it, we will try to cut the files into blocks. And we will try to use our encapsulation protocol to add some headers for each block. In addition, we provide a storage interface, and we can try to start the blocks in local file system, or some in memory file system, or some third-part storage services. This is the CDN part, and I try to introduce some storage-related part with us. The second part is the P2P phase. The P2P phase, the first one, is how to transport, and then it can consist of three steps. The first one is how to construct the P2P network, how to emit the P2P network, and every node we will try to reduce it as a peer in the Supreme node. The second part is that when lots of nodes wish to pull the same request, pull the same image in the Supreme node, how to schedule. Actually, for the questions, I have already introduced some details about the scheduling system. When the scheduling result exists and produces the node, we will try to implement, try to execute the downloading, the detailed downloading parts for the blocks. And this is the transport part. The second is that how to do the IO control. IO control can be taken affected onto two nodes, two kinds of nodes. The Supreme node will do the IO control, and the agent will be taken in fact as well. The second part is the data reorg. The data reorg, the first part is when all the blocks has already been received on the node, we should try to make a block, try to make the blocks right on the disk. And this part, we will use some algorithm to improve the efficiency. For example, because every block has its own position of the fail. So when we write the blocks into the disk, we try to find some blocks nearby. And we try to write a bunch of blocks who are nearby into the disk. We try to make the disk writing to reduce some time cost. And we will try to check the fail. And the fail check part will consist of two parts. The first one is the block checking. When the block is not correct, it's not correct. Then the whole downloading task could be regarded as a failure. So we can return fast so we can adopt the whole downloading task. When the block validation is successful, then we will try to check in the fail. After the fail, checking is okay. Then the fail downloading is okay. For the docker part, we will try to make the image fail as a stream to feedback to the container engine. And for the fail, it is already a complete fail, which has already been distributed to the node valve PPP. And here is some procedure, some detailed description here. The procedure is in CDN. The first one is download. We can support the HTTP and FTP. For the compressed part, we provide such a kind of mechanism. The super node, we will try to do a sampling from the downloaded fail. It will cut a sample from the fail. And for the sampling fail, it will do the compression work. If the compression ratio is less than 60%, it means that it can guarantee the efficiency if we do the compress. So the super node will do the whole fail compression. And we provide the GZ and LZ4 policy tool to do the compression work. And the third one is the encryption part. The first one is we use DES by default and then make the cache fails encrypted on the super node. For the storage cut, our policy is that super node will cast blocks in range among one megabyte and 15 megabytes. At the bottom of the blocks, we have a picture. On the second part, it shows four bits. Four bits is used to identify the block size of the block. It's a structure of the block's organization. In the middle, it is block data. On the left side of the block data, it has 24 bits to identify the block data size and three bits to be reserved. And one bit to tear others, if it is compressed. And the four bits is used to identify the block size. And this is a very simple encapsulation protocol of Dragonfly's block. I got two questions. The first one is how do you version this scheme? The second question is, you give 24 bits for the block data size, but you only give four bits for the total size. It looks like 24 bits can express more than 15 megabytes, I believe. Okay. I understand your second question. My first question is, how do you evolve? This is a protocol, I believe. What if you want to change it? You may want to migrate from V1 to V2. There has to be a versioned bit, I guess. Okay. For the first question, you mean the API version of Dragonfly's Supernode? So it looks like you're giving the block data encapsulation protocol. I specifically mean the version of this schema. So encapsulation protocol's version schema, if we have some upgrade from version one to version two? Great. If you want to add some new fields, new bits, there might be a future version. That's all possible, I believe. Actually, we have considered these features into the design. You see three bits to be reserved. Actually, we can try to take advantage of these three bits to be compatible with all the versions. This is what we decide. But actually, we currently provide only such kind of a protocol. We have never done some protocol upgrades, greater than now, currently. This is the first question. I think the figure is not the full protocol. It's only the payload. The payload, whether it is compressed or not, how big is it, and the ending. So it's not the full protocol. So I mean, maybe not all the information is here, but I've got a few questions to try and understand how you would actually store this information because just having compressed on and off probably isn't enough to tell you if you're, say, compressing with two different algorithms like GZip or Z4. But also, you know, the S seems like quite an old encryption standards. And what if you wanted to use a different encryption algorithm? How would you switch to that if you don't store the algorithm type? Okay. Okay. You mean how to change or diversion, for example, you know? Okay. I'm wondering if I understand your question clearly. You mean... So the question is this, without storing some sort of version information, assuming the version has a link to specific compression types or specific encryption types, you can't really change any of those algorithms. And it would seem to me that, for example, because you use DES for encryption, which, to be honest, is fairly old. It's about 20 years out of date at this stage. If somebody wanted to switch to something else like AES, for example, I don't see how you would be able to do this without rewriting on this format. Okay. Okay. Actually, yeah, what you mentioned is correct. Actually, we haven't taken that into consideration in our design. We assume that the users will try to fix the two compression algorithms. So this is a good question. Later, we will think about it and then we will try to find a solution to solve it. Thank you for the question. Yeah. Okay. Okay. Any questions here? Okay. Let's go next. And here is the P2P part. And the procedures in P2P, the first one is the scheduling part. And we have provided some policies here. And the first one is sparseness. And the supernote that we are scheduling the most sparse block among the P2P network first. It means that if we have block one, we have already catch the first into many blocks. In these blocks, we have a block one. And after sometimes distribution, actually the block one block is only existing on node one. Then we can regard it as the most sparse block. This block one could be regarded as the most sparse block. Then the supernote will schedule these blocks first because we try to make the blocks leverage the two among the P2P network. Then on a node, it will have more possibilities to have the complete blocks of the fails. It will ask the fails to reduce more time cost. And the second policy is affinity. The affinity part, we can understand it like this. The supernote will choose the best network condition node to schedule the block. For example, 10 node will try to pull the block from the supernote. But the network condition among the 10 nodes, they are very different. And the supernote will try to find the best network condition node. And it will schedule the blocks to this node. And the third one is CF isolation policy. It means that never to schedule one node if it's already filled to schedule once. Actually once or twice the time can be configured. So if the network issues, the network fails, the network issues, the network fails, or this node crashes, like I just mentioned in one question. So node will do the CF isolation and the supernote will not schedule this node. And the workload control. The workload control is very easy. The supernote will limit the parallel piece task capacity dynamically. If the supernote's workloader is very large, then it will dynamically limit the task capacity. And for the agent, it also can control the numbers of task callings. But it's not related to the supernote scheduling. The second part is downloading. For the downloading part, it is regarding, it is related to the CF get with the agent. The agent can dynamically adjust the downloading options. It can dynamically configure the calling, the image downloading timeout. Some cases, block downloading should be filled after several timeouts. But it can dynamically to dynamically configure the timeout to be a larger number than it can still work. Because sometimes the network could be under some poor network conditions. And the IO control, we can provide the net IO bandwidth limit. And we can limit the disk IO. For the disk right part, the first one is that we provide the position block offset. Just like what I mentioned just now, when a file is finished, where all the blocks of a file are finished downloading from the pure network to a node. The agent, the CF get, should try to combine all the blocks into one node. But actually every block has their fixed position, has its own offset of the file. Then we should try to find the position of the block offset and combine the nearby blocks together. And try to make them together to be returned into the disk. This kind of action could reduce the time cost when we combine the blocks into a file. And it can schedule block write sequence to reduce disk write time cost. And the fail check. The fail check part is that after agent finish to downloading one block, it will try to validate the check the block. If it fails, it can fail fast. There's no need to download other blocks. And after we combine all the blocks into one single file, we should try to check the files and we should check the integrity of the file. And this the P2P part and some storage related details. Thank you. This is really useful. Is there a lot left to the presentation? Pardon? Is there a lot left for the presentation? A lot left. Are there many more slides? Most slides. Yes, we have a lot more slides. Okay, because I think we're running out of time, so we might need to... Only a few slides left. Only a few slides left. Okay, we're holding three slides. Okay, excellent. Okay, and this part is talking about the Dragonfly's cloud native ecosystem. Actually, Dragonfly has succeeded in integrating Dragonfly with some cloud native projects. The first one is the promise use. The supernode provides the metrics API for the promise use to collect metrics. And it can send the metrics to the graph node to the display work. And the second one is container D. Actually, we all know that container D is one CNCF graduation project, which is a container engine. And actually, Dragonfly can be easily to integrate with the container engine. The third part is the harbor. Harbor, with Dragonfly, can collaborate in image pre-feature, and it can improve the efficiency when all the nodes try to pull the images. And the Dragonfly supernode can be installed as the user home chat. Actually, the nodes agent of the dev get could be deployed as a Kubernetes demon set. So this is the integration of Dragonfly with cloud native ecosystem. And here is the industry adoptions. Actually, we are very lucky to see lots of variety of industries they are using Dragonfly in their production environment. The first one is the telecom and communication industry. We have China Mobile, China Unicom, Huawei, and ZTE, they are using Dragonfly. And for the second part, e-commerce, e-commerce companies, they usually, they have large-scale clusters, and they are taking advantage of container. They have large-scale, say, we're easily met the distribution issues. So Taobao.com in China, and JD.com in China, and Shopee, this company is in Singapore of East Asia, and the Lazida Groups, they are also using Dragonfly to make it. The cloud service providers, and Alibaba, DaoCloud, TsaiCloud, and West 2C, they are using Dragonfly to provide distribution services for their customers. For some live streaming companies, Huawei.com, Billy Billy, they are using it. Meituan, Erlema, DD, they are also very famous public live service providing vendors in China. So actually, lots of companies in China, they will match the image distribution issues, and of course Dragonfly is their first choice. And the last one is artificial intelligence company, iFlytex. They are a very famous speech intelligence companies in China. They also use Dragonfly to distribute very, very large images. Usually, their images is larger than 20 gigabytes. And here is the roadmap of Dragonfly. First is the feature. We will try to make Supernode HA to be more mature. And actually, you will see Dragonfly, to deploy Dragonfly, we still need to deploy Supernode or two Supernodes to provide the HA. But how about decentralized scheduling? How about makes a scheduling ability to exist in each node? This is eBase concept. eBase, they are providing the P2P distribution abilities without deploying a Supernode. We will say we will try to make every agent to record the scheduling system and try to make the agent see of governing the distribution data. And the eBase maintainer is to design this part and try to make it. And some flexible planning framework, enhanced encryption, more file transfer protocols. And in addition, we try to, we hope we could provide a very good Dragonfly UI to be user friendly. For the sceneries, actually, we currently, we provide the Dragonfly in some online business physical machines. But actually, we found that we should do some optimizations when our environment changes from the physical machine to cloud disk. So the IO condition and the network condition can be totally different from the physical machine. We will try to do the performance optimization. And actually, we got lots of feature demand from the IoT sceneries. Lots of devices, they try to distribute the data actually distribution is the most important, most challenging part of the affineries. And we provide more computing at architecture. For the ecosystem, actually now, Dragonfly's ability is not so good. So we will try to support tracing in Dragonfly. And we will try to make the operation of Dragonfly in Kubernetes ecosystem very easily. So in our loader map, operator support is one part. In addition, I'm sorry to interrupt. I think we're out of time now. Alex, I was wondering, have we got any tech lead volunteers to lead the due diligence for this? We haven't discussed this yet, but we could, we could discuss that line. Okay. Actually, actually, we are wishing to get some feedback from the six storage and Li Xiang won TOC from the TOC members. And he is willing to do some due to the given diligence of Dragonfly. So I think, and Quinton, please feel free to interject here. I think we'll, some of the next steps are we'll collate some of the information and any other questions that we have. But we're also, we'll also need to probably speak to two or three of the users of Dragonfly to sort of better understand how they're using it in production and their use cases and that sort of thing. Because that's required for the due diligence. Anything else that is important for the next steps, Quinton? No, no, I just, I just wanted to, I guess, and I could clear that we're sort of on the hook for a deep dive into all of this stuff, including the users, but you know, set it up, test it, look at all the details. And, and yeah, we sort of need to, need to get that started soonish. Yes, definitely. So I tell you what, we'll, we'll, we'll start you doing this. We'll start you doing this over email if that's okay, Ale. Sounds good. All right. Thank you so much for the, thank you so much for the presentation and we'll, we'll get back to you shortly. Okay. Thank you, Alex. Thank you, Quinton. All right. Okay. We used to get your feedback so yes, thank you. Perfect. Thank you very much. Okay. See you. Bye-bye. Bye-bye.