 Hello everyone. Welcome to today's webinar talk. I'm Ji Cheng Shi, the CTO of Datanlord. Datanlord is a startup company focusing on the cross-cloud storage solutions. Today I'm glad to introduce a new metadata management system, beautiful geo-distributed environment named X-Line. Here is today's main contents. First, I will introduce the project's background, sky computing, and show how it changes the landscape of cloud-native environment. Then I will analyze the current cloud-native metadata management system, ETCD, and its limitation in geo-distributed environment. And finally, I will try to dive into today's topic, X-Line, the new metadata management system. So first, let's talk about sky computing. What is sky computing? It's a way to view computing as utilities and infrastructures, just like telephone service. Though we use phone service from different companies, we get the same service interface, the phone number, the phone card. We can easily migrate from one company service to another, while keeping the phone number. But what is the situation in computing area? We have cloud computing companies, which provide cloud platforms. They provide hundreds of services, and many of them have private APIs, which means the application developed for one platform cannot run on the other without any change. So sky computing is a way of breaking the barriers of clouds, trying to make cloud app be written once and run everywhere. However, people barely heard sky computing these years. The reason is simple, that every cloud computing company is trying to build a war, locking the users inside the private API garden for commercial reasons. It's not a technical reason. Cloud users are trying the way to leverage resources for multiple clouds or clusters. The trends are unstoppable. Fortunately, the situation is changing as computing itself is becoming more and more easier to migrate. For example, migrating a container is pulling an image. It takes a while. But migrating a serverless function is faster. We just need to upload a piece of code. However, compared to the computing, data moving is not that easy, especially considering data consistency. Cross-cloud data access usually means high latency and low throughput. With that base, it's hard to implement high-performed data consistency. Our open-source project X-Line is built to resolve the related issues, trying to bring the sky computing to the cloud storage. So let's first take a look of current cloud-native metadata management system, ETCD, which is the most popular solution. First, we will discuss the impact of sky computing to the distributed systems. Then we will show how we deploy an ETCD service in the multi-cluster scenario and analyze the disadvantages of these deployments. And finally, we will dive into the underlying wrapped consensus protocol and tell the root cause of the poor performance. Most distributed systems are developed for the single-cluster use cases. The message passing between servers is not a big deal, as it usually takes less than one millisecond to send a message. And the throughput is high enough in most cases. As a result, the network is usually not a bottleneck of the whole system. But the situation changes when sky computing comes. More and more cross-cloud communication appears, and usually it's unavoidable. Sending a message in that case usually takes up to hundreds of milliseconds, and the bandwidth is limited at the same time. The high latency and low throughput leads to the poor performance of the whole system. Now let's take a look of the popular ETCD service. Try to deploy the service in multiple clusters. Obviously, there are two methods. The first is straightforward. We just locate all the ETCD servers in one cluster. The good news is that requests in that cluster get the farthest response for that they are in cluster. The opposite side is requests from the other clusters suffer the long latency. Clients have to wait a long time to get the result. Besides the performance, this deployment cannot tolerate the cluster level failure. What is cluster level failure? That means the whole cluster server cannot respond to the requests from the outside. Even though the whole cluster servers fail, at the same time it's almost impossible. Misconfigured networks may isolate the cluster from the outside world. At the same time, we can't reach the service at all, even if it's built with high availability in mind. Unfortunately, we've seen that mistake many times in reality. The other way to deploy the ETCD service is to divide the servers into different clusters. We try to place at least one ETCD server in a cluster. It seems to be a perfect workaround. Each cluster has a local ETCD server. The client won't suffer the high latency. But at the same time, it can tolerate the cluster level failure, as minor server failure is not a big deal for ETCD. However, the performance of ETCD service itself drops quite a lot, almost unacceptable. So what's wrong? The answer is the server's internal message passing. Usually these messages are triggered by the underlying consensus protocol, RAFT, which is used for data consistency. Now it's a good chance to take a look at RAFT protocol. Then maybe we can find the answer to the performance drop. In that protocol, the client sends a message to the server's leader. Then the leader broadcasts the message to all the followers. The leader will wait for the response from the followers until enough OK responses are collected. Finally, the leader sends back the result to the client. The client can send a request to any server instead of the leader. But considering the followers will eventually redirect the request to the leader, let's keep the process this way, just for simplicity. To abstract the process, we gather the right graph. It's easy to see that to complete a RAFT protocol, there are two round trips. The first is between the client and the leader. And the second is between the leader and the followers. Both channels are cross-clusters. It suffers twice the high latency. Now we know EDCD cannot work well for the multi-cluster or geo-distributed scenarios. Can we find a way to work around? X-Line is our answer to this question. I will first show the architecture of X-Line and then analyze underlying Kerber protocol. To work with the new protocol, we also need to consider the client-side SDK and its compatibility. Finally, we will show the performance of X-Line with a test and a benchmark. In X-Line, when the client sends a request to the server, it wrote the request to the consensus protocol. Once the protocol completes the message thinking, the server will try to apply a command by checking all first and then applying the storage level change. This architecture is used for EDCD compatibility. If we want to achieve better performance, we need to move part of the consensus protocol to the client part. We will show that later. The green part in the graph is the main innovation of X-Line. That consensus protocol is called Kerber. Kerber is a geo-distributed friendly protocol. But how? Let's discuss it in a job-done way. First is the main idea behind the protocol. It's separating the data duration from the global ordering. Let's revisit to the roughed protocol we discussed earlier. In roughed, a committed requester has two properties. The first is that it will not get lost if the majority of the cluster servers are live. The second is the ordering of the request will not get lost or revised if the majority of the cluster servers are live. Do we need to keep them at the same time? The answer is no. Sometimes the second property ordering is not that important. Let's take KV as an example. If two requests are changing different keys' value, the ordering doesn't matter. We will always get the same final result. So exchanging order should be allowed in this case. But if the two requests conflict with each other, changing the same key, the order doesn't matter. For the previous scenario, the ordering property can be skipped. The next question is how do we know whether the request can be reordered or not? The answer is a speculative execution pool. Every server will check the incoming request. If it conflicts with the request in the pool, the server replies to the client telling there is a conflict. Otherwise, the request is inserted into the pool. When we hit conflicting, the backend protocol helps. All the requests will also be sent to the backend protocol. That protocol will broadcast the request to all servers and decide a global ordering. The backend protocol can be any consensus protocol, such as raft and multipack source. Here for simplicity, we choose raft. We can see the backend protocol is a second barrier, making sure every request has its own position in the global ordering eventually. To summarize the current protocol, it tries the request optimistically first. If it fails, let another consensus protocol handle the rest of the thing. Here, the consensus protocol is raft. From the above description, we can tell that the KERP protocol can complete the consensus in one trip if there are no conflicting requests. Even in the worst case, two round trips are enough. As a KERP protocol user, X-Line should provide additional information to tell whether two requests conflict with each other or not. If the users are using the ETCD service, they can switch to X-Line easily. Because X-Line provides ETCD compatible APIs, they are listed in the table. We have implemented KV, ORs, Lease, Watch APIs, and the others are in progress. The completed parts are the most used ones, so if you want to have taste, don't hesitate. We also provide two versions of client SDK if the user don't want to change code. The original ETCD SDK works well with X-Line. The only drawback of this option is that the performance is also the same as ETCD because two round trips. Message passing happens. If you are searching the best performance of X-Line, you should use X-Line SDK, whose programmer interface is very similar to the ETCD. With limited code change, your application should work with X-Line. In the X-Line SDK, the client side is also a part of a code protocol. It sends a request to all the known servers. So one round trip is enough for lucky requests. Though we know X-Line should outperform ETCD in multi-cluster scenarios, theoretically. We have to demonstrate it with tests and benchmarks. Here's how we build a benchmark environment. We simulated the multi-cluster's network latency, making inter-server communication latency 50 milliseconds. And at the same time, the client should not be in the same cluster of the servers. The communication latency between client and the servers is 50 milliseconds or 75 milliseconds. Actually, the latency is larger in reality. We test two cases. The left is the worst one, and the right is the regular one. In the worst case, all the clients send the request changing the same key. Kerb's speculative effusion pool can't handle any of them, so it works the same as Raft. Just like the result shows, X-Line's throughput is almost the same as ETCD. In the regular case, clients randomly pick a key from 100,000 key space. As a result, there's almost no key conflection. In this scenario, X-Line doubles the throughput as expected because the latency is halved. In the real world, it's almost impossible to make the worst case. Now we have demonstrated X-Line is still distributed friendly. Here is our roadmap. Last year, we implemented the major API of ETCD for X-Line. We plan to add more features this year, such as persistent storage support, snapshot support, and cluster membership changing support. Many more features are not listed here, but one thing is clear. In 2023, our main task is to make X-Line feature complete. And in the next year, we want to make system more robust, so we plan to bring in the Chaos Engineering, helping validate the system. At the same time, we want to have more and more cloud native components to integrate with X-Line, bringing sky computing to cloud natively, eventually. The right-up corner is our open source link. Welcome to join the community and contribute to it. That's all for today's talk. Thanks for your time.