 Hello, thanks to everyone for joining me. I'm so glad to be here and share with you a way to inject force in the HTTP messages. You can see my subtitle, no investment to applications means we do not need to change any code of our applications. We can inject our car HTTP force in the entire applications are running. Before getting started, I'd like to briefly introduce myself. My name is Chen Xi Li. I am a car developer of Cosmash and Pincap. During the last year, I was working on improving stability and the recoverability of Cosmash and enhancing tools to debug and foster recover faults. Moreover, I also worked on implementing and improving the HTTP chaos and IO chaos. So experience give me a fresh look at chaos engineering and Cosmash. So I'm very glad to share it with you. Okay, let's start today. I'm going to talk about foreign contents. I will introduce the Cosmash projection and show the implementation details of HTTP chaos and show the usage of HTTP chaos. And finally, I will talk about more features about the TRS support and the parking system. Well, does everyone know what is the chaos engineering? In my understanding, chaos nearing means we should inject arrow on a system and emulate the incident in order to build confidence in the system's compatibility to withstand turbulent conditions in production. But how could we emulate the incident environment? How could we inject arrows into a running system? This is the main program, chaos engineering going to resolve. There are many different tools you may need for a chaos experiment. You need a stress or BBC to inject a system call. You need IP tables or IP set to emulate, to emulate network partition. You need to understand the traffic control system of Linux and use TBF or net emulates and there are other tools to emulate the bottom wise and the network latency. You may need fields to inject fast system arrow and you need to use stress energy to burn your CPU or memory. You need to inject in different layer and different aspects. So you need a lot of different tools. You should also be careful with them because your applications may run on cloud but most of these tools work on bare metal. If you have tried to use these tools manually, I bet that before you fell back through the chaos, your brain will get chaos and burn. So if you are experiencing the difficulty to use chaos engineering tools in your situation or if you are willing to give it a try but haven't decided where to start, fortunately, chaos mesh can do the work. Chaos mesh is a cloud native chaos engineering platform with a powerful chaos to a kids and friendly in the face to use and the program. These highlighted words are three key features of chaos mesh and are also the design targets of chaos mesh. First story, let's talk about the cloud native. Cloud native is the most career feature of them. Our software will run on the cloud and we also should test them on cloud. Chaos mesh highly depend on Kubernetes. It runs in Kubernetes cluster and its components are scheduled by Kubernetes. Secondary, all the chaos experiments of chaos mesh are port-wise or container-wise. We will not affect other services running on the same machine with a target port or container as factory. This feature brings us a lot of the quality in developed as the isolation ability of the last processes is not enough. So there has been nine spaces and screws. And chaos mesh can be deployment with scale which makes the installation very fast and easy. You can see just one line is just script. You can install chaos mesh. And finally, chaos mesh is also featured with friendly interface and the chaos experiment is managed as custom resources, which means if you are familiar with Kubernetes, you do not need any other technologies to create or cancel chaos experiment. But in the face of chaos mesh with other resources such as port and deployment, you can create, delete, missed or edit them by the control. Also, you can use the programmable way like the go client and the draw client of the Kubernetes. You have too many ways to handle them. And there is much more friendly interface is our dashboard, building dashboard. Which is another as the sport. Yes, it provided more features for chaos mesh. You can just use the graphics interface and mouse and keyboard to create experiment. As chaos engineering platform, I briefed the chaos to a case as a basis of these competitive jobs. So here come with the third key feature of chaos mesh, powerful toolkit. It's always on our first priority to craft new diamonds and enrich our weaponry to break your system. Here, at least several different chaos tools provided by chaos mesh. We designed this tool to emulate a different situation with these tools. You will be able to during much incidents before running production, such as during for lost network connection or kernel fault. Now, let's focus on our main topic today. Yes, do you know what at least as a podcast is a function to cure of for port and network chaos into delay loss network. And let's talk about the HTTP chaos, the abort delay patch and replace functions. Why? So why we need HTTP chaos and how it is implemented and how do you use it? Let's see. The hypertext transfer protocol HTTP is one of the most popular application-level protocol, especially web applications and cognitive applications. As developer of cognitive application and a human in the network society, I use the HTTP or most in any time I stay as screen for I use a Twitter, YouTube or Google in my play time and use GitHub communities in my work time. I believe many is the same, almost the same with me. Once you used web applications, you must have met 30 responses. Actually, force widely existed in HTTP messages, some of them I expected and others aren't. Once an unexpected force occur, both human and the programs are likely to have no idea. Our program may also not handle it accurately even when meeting an expected force. Some distributed applications are designed to tolerate photo force on path of nodes and the fault tolerance need to test. To ensure the home system works as expected when receiving requests or responses, we create the HTTP chaos to hijack and modify our HTTP messages. You can see the picture. We pray the man in the middle and you both the client and the server and we will inspect the messages and do some actions, some evil actions. Before we get started to share the implementation details we should talk about technology base. The forwarding contents will be related to some terms of Linux, then work Linux, then space and HTTPS. If I'm not clear enough or if you have any question, please ask in the chat window and I will try my best to answer after the presentation. We need enough network like bridge, routes or IP tables, IP tables and the net Netspace and the HTTP and the TRS support. Yes. First let's look at the net Netspace of target process. To hijack the HTTP messages we need to run a tool named Calcity Proxy in the net Netspace of the target process as help of NS Enter. So Calcity Proxy will reconstruct the network pathology of target network Netspace. You can see the picture. This is the net Netspace of target process and the NS Enter runs in it and the folks of Calcity Proxy sub-program, sub-process is origin network environment of target net Netspace. So in bone and out bone network packages are passed through the device in the S33. In the new network environment all network packages passed through the device in the S33 will be redirected by the bridge. You can see this is the origin network environment. Packages are passed through the device in the S33's directory and this is the reconstructed network environment. All packages will be redirected to the VTH one. Now let's focus on the new network environment. We create a sub-network Netspace and two VTH pairs and send all network packages to the VTH one and in the sub-network Netspace the bridge two receive packages from VTH two and send packages to VTH three. In such a sub-network Netspace all packages are passed on data link layer. We can pass them on network layer again by the EP tables legacy. After we pass them on network layer again we can run a special transparent proxy which is spotted by the EP tables extensions named tproxy here to hijack and modify all network Netsp packages. You may think the network topology is too complicated could we have a simpler network environment? As we all know more complicated may cause more bugs. Yes, why we need the sub-network Netspace could we run transparent proxies generally? The topology above seem to also work. See the picture? Actually a transparent proxy run in origin network Netspace can also hijack all HTTP messages. However, there are some drawbacks in this solution. Following is the drawbacks of topology without sub-network Netspace. The first, the TCP ports of clients or service inside may change. And the second, the IP addresses of clients or service outside change. And third, outbound traffic cannot be transparent. So first and second drawbacks may cause existing connection interrupted and cause different behavior of clients or service. For example, if a server inside only allow requests from specific IP address, the request set by proxy may be rejected. And the third drawback means all outbound packages have to change IP addresses apart because all output packages will be redirected to input chain during this redirection or months in the before the output chain will be creamed and we cannot distinguish if the package has been hijacked. Now, our proxy can receive the requesties and returns the response naturally. And we let's talk about the inner works of Tproxy, yeah. So Tproxy consume requesties, send requesties to target server and returns responses to clients. Before it send requesties and return responses, you will select them to a practice according to the rules. There are six steps in each rule of HTTP calls. Select request and apply actions and send request then receive response from the server and select response then apply actions. Now, let's try to write and apply a rule. Let's study. First story, we will try to abort request or response. All examples are written in format of Cosmash CRD in Kubernetes. You can see the picture is a configuration. After this config being applied, all requestsies send to the 80 port by gas method we abort before being sent to target engine server. You can see the package request applied is 80 and method is gas. Pass is a wire card means any pass and abort is true. Okay, you can look at this picture for a while. Next, we will try to delay request or response for duration. So duration can be human readable. You can see this configuration. It removes the abort field and set the delay field to one second compared to the abort example. After this config being applied, all requestsies send to 80 ports by gas method will be delayed for one second before being sent to the target engine server. Then we will try to patch the body of request or response. In this example, the body will be patched by the provided JSON following the JSON merge patch of C. After this config being applied, body of requesties send to the 80 ports by gas method will be patched with this JSON. And finally, we will try to replace the status code and body of target responses. You can see the configuration. The response of request send to 80 ports by gas method will be modified to status code 502 and replaced with the empty body. Yeah. And now we know how to use HTV calc. Next, we will talk about some other important features. First is a hypertext transfer protocol secure spot. It is usually used to protect HTV connection from main in the middle attacks. To hijack HTVS connection, we must provide the trustee certificate and the private keys for our Calcity Proxy. And in the simplest case, user can provide the same certificate and the private keys with the service. Or most cases for the insights in cluster server, then we can chase clients with the two certificates. You can see the picture. Our T-Pros have the same certificate and the private key with the server. So the client cannot realize that the HTV connection is hijacked. And sometimes, especially for the outcast server, users cannot provide the same certificates and private keys with the server. So we must sign a fixed certificate and inject into the root series of clients. As we step sign a fixed set and private key for our T-Proxy, we must inject the fixed set or the root set to the client root CA list. Yes, and in this case, the client also cannot realize the messages is hijacked. And next, let's talk about the plugins. Currently, we lack actions to be applied to requests and responses, especially to deal with the body. The format of body is too flexible to apply rigid actions. So we decided to plug in the faces to empower users to define their action by programming code. The web assembly is a really good choice for implementing the user defined function, UDF. We create a web assembly runtime for users to define the custom actions by different programming languages. Here is a plugin example, which is just 15 lines of Rust. You can look at it if you're interested in the Rust program language. Yes, we just use the macro, we just response handler and pass the lambda, yes, or you can call it a closure. And the handle receives the response and do some processing and return the body with the serialized by JSON. Yes, but create and maintain a web assembly runtime may cost too much works. Compared to web assembly, the native executable file has different advantages and disadvantages. The main advantage is do not need a custom run time and it is very fast. And this advantage is no natural isolation and binary may be too large to distribute because the web assembly binary runs on our provided customized runtime. So it is isolated naturally, but for the binary native binary or native script, we must isolate it by the afterwards like Linux and spaces, yes. And the binary, for example, for the Go or Rust, the static linked binary may be too large to distribute. And for the C++ with a dynamic link or some scripts, it may need more dependencies to distribute is much more complex, yes. We now use the web assembly, yeah. But in the future, we are considered to implement a native one to be choiced by the users. That's all. Thanks again for joining me. If you have more questions, you can join our select channel or send me an email. Send me an email, yes. Thank you.