 Hello everyone. My name is Kobayashi Daisuke. I joined Fujitsu as a software engineer. I work as a developer in the OSS community. I have been involved in Kubernetes development for two years, and I acquire CKA. In this section, I will talk about introducing a log-tracking feature that revolutionizes the availability of Kubernetes. As a future to evolve the availability of Kubernetes, we will introduce a logging function that we are currently considering with the technical leader of Kubernetes community. If this function is introduced, even many user operations are issued at the same time. It is possible to easily extract how a specific user operation was processed from the huge log of each component of Kubernetes. And the time required for trial shooting may be greatly reduced. This is today's agenda. First, I will explain the mechanism of Kubernetes itself to make our proposal easy to understand. Second, I will talk about Kubernetes logging and show the problems for the Kubernetes logging we are thinking about. Next, I will talk about the future we are proposing. Finally, I would like to introduce the implementation issues we are facing now. The first topic is overview of Kubernetes. What is Kubernetes? Simply put, Kubernetes is container orchestration tool. Its future includes three points. First point is container deployment and resource allocation. Second point is container lifecycle management. And third point is container network management and container connectivity. Kubernetes was developed by Google and released as OSS. Originally, it is based on Google's in-house system. And it was donated by Google to the cloud-donated computing foundation, so CNCF. And it is currently managed and operated by CNCF. It is now the default standard for container orchestration tools. The advantages of using Kubernetes includes four points. First point is automating manual container management and monitoring. Second point is automating the deployment of developed applications. Third point is mass container status monitoring and logging on multiple nodes. A fourth point is automatically scaling when the processing load is concentrated. From the next page, I will explain how Kubernetes is configuration and how these functions are realized. Kubernetes has a declarative operation mechanism for automatic management and monitoring. A declarative operation means that the application is executed by describing and defining the elements and conditions to be deployed in the cluster in JSON. This definition does not describe how you want your application to run, but what you want it to be in the end. By this mechanism that we define the desired state, Kubernetes realizes automating the cluster management. Since that definition is coded, we can check the challenge history with a version control system such as Git. If you need to update the application, you can do it just by changing this code. So all you have to do is change the code. It seems difficult to do everything from container deployment to operation just by defining the final state of the cluster. Then, I will realize the declarative operation. This is realized by reconciliation loop. I will explain these reconcile mechanisms in this reconciliation loop. Component always compares the user defined desired state with the actual state, which is the state of the system. When the actual state equals to the desired state, the component does nothing. If there is difference between two states, the component updates the system state along the desired state by making such a mechanism fires automation such as auto hearing is realized. What does Kubernetes cluster consist of? The Kubernetes cluster consists of a control plane and nodes, like this picture. The control plane manages the worker nodes and the port in the cluster. The worker nodes host a port that are the components of the application workload. The control plane has components such as IBS server, ETCD, control manager, scheduler, and cloud control manager. In the nodes, there are components such as Q-Rex and Q-Froxy. What is control plane doing? The control plane's components make global decisions about the cluster, for example, scheduling, as well as detecting and responding to cluster events, for example, starting up a new port when deployment separators feels dissatisfied. I will explain that each component of the control plane is responsible for. The API server is a component of the Kubernetes control plane that exposes the Kubernetes API. The API server is a front-end for the Kubernetes control plane. It accepts the request, determines if it is valid, and processes it. It is only component that communicates with ETCD. ETCD is a key value store that holds all data about cluster, state, and configuration. The cluster behaves according to the state stored in this ETCD. Scheduler watches for newly created port with no assigned node and self-tips the node for them to run on. It determines the node to schedule according to various conditions such as cluster state and user settings. Controller manager runs multiple controller processes. Each controller migrates the object and updates the cluster to move the current state to the data state. So what is node doing? The node runs an application deployed in the cluster. In addition, the node component runs on all nodes, manages running ports, and provides a runtime environment for Kubernetes. The node components are described here. Cubelet is an application that communicates with the control plane. Cubelet confirms that the container is running on the port. If the control plane needs sound processing in the node, the Cubelet will take action. Cubet proxy is a network proxy for facilitating Kubernetes network services. Cubet proxy lives on the packet-fitting layer of the operating system or the forwarding of the traffic itself to handle network communication inside and outside the cluster. So I have explained the basic components of Kubernetes. Kubernetes uses the component just described to enable container orchestration. Kubernetes components work independently and run an application. I will describe some of the objects that Kubernetes defines to run an application. Kubernetes objects are persistent entities in the Kubernetes system. Kubernetes uses these entities to represent the state of the cluster. Kubernetes provides several bridging workloads. For my explanation, I will introduce three workloads. POT is the smallest deployable unit of computing that we can create and manage in Kubernetes. A POT is a group of one or more containers with shared storage and network resources and a specification for how to run the containers. POT's contents are always co-located and co-scheduled and run in a shared context. A POT models an application-specific logical host. It contains one or more application containers which are relatively tightly coupled. In non-cloud contexts, applications executed on the same physical or virtual machine are analogous to cloud applications executed on the same logical host. A replica set's purpose is to maintain a stable set of replica pods running at any given time. As such, it is often used to guarantee the availability of a specified number of identical pods. A deployment provides decorative updates for pods and replica sets. We describe the desired state in deployment and the deployment controller changes the actual state to the desired state at the controller rate. We can define deployments to create new replica sets or to remove existing deployments and adopt all their resources with new deployments. Kubernetes uses these components and objects to realize congenital restoration. The second topic is problems for Kubernetes logging. We are proposing enhanced logging capabilities. So first, let's talk about the outline of the log and the current problems. First, I will talk about the logging function of Kubernetes. Let's talk about overview of Kubernetes logging. There are four logs in Kubernetes. There's a pot log, event log, component log, and audit log. Pot log is standard output and error output of applications running on the pod. It's mainly used for trouble shooting for applications. Event log is logs for events that occur in the Kubernetes cluster. It is mainly used for trouble shooting for Kubernetes cluster. Component log is individual log for each Kubernetes components such as Quibret and AWS server. It is mainly used to follow more details in general operations than the event log. Audit log is log for HTTP request response to and from API server. It is used for security, audit, etc. The pot log and event log are often used for trouble shooting. If we cannot narrow down the code with these two logs, we need to look at the component logs. When it comes to investigating component logs, investigating can be a bit tricky. Let's talk about why. The problem we want to solve is that the investigating logs across components is difficult. Why is trouble shooting with component logs so difficult? This is because the components are distributed and the logs are also distributed. It's difficult to correct and analyze logs. Therefore, it is difficult to investigate logs across components. In Kubernetes, multiple components work as an extension of any operation such as CubeControlApply. Therefore, if you want to follow a series of operations inside the cluster, log analysis across components is required. However, the Kubernetes component log is not good independently for each component. So, it is necessary to extract the related logs from each component log and associate them, not only as a component distributed, but there are other factors that make investigating difficult. It is difficult to link requests and processes. Why is it difficult? This is due to the Kubernetes architecture. Kubernetes stores the desired state of the cluster in EDCD, such as a number of deployments. Each component only communicates with APS server to compare the current state with the desired state in EDCD. If component finds a difference, it works to keep the actual state in the desired state. This means that each component does not communicate with other components. Moreover, components work to keep the cluster state in the desired state, not one request satisfaction, since each component is driven independently according to desired state. The log is also output independently. So, in podcast, would it be difficult to investigate? For example, we want to investigate the cause when the port moves for some reason. In this case, we can investigate how the port works created from the logs of the newly created port and the logs of other components. However, it is difficult to investigate why the original port works moved. This is because each component works independently and the connection between processes is not clear if shown. In this case, the reason the port moves may be in the logs of the original node, but it's not clear when fit node it is. So, the larger the cluster, the longer this investigation will take. Some topic is future we propose. In this topic, I will introduce overview of this future and I show in first scene this future can be used effectively. So, first social idea. From shooting the Kubernetes is difficult because of this architecture. To solve this, I want to be able to easily correct and analysis distributed logs. I thought that it would be better to give a new leaf ID to the processing log output by each component as a result of the request of the user or controller. Here is an overview of the idea. Each component of Kubernetes cluster does necessary processing in response to user request. The processes of each component and the first request are linked with a unique ID so-called trace ID. By assigning the associated trace ID to the log output by each component, distributed logs can be easily corrected and analyzed. See an appearance of what happens when the trace ID is output to the log. As an example, I extracted the log related to the deployment creation operation. In this example, the processes of creating a deployment is distributed across multiple components. So, I guess I have a controller manager and keywords. Processing log is output in distributed manner in multiple component logs. We are trying to output trace ID to the log. The related processing can be extracted by using the trace ID given at the end of these logs as a key. By extracting the logs using trace ID as a key and organizing them in chronological order, it will look like this figure. In this way, it is possible to grasp the processing flow. To investigate the cause from the error log, we can correct the log by using trace ID assigned to the error log as a key. When actually correcting and analyzing logs, it is assumed that other OSS were reduced, such as R6 Such and Kibana and so on. This feature can reduce the time it takes for cluster administrators to troubleshoot. For example, suppose the cluster administrator looks at the error log when something goes wrong with the cluster. He wants to investigate the reason why the cluster output this error. The error log is given at trace ID that associates with the related processes. The cluster administrator extracts the logs by that trace ID. Then he can discover at once how the cluster behaves before the problem occurred. If we go back to the log we found, he can easily find out what caused the problem. From the next page, I will introduce in what kind of scene this feature can be used concretely. Usage 1 is tracking the user request behavior. When the user makes cube control a prior request, the request causes multiple components to work. There may be a use case where we want to track the behavior of components driven by this request. In this case, the trace ID will be generated when the user request arrives at the API server. Later, when saving the object to ATCD, the trace ID will be given to the object and saved with it. Then the components detect the creation of this object and it works. The component refers to the object processes it and at the same time propagates the trace ID and outputs it in the log. After that, the related components perform those processes in the same way. The trace ID is output to all logs of the processes related to the request. So using trace ID, we can extract the logs that is related to the user request. We thought extract it those in chronological order to keep track of what component did for the original request did. In this way, it is possible to grasp the processing flow. Finally, we can track the user request behavior using related logs like this red line. Usage 2 is tracking the cause of operation. Currently, the bot is running on the node. The bot has moved to another node for some reason. The following example is when we notice this move and we want to track down its cause and behavior. Even this time, we can easily track it by using the feature. First, if we notice that the bot is moving, check the trace ID given to the airport. Then, by using trace ID, we can extract the component logs related to the bot moving. By arranging the logs in the chronological order, it is possible to track what was done in the cluster before the bot moved. So we can grasp the following processing flow. If we check the extracted logs, we can see the bot moved because of problem with node. By quickly finding the nodes where the error occurred, the range of impact of failure can be reduced. Here the investigation of the logs will help track down the cause of nodes problem. Another example is when we find a system failure. For example, a node problem. And we want to see what happened as a result of this node failure. First, we notice that a node problem occurs and we get the trace ID from this curate log. Then we extract the related logs by trace ID, as in the previous example. We can extract the logs of the processes that were triggered by the problem of this node. By analyzing this log, we can see which component did what with this failure. So finally, we can see first range of affected by this node problem. I will summarize the specification for ID assignment and propagation required to meet these use cases. First, the timing of ID generation. There are two main types event to give new to this ID. One is when the user request is received. The other is when change in the actual state is detected due to the system failure or the like. The original trace ID is inherited for the request generated as a change action by the code controller. At the point, if multiple requests occur before the request is processed, all the requests related to processing can be traced. Regarding log search, we can search the component log from trace ID related to the error or the trace ID given to the event. The component log is timestamped and can be displayed in the order of occurrence. First topic is how to realize the future. I will talk about overview of realization. So first I will talk about our realization plan here. The element to realize our idea is this four points. First, we generate an ID for the request to the APS server. So this is realized by the KP and APS server iteration. Second, it gives a generated trace ID to the object. So trace ID information is stored in EDCD. Next, when the components work, it gets trace ID from the object. Then, it propagates the acquired trace ID during processing and outputs it to the component logs. So I will explain the overview using our figure. First, use app rise and manifest. For example, keep control apply. This request comes to APS server and new trace ID is generated as APS server. The APS server makes Kubernetes object and persistent it to EDCD. We plan that APS server stores the trace ID to the object in EDCD at the same time. The next step, a component detects the change of the data usage. So the object is created. The component starts the reconciliation route. Then, it gets object information. This information includes trace ID. The component puts the trace ID to the component log and propagates trace ID to other components. These components also get the trace ID and output it to the component log. By doing this, we think we can link from request to processing with a unique trace ID. Next, item is implementation issues. We are facing various problems with the implementation of this feature. So I will introduce them. There are implementation issues in the following situation. Issue 1 is multiple requests as a causal processing. Issue 2 is multiple components operates in a chain with one request. Issue 3 is the object to be processed is different from the object of the tutorial. Issue 4 is requesting and processing components are not one-to-one. So I will explain these issues. This section describes the case that multiple requests are the cause of processing. In Kubernetes, each component works independently and works to satisfy the desired state. Therefore, depending on the timing of the request, multiple requests may be processed at once. In this case, it is necessary to associate multiple IDs with processing. So it is desirable to output two trace IDs in the log. The first challenge is to achieve this behavior. The second point is that multiple components work in a chain with one request. In Kubernetes, one request may cause a component to work and that component behavior may cause other components to work. For example, if we request to create a deployment, so the deployment controller will also create a replica set, then the replica set controller works. This means that multiple components work in a chain with one request. Therefore, we need to propagate the trace ID to each of the components that works here. However, the behavior of each component is independent, so the issue is how to link them. Issue three is that the object that triggers the process and the object that is processed may be different. In this figure, an object means A triggers the operation of B's controller. At this time, the desired trace ID is given to the object A, but B controller processes only the B object, so B controller cannot get trace ID AA. Therefore, there is a problem that B controller wants to use the trace ID of A, that is triggers, but B controller cannot get the trace ID. To give concrete example, the replica set controller may take action when the pot is updated. It is difficult for the replica set controller to identify the pot that triggers the action. The issue is how to propagate trace ID to controller that does not process the trigger object in this case. Issue four is the case where the requesting and processing components are not one-to-one. The figure shows an example in which A's controller and B's controller operate triggered by a request to an object named A. In this case, it is necessary to propagate the trace ID issued in one request to multiple components, give trace ID to A and propagate it to the component, but the B controller cannot get the trace ID given to object A. Here is concrete example. The scene consider is seen in fit the replica set and deployment controller operates triggered by the update of replica set. Updating the replica set will run the controller manager. From fit the replica set controller and deployment controller will run. At this time, replica set controller processes the updated replica set, so it can get the trace ID of the request that triggered it. However, the deployment controller handles the deployment that manages the replica set. At this time, the information that trigger was replica set is lost in the process of enqueue to the work queue by event handler in this figure. Therefore, it becomes impossible to get the trace ID of the request that triggered it from the deployment controller. The problem is how to resolve the missing link. So we need to think of an implementation that can solve these challenges. And in addition, the solution must have the least impact on performance. So the final chapter is the current status. So we propose in KP to enhance the logging function. We are thinking of an implementation and system design that can solve all implementation issues. We want to make it a more valuable feature by understanding the needs of various operators. So last, I summarized these contents. I showed today's agenda here. So as a feature to evolve the observability of Kubernetes, I introduced the logging function that we are currently considering with the technical leader of the Kubernetes community. If this function is introduced, even if many user operations are issued at the same time, it is possible to easily extract how a specific user operation was processed from the huge load of each component of Kubernetes. And the time required for troubleshooting will be greatly reduced. Okay, thank you for listening my presentation.