 Hello, everyone. My name is Kobayashi Daisuke. I belong to Houditsu as a software engineer. I work as a developer in the OSS community. I've been involved in Kubernetes development for three years, and I acquired CKA. In this session, I will talk about introducing the latest Kubernetes features for troubleshooting. I will share three of the latest troubleshooting features in Kubernetes that are currently being discussed and developed in the community. The first is structured logging. A quick difference is the standard structure for logs and standardized references to Kubernetes objects to make the most common logs easier to query. The second is future code contextual logging. The key idea is that the library is passed a logger instance by the caller and uses it for logging instead of accessing the global logger. The cell is a function called distributed tracing. This is a feature to enable distributed tracing across Kubernetes. Most logging features can improve existing log formats and facilitate log analysis, aggregation and development development and troubleshooting. Distributed tracing is a feature that makes it easier to debug requests that cross-service boundaries by introducing open telemetry into Kubernetes core components. As more components introduce distributed tracing, Kubernetes will become easier to monitor, match and troubleshoot. In this presentation, I will explain the specific log format and usage, including an actual example of drug tracing. This will give Kubernetes system administrators and developers some insight into how these newest features and great and hard-rogan traces in the future. The theme is structured logging. I will talk about an overview of structured logging. This is a future proposed in CAP 16.02 by Malek. This CAP proposes to define standard structure for Kubernetes log messages. Add a method to K-log to enforce this structure as ability to configure Kubernetes component to produce log-in JSON format and initiate migration to structured logging. Conventional Kubernetes logs do not have the concept of key value and the values corresponding to keys are not unified, so there are issues during log analysis, processing, storage, query, etc. It is difficult to search if the naming format differs depending on the log, so by making the key a uniform naming format, it allows users to search with a uniform filtering pattern when analyzing logs. This image shows what the logs will look like when they are migrated. These are the advantages. The advantages of this format include this point. For users, by having log messages in structured formats, it is possible to standardize how to refer to Kubernetes objects, for example parts, nodes, etc., during log analysis. These make it easier to search many Kubernetes logs. By using the JSON format, we can lessen the following benefit. It works with many logging backends, so, for example, L6 search, start driver, a weak query, and sprunk. And it is easy to convert and pass, so analysis to write JQ commands are also available. And for DELPA, new K-log methods are allowed to output all structured log-in. Some methods introduced in K-log, and the methods introduced by CAP include these methods. The K-log library has methods called info-s and lrs. And this is a method used where previously info-f and lr-f were used. With these methods, the output of the log methods will be the message and some key value pairs. The example below shows the usage of the methods and the resulting log output. I will talk about the log-message structure. This CAP would write for the Kubernetes community to set around one preferred log-message structure that will be enforced by new K-log methods. Proposed structure should separate log-message from its argument and to read log and argument as key value pairs and be easily to pass and query and have a specific guidance on log-message and its argument. For this purpose, CAP suggests to use the following log-message structure. Structure is a message, key1 equals value1, and key2 equals value2, and so on. And so the message is formative using %q format logic, and keys are formative using %s format logic, and values are formative using %q format logic, except for number and vm. And keys are separated from values with an equal sign. And messages are key values are separated with a single space. The values to be put in the key value pairs part can be literally determined by the developer or code. So there is a possibility of variation in naming. For example, when representing a bot, use only the port name, or the name space is included, or use uuid. To achieve a uniform representation of a Kubernetes object, structure logging created a helper function to format it in a specific way. These functions are k object and krf. This shows one free file format for responding reference in logs, and build this format. For example, if we create the following pot objects, specify the pot object itself with the krf function. Although need to be, if we specify the name space and name of the pot object with the krf function, logs output in a uniform format of pot, equal, name, space, slash, name, as shown below. Pass some object left to info s and lrs function, and it will output in one prefaced format. For node names placed object, object left with name space part empty is recommended. Now let's talk about the flags that will be introduced. Structure logging arrows used to set the output log format from text format and JSON format. For that to a flag logging format is introduced. This is implemented as an option in component base. Some advantages of using JSON are broadly adapted by the logging library with value-efficient implementation, the adopt and the allow. And out of the box supported by many logging background is easily possible and transformable. And are existing tools for ad hoc analysis, so JQ. And some JSON keys are reserved words. So tlc's time span are the next time, and v is our velocity, and ar is our string and mfc's message. At the end of this chapter, I will introduce the development plan and progress of the future. This future is already in upper-end base development. Calog has migrated to v2 and supports logging instruction format, along with that the logging format flag has been implemented, and it is also possible to output the JSON format. Let's talk about migration. 21 logs of free account to the majority all equivalents of all logout groups have been migrated to structural format. Among the components, key bread and key scheduler have been migrated. Our performance benchmark is also being done with this transition. The next contextual log-in. Contextual log-in is future Proposing Caps 1377 by Patrick Owe. The proposal is to extend the scope of the ongoing convention of logging calls to structural logging by removing the dependency on the global care log logger. Like the conversion to structured logging, this activity can convert code incrementally over as many Kubernetes releases as necessary without affecting the usage of Kubernetes in the meantime. Contextual log-in replaces the global log by passing a logr.logger instance into functions where the context.context or an expressive parameter. It enables the caller to attach key and value pairs that get including all log messages. It enables the caller to add names and describe the component or operation triggered by a log message. It enables the caller to reduce the amount of log messages emitted by call-v by challenging the diversity. This works without having to pass any additional information into the call-v because the additional information was having stored in the method logger instance. That is passed to it. So the party components that use Kubernetes packets like client-go and node logger forced to use KLO they can do an arbitrary implementation of logr to logger and configure it as if they did. As contextual log-in does, it is possible to add information about the function, the caller's and relationship to the log and making it possible to trace the flow processing. For example, give scheduler automatically adds the following information to the log in the plugin to having filter processing by the volume by new plugin who supports etc. As a guess, we can adjust the amount of logging by challenging the diversity of the logger. When give scheduler performs common processing for parts, I was put to detailed laws for implement port and rest for rest important ports. Overview of log-out. Log-out define common logging API and interface for the purpose of not locking go-run-gauge programs to a specific implementation. Log-out itself does not actually have a logging implementation, but a log-out defines what method should be implemented as a logging library. The actual implementation depends on the backend logging library that supports the log-out interface. Log-out consists of a logger type and a log-sync interface. These definitions are shown in the great boxes. A log-out type is structured with log-sync interface and log-out type have some methods of log-out type. It can be passed as an instance to a function argument. A log-sync interface is an interface for the main method of log-out and its actual implementation depends on the logging library. Here are the methods that log-out defines. Info is a standard log-out width for log-out. And error is error log output. And three is velocity specification of log. The higher the number, the less log-output. There is default. Return the new logger instance after setting the velocity. Note that the specified numbers are added instead of overwritten. With name function as the name of the logger instance, return the new logger instance after name selfing. We can pass this logger instance to a function. If we use with name function inside a function, the name will be appended and this name will also be appended to the output logs. With values, with values as a key value pair values to log instance, the value of this key value is also added to the output log. I will introduce one use case of contextual logging. Contextual logging has the ability to trace function calls and relationships. Give scheduler developers want to know which port associated with which operation they are doing those. When queue scheduler starts processing port, use logger.with-value function to create a new logger and pass it to subsequent processes and functions. Then logger is used here after port name is automatically assigned. Then execute logger.with-name function for this logger when calling the volume binding plugin for a specific operation. From now on, logs using this new logger will automatically be given the information of port name and filter-slash-volume binding. Second use case is data and bubbles. In unit tests, developers can provide their own logger and distinguish output for each test case without challenging the logarithic code. We can change the implementation of functions such as info for each logger. The value set with values and with name can be applied to the logger. So we don't have to pass the content to be displayed individually. It can basically work independently of KL log output. For example, we can have our own logger. We can completely replace our binary logging backend and see that our KL log base Kubernetes code picks it up and use it. And second guess, if we use client-go in our application, we can reduce the diversity of the log messages from client-go by creating a logger.v1 and passing it to your client-go code. Logger v3.info called in client-go is the same as logger v4.info in application, not visible with v equal 3. Since the free value is associated with the logger, we can collectivity hide the logs output by components that you are not interested in. I will explain how to use each features. If we want to assign a value to the entire log, so use with values and with name method, we can add messages to embedded in all logs that inherit that logger using with values and with name. This also makes it possible to follow function calls. As a case, if we want to specify a logger, so set logger when calling when it is library. How to set the logger is to attach the logger to the context and pass it. Some functions to support context were added in the catalog library. klog.newcontext makes new context that has the specified logger and klog.fromContext returns the logger that is included in the specified context. The features can be used just by importing klog and context library, which values and with name functions as an example are here, like these yellow boxes. First logger has only pot key value, so the output is added, the pot is called ns-potname. Second logger, the logger has name filter and volume binding, so the output shows the filter-trash volume binding and the pot is called ns-potname. Third logger, the logger has another name, filter and node resources. So the output shows the filter and the pot is called ns-potname. Next, I will explain how to use the logger. As I said earlier, the logger can be attached to a context and propagated between cards. In this example, we passed a logger with the name example and the value if we use the logger to output the log, the value so example and the value will be added in addition to the message. To follow an example, attach a logger to a context and propagates it. Here, we use a function klog.newContext to create a context with the logger attached. We pass that context to a function and within that function we deduise the logger by using klog.fromContext. After that, we pass new information to this logger in the function and add to the log. As a result, all rows in this function will have the prefix, example, slash, something and the values are full e-golper and the root is called ns-name. Using loggers in this way makes it easy to trace correlations. At the end of this chapter, I will introduce the development plan and progress of the future. This future is already fully developed up to alpha. For contextual logging, klog can log out implementation based on the log-out API. The key idea is that libraries are faster logger instance by their caller and I use that for logging instead of accessing a global logger. The library decides about the logging implementation, not the libraries. The initial version of this cap describe a plan for moving or code for logging into kubanet.io slash klogr repository. Transition to ZAP would have removed all legacy code from kubanetis. However, ZAP transition would have been complicated to force all consumer of kubanetis code to adjust their code. So, the scope of the KP was reduced from removal dependency on klog to removal dependency on global logger in klog. Next step is deprecated klog-specific flag in kubanet components. This future is proposed in cap 2845. Klog has a lot of flags which might not be meaningful nowadays. This cap proposes to deprecate in the future the removed subset of kubanet's command line flag from kubanet components with goal of making logging of kubanet's core components simpler easier to maintain and extend by community. Luck of investment and growing number of klog futures impacted project quality. Klog has multiple programs. For example, addison format. It doesn't support throughput to fulfill kubanet's scalability requirements. It's complexity and confusion caused by mounting backward compatibility for real CG log future flags. In this future, what kind of flag are left? All klog-specific future flag except blue and green module are removed from kubanet's components flags. The components with log-flag frequency flag is also kept. Let's see what we change. This table lists deprecated flags. These flags include futures such as log file lighting or load load editing. We are using those features. We have suggested some migration options. If we want to output logs to the file, we can use kubanet's log-runner solution as an alternative. There are solutions like load-stache etc. or load-dated. We introduced a development plan and progress of this future. This future is always being offered in a better environment. Among the kubanet's components flag, the target flags are already deprecated. Remaining the kubanet's components flag for klog are blue, green module and log-flash frequency. Flag deprecation should comply with standard frequency and required three lists before removal, so these flags will be removed eventually in the 1.26. The next theme is APS avatar tracing. The next feature to introduce is APS avatar tracing. This feature was proposed in CAP 647 by David Ashpole. This CAP proposes enhancing the APS server to allow tracing requests. For this, it proposes using OpenTelemetry library and exposed in the OpenTelemetry format. So first, what is distributed tracing just briefly? If a user makes a request to something, it often passes through a variety of services along its path. We'd like to know what path a particular request took and it would be great if we could get a graph or something to show it visually. So OpenTelemetry is a better neutral open source of the ability framework for instrumenting, generating and collecting and exporting telemetry data such as traces, metrics, rows. As an industry standard, it is natively supported by a number of vendors. So why instrument the APS server? The Kubernetes APS server is a great candidate for tracing for a few reasons. It follows the standard RPC model and serve a request by making requests to downstream components. Which makes it easy to instrument. Users are right in the center. If a request takes more than 10 seconds to complete, many clients will time out. It has complex service topology. A single request could require consulting a dozen webhook or involve multiple requests to EDCD. This feature was proposed as there are many benefits to distributed tracing in APS server for this reason. My use case is to allow control plane administrator to easily find the APS server's issues. In distributed system it can be hard to figure out where problems are. It also has to find out what route a request took. If we can trace it, we can see which component was included in the path of the request. In addition, traces are easy to visualize so we can understand the seconds of events how long they took and easily spot potential problems area without being familiar with the internal of the infrastructure. In APS server, normally corrects traces at low sampling rate. Sampling only some requests have less impact of system performance. Even for issues that don't happen every time, a low sampling rate is generally still enough to suppress a representative trace over time. Traces can be also corrected on demand when problems occur. If we know the request we want to correct so we can correct the trace of that request by setting trace context and sampling the request. When tracing a request that reads nodes, we can press the request by using the command like below. I will introduce the future to be added. Tracing of APS server requests this can be left APS servers, HTTP server and HTTP client with other HTTP to get spans for incoming and outgoing HTTP request. This generate span for all samples incoming request and propagates context with all client request. Second, this cat proposes the use of actually tracing framework to create and export spans to configure backends. Thus, as context propagation to APS server, context propagation is implemented across APS server to propagates spans. A process provides function related to tracing utility libraries that wrap all of them. Having control over how the client requests are used in this can enable maintenance to enforce policy and make road improvements to the quality of the term. How to use this future? Let's turn on future get and list of APS server. First, enable the APS server tracing feature get then start the configuration for tracing by pointing the tracing config file flag on the client's APS server and our config file. Of it contains APS server kinds and sampling rate per million. In addition, ADCD can tracing is enabled by setting the following three flags in ADCD and then open them to the corrector to correct exported spans. Service export spans in ODLP format and correct corrects them. It is the load loop backend to visualize and correct its information. We quickly chose the backend. This image is visualizing traces with data and we can easily see how long the process is taking. At the end of this chapter, I will introduce the development plan and progress of this feature. This feature is already fully developed up to alpha. Tracing of incoming and outgoing HTTP and the RPC request is implemented in QAPS server. This makes it possible to correct traces of APS server requests using an OD corrector. In the data version, they will implement tracing 100% of request does not break scalability test and publish example of how to use OD corrector with Kubernetes and party with all text-based traces. In the future, they plan to solicit feedback and make improvements and migrate text-based tracing and more. I will also introduce keyword tracing as a related cap. This cap is proposed in QAPS 2831. This cap proposes to enhance the keyword to our tracing gRPC and HTTP API request. Keylet traces can also be visualized with Yega like this feature. It is easy to see how long the process will take. I will introduce the development plan and progress of the feature. In QAPS 2831, they will route up to alpha. They implement tracing of incoming and outgoing CRPC and HTTP request in the keylet. They plan, publish example of how to use the open temporary corrector with Kubernetes and all time to feedback and implement a revisit format Finally, here is the URL for QAP, which is a function introduced in this time. If you are interested in, so please refer to the detailed information on the content. Okay, that's all. Thank you for listening my presentation.