 Hi, my name is Pisley, working at Hyundai Motor Company. In this talk, I'm going to introduce Guider, a new performance monitoring demo, and explain how to use it. This is a summary for my talk. First, talking about performance analysis and optimization and tools be required. Next, introducing Guider and its various features. And finally, explaining how to monitor performance of our systems automatically using Guider. Let me talk about performance factors. First of all, the major performance factor is CPU. There are many reasons to make your system slow, such like CPU intensive jobs, and frequent context switching, and busy rate tasks, and so on. If your system is slowing down, the first thing to do is watching total CPU uses and which tasks are using CPU cores. Memory is also important. Frequent memory allocation and deallocation jobs will consume CPU more than we expected. And infrequent allocation and missing deallocation, such like leaks, memory leaks, can cause the out-of-memory, and then system will be slow seriously. In the worst case, it will restart finally. To free up memory, links corner try to flush file caches, swap pages out, it called the reclaim. Once reclaim start, the system will slowly start to slow down. Next one is IO. Generally, block devices is the slowest in our system these days. So optimization, such like caching, preloading, compression, workload tuning is required. Especially, unnecessary IO operations should be removed and contiguous operations must be merged. Last one is for communication between tasks. Lock, it's very important to prevent data corruption shared between multiple tasks, but it can have a huge impact on performance. Accessible lock contention increases CPU usage and also response time. So moreover, performance can be worse depending on lock attributes, such like priority inheritance protocol and easy way and wake up storm, in few text and in hooks. So in addition to these factors I described, there are many other performance factors. Most importantly, we need to able to recognize each performance factor and measure its actual impact and verify them. We must think about how to measure them. So logging and using tools, most effective way to analyze performance. Logging is very useful for recording specific information, but understanding log requires domain specific knowledge. So system level engineers or new members, new engineers are difficult to understand them usually. In addition, adding new logs requires source code and tool chain for a reveal. It's very boring and time consuming job. It's also difficult to record and analyze too many logs because of the limitation for memory and time. So we prefer to use performance tools like this. It's very comfortable and effective to analyze performance at system level. So as shown in this picture, there are many performance tools in various areas, but sometimes too many tools confuse us and determining the right tool from a variety of performance issues is not easy. So even though they are not all installed in our system usually, so we should install them in manual, but they may require latest kernel and libraries and dependent packages and more kernel configurations. So after installing them, even require rebuilding their target program and restarting target tasks and loading kernel modules again. So there are too many limitations. So I introduce Guider, Unified Runtime Performance Analyzer. It can monitor, profile, trace and visualize various performance factors. And it also contains many embedded units. Now it consists of about 135 commands and each options. So monitoring features provide continuous performance stats every interval in real time and profiling features provide a statistic overview or collected data during a specific interval. Tracing features provide specific data on the execution of the task in the form of logs. So Guider is a kind of command line interface tool. So it offers a lot of features by the combination of commands and options. But in this talk, I'll try to explain only some useful features because of time limitation. So it's open source program and written in Python. So it doesn't require installation, but PIP and Yachto recipe are also supported for your convenience. Actually, just executing Guider.py file is enough to use it. Guider never use external binaries such like executable programs, libraries and Python packages, except for map plot for some kind of visualization. Most of Guider features are implemented directly using standard libraries such like Lipsy. That's the reason why Guider doesn't require reviled install and configuration. In addition, it can be applied with only one megabyte of storage space. It's very light. These characteristics are very attractive in embedded system. All features of Guider are supported in Linux and Android. And it also provides some limited features on macOS and Windows. From now, let me introduce some killing features of Guider. First one is monitoring system resources in real time. It looks like Linux top, but this feature works by periodically updating stats for system resources and events. So system resources are about system CPU, memory, swap, block and network storage. So as shown in the picture, in the first part, system resources shown on the timeline here, such like number of core number and RAM and swap. So additional system information such as context switching, interrupt, running task and memory zone and performance stats using our performance monitoring unit are also displayed. And second part here, important system library resources and events are displayed. System states such as CPU uses and available memory and swap uses and memory reclaim events and lab IEU and network IEU are most delicious information for performance analysis. In addition, core usage is also displayed not shown in this picture, governor, CPU governor and clock and temperature for each core can be shown together with FU using some specific options. In the third part here, storage information about busy and workload and available space is shown for each devices. Habit storage workload can cause serious performance degradation, so that's the reason why we should check these stats. In the upper part of this picture, here is similar to block devices, network information about inbound and outbound is shown for each device. In the lower part of the picture, not only system resources, but also task resources are shown with their attributes in real time. So it's a little bit similar to Linux tab also. Uses for CPU and virtual physical shared memory and swap, lab IEU and memory details displayed well. The shown tasks are sorted by CPU usage by default, but you can change the sort of order using some other options. And task filter is also available to show only specific tasks. All or specific function calls are monitored for specific tasks in real time also. In addition, task stats about function calls also displayed such as average, minimum and maximum time. At this picture, all function calls are shown with vector assist. This part is called the function called last and this part is for vector assist. So the usage is not about CPU, it's the proportion for total of actual function calls. So this feature is used when finding frequent calls or measuring specific function call count, including backtrace. Of course, there is another function monitoring feature to measure CPU intensive function calls by sampling techniques. The task filter and function filter are also sorted. Yeah, this is for these calls. It's very similar to prior function call profiler. All these calls including backtraces are monitored for specific tasks in real time. And all opened files and sockets and pipes are monitored for each process in real time. Files are displayed with positions and open plugs and TCP and UDP sockets are shown with binding and connection status and UNIX domain sockets are also displayed with file pass. This kind of information is very ambitious with debugging issues or performance tracing. The process filter and file filter are also supported for this one. Yeah, the tracing feature for native functions such as C and C++ and Rust and Go is provided. Native function tracing is started by bit trace command in Guider. And the command is implemented using backtrace called trap. Breakpoints for all symbol addresses from ELF and DWARF sections are injected to the target tasks virtual memory Guider itself. So Guider can detect events for function call and function return from target task using bit trace. So Guider can even read and manipulate registers and memory for the target task when function events calls. As shown in this picture, call stacks are shown with various steps for go program in real time. Arguments and binary name for each function are also displayed together in all line. The G option in command line is task filter. So yeah, we can filter some tasks including specific words. And H option means printing backtraces like this here. So next tracing feature is about Python function. Yeah, Python function tracing is started by PY trace command in Guider. So the command will print all Python method calls not native calls. So as shown in this picture, Python call stacks are displayed in real time at various steps depending on the stack frame. Files and line number for the each function are also displayed together. Call commands used in previous native function tracing are also available for this feature. Next tracing feature, yeah, is for this call. Yeah, it's very similar to previous commands. The difference with previous one is this call that is applied with backtrace and return value here and elapsed time for each this calls. Yeah, next trace feature is for signal. Yeah, signal tracing is started by sigtrace command and it will print all signals received signals for specific target tasks. So we can trace what signals are received and why the signal is caused like this. This is segmentation fault event. So text-based analysis is specific but less readable. So that's why Guider provides visualization features in SVG format. So using the SVG format output in your web browser, it provides an easy to view and responsive interface. So first visualization feature is about resource graph like this, this is for CPU and IO and memories. So this visualization feature makes it easy to understand big data collected for long time. So it also helps us to understand trends in resource uses and good for communication with other people. Next visualization feature is about scheduling. Scheduling data is very large and very difficult to analyze one by one. Therefore, as shown in this picture, scheduling data such as time slice and pre-emission and blocking should be visualized prior to detailed analysis. Using the SVG format output in your web browser, you can view details for time slices like this. So it's very effective for analyzing multi-threaded programs, interactive services and delayed tasks and core utilization. In addition, this feature also allows for scheduling events as well as other custom events having timestamps for start and stop. And last visualization feature is about call stacks. Analyzing only last called functions without who call stack is not important because standard functions such like read and write can be called by any other functions. Above all, in most cases, last called functions will not cause all the problems. The problem is likely some other functions that called those last functions. Therefore, to analyze performance problems in function level, we need to be able to see the who are including each call stacks. In this case, this frame graph feature is very useful to analyze call stack-based profiling result for CPU uses and IO blocking status and memory leak and syscall trigger and specific function calls. As shown in this picture, last functions at the bottom of each tasks are various here. So we need to analyze upper functions that contains them here. So I guess modifying those functions will improve your application performance actually. Opening the SVG format output in your web browser, highlighting and zooming and searching specific functions or stacks also available using mouse and keyboard. Yeah, so far I have explained how to analyze problems that have occurred already. However, how to analyze problems that are not well reproduced or had complex conditions, sudden stuttering and screen freezing, system reset and audio chopping and slow app launching or switching. These kinds of problems are difficult to analyze using simple logs and often require system level information, especially for temporary problems that occur suddenly and disappears. So there is no time to try to analyze something by connecting a terminal. Sometimes there are potential problems that are not even visible to the naked eye. How can we analyze these problems easily and quickly and accurately with minimal effort? For this purpose, Guider runs as a system demo at all time and monitors system status and activities. And based on threshold for predefined resource or event, specific commands are actually automatically executed to handle them. The specific options is shown in the figure. First, GuiderDemon loads the configuration file during initialization here. And registers resource thresholds for each defined event and commands to handle them. After that, their target resource uses and other states are periodically saved and checked. And if the status meets their conditions of specific events, then events are generated. Here. In addition, a separate report file is created and stored in storage by summarizing the stored system information, including resource uses. This allows you to automatically handle problems with predefined commands. When a specific problem occurs or start problem analysis with a generated report, these functions, which works all the time, are usually performed using a small amount of resources, about one to five percent per second based on one core CPU. So it's very lightweight and suitable for embedded systems as well. In the demo, the occurrence condition of each event is defined as the threshold value of each resource defined in advance through the config file. Each event is largely classified into system and task and device units. System type attributes are defined for system-wide resources. Task type attributes are defined as well as all or specific tasks. And device type attributes are defined for specific stories or network devices. In this case of resources, as well as hardware devices such as CPU, GPU, RAM, GPU-MAM, stories, network. Logical resources are set for such like block and load and file descriptor and sockets and specific files and tasks. Additionally, various types of logs and functions and IP messages are also available to be monitored. So inside Guider, a lot of, oh, it's difficult to see. Inside Guider, a lot of resources and statistical information is updated in real time. So it's continuously checked to see if there are any events that exceed predefined thresholds. Some of the information collected inside the Guider is displayed on the screen here. The command to be executed when an event occurs through the config file can be also defined. If you look at the config in JSON format on the right side of the screen here, events of system and tasks are defined for CPU resources. This is for CPU. So through the system part defined at the top, if the average system CPU usage exceeds 95%, the saved command will be executed and Guider creates a report file based on the information, system information, it has been being stored in the buffer itself. Through the task part defined at the bottom here. When the average CPU usage of a specific task exceeds also 95% for five intervals, the CMD theta frog commands is executed. So Guider starts monitoring for all threads of the target process and save the result into specific file. In this way, when each event occurs, the commands defined in the command list are executed sequentially and in parallel. So these commands may be predefined in the config file. So it can also be a user defined command that the user can execute immediately in the shell. So the left side of the screen shows the command provided in default here. So the top command system wide monitoring and FTAB command monitors open files and MTAB command provides specific memory profiling and disk top command provides specific IO monitoring and net top command provides specific network monitoring and fungal rack commands monitors functions for all threads and theta frog commands monitors all threads of specific target task and TTAB UTAB, yeah, here. Command monitors those specific functions that perform a specific target stress. And finally, leak command here, yeah. The command performs function tracing to find memory leak point for specific processes. In addition, various functions provided by Guider can be easily combined and used as event handling commands. All of this is done by Guider itself without any installing packages or specific libraries. Yeah, as shown the bottom of the screen here. When an event occurs, the names of the report files that are automatically generated have a regular format. All the files start with the execution order and number of reports and the resource and occurrence range and threshold value and time information of events are generated, the report added to the file, these files. In default, each report is just text file but it can be compressed and generated according to the Guider execution options. The reason that the compression function is supported because report files are automatically created in background, so storage can be excessively used when many files are created. So yeah, the biggest issue is no storage space. So in addition, if you use the option of the maximum size of the directory where report files are created, the demo tries to maintain the specified capacity by erasing the oldest one if a new report file is created that exceeds the specific size. This is very important for mass products for embedded world. From now on, we will explain the report file that is automatically generated when an event occurs. At the top of this report file, information like the picture is displayed. So execution options and versions and runtime, load, number of tasks, corner command line, this information helps to understand each system and execution options through report files generated in various environments. The following is information about system resources, information about RAM and storage network. Other one is displayed. It displays the system resources at the time. The report was generated as well as the resources at the time. The demo was first launched. Using this, you can see the approximate resource changes and easily determine the resources available at the time of event creation. If the previous page showed a snapshot at the time of the event as each resource table, from now on, based on the information stored in the buffer inside the guide, before the event occurs, it shows the amount of resource change in each section for a long time. For example, the top summary table shows the system resource usage line by line for each segment. This plays various resources such as CPU, memory, block, swap, reclaim, fault, context switching, interrupt, number of tasks, networks, and other ones. So this table means the change of resources, CPU, your memory, block IO, reclaim, like this. So next is the top CPU info table here. This is for CPU usage for processes or threads running. So it displays the minimum, average, maximum, and total stats for each processes CPU here. So it's very effective when we analyze specific tasks or use CPUs for a long time. And other resource tables like this one is for memory and delay, and IO, and C group, and other ones. When analyzing a report, it can be useful from several effective properties. Finally, the part that shows most specific information. So it shows detailed events and resource information that occurred in each section. It's very similar to Linux top. So it shows detailed events and resource information that occurred in each section. So based on previously summarized information, it's information that can be referenced when you need specific information on a specific section. In particular, the information of the processes is specified and it's possible to check how much each resource is used in each area. So finally, there are part, this part that shows the most specific information. Okay, so as I show you, this report is converted to SVG format visualization files. So, okay. There are cases when it is necessary to control the demo under special circumstance. At this time, since the demo operates in background, so commands cannot be sent to the standard input on the share. So the guide provides an event command separately to control the background demo. So this command is originally used to generate the specific event, but if string start with CMD is used, it's interpreted as a command to be passed to the demo. The command to be delivered to the demo is as shown in this figure here. There are buffer size change and buffer initialization and monitoring stop activation and deactivation of monitoring for specific resource change of monitoring cycle. Config reload and first report generation and demo restart and other ones. So I have explained some kind of features of Guider, including tracing and monitoring and demo. So there are many useful features besides the ones I described, but I couldn't explain because of time limitation. So I introduced performance monitoring demo features. So it will be more expanding to manage and analyze performance issues itself. For specific details, please refer to the readme file in Guider repository. If you have any questions, please contact me using email or GitHub issues. Yeah, thank you. That's opposed to many features. I felt that some of the features in this tool used the EVPF feature in the Linux. Yes, right. So which features are using EVPF? Can you explain? Yeah, right. EVPF is very nice and it's very latest techniques, but it requires latest kernel and some kernel patches. So Guider doesn't use it just using ptrace or ftrace only. Is there any unique features that the other performance tool don't provide? Unique features. Yes. Function monitoring and task monitoring and other ones, stack trace. Yes, right. So that's right. It's all right. Thank you. Yeah. So question is, can performance analysis tool works on the Yokto OS of embedded systems? Yeah, sure. Guider, there is already Guider recipe on embedded recipes, so you can use it easily. And actually, as I told already, just copying Guider.py file is enough to use it. Yes. There's no requirement for using Guider on any Linux systems. Yeah, it only requires Python, yeah, default Python. Thank you very much.