 Welcome to this session, and thank you for coming today. My name is Hirotaka Motai. This session is real-time testing with Fuego. Here is a basic outline of what I want you talking about today. Firstly, I'm going to introduce myself. Secondly, I'm talking about overview of this session. After that, I'm going to move on to three related tools. And then I will show you our issue and approach with our use case. Finally, I will conclude this session. And we will have time for questions. So I'm just to introduce myself. So my name is Hirotaka Motai. I'm a researcher for embedded systems at Mitsubishi Electric Corporation. And our group provides Linux and hypervisor and related technologies for various products. My research focuses on real-time system and automated testing. So we've started to join the Linux Foundation project like LTSI, and AGL, and Fuego since 2015. Sorry, 2015. Let's move on to overview. So recently, Linux can be adapted to various embedded devices, kind of a controller navigation system, and some. Even though they need hard real-time response, in other words, projectable latency, we need a ton of time to end your advocate performance of real-time because real-time application needs to satisfy timing constraint. And they should avoid kernel changes, which might cause long delays. So it's my suggestion. I have changed a new script on Fuego. It can measure the real-time performance and get tracing at the same time. So we will get a hint to isolate the program whether the delay was caused by our changes or not. So in this session, I'm talking about the detail of a new test script. So next is Related Tools. I'm going to introduce the Fuego. You know, there are a lot of open source software tests and test frameworks. Fuego is one of the test frameworks. The reason why we have chosen the Fuego is that Fuego is created by LTSI project based on Jenkins. If you have experience of Jenkins, you could be easy to use the Fuego. And you can also customize it. And second reason is that Fuego is open source software. So anyone can use it and also can contribute it. Additionally, automotive grade Linux project, AGL project, chosen to Fuego as a standard test environment. It's called AGL JTA. So anyway, Fuego is a kind of host distribution plus a bunch of host scripts and test-packed along with Jenkins and Jenkins interface all inside the Docker container. Fuego has a built system due to architecture neutral and inherently cross-platform. Each test program builds from source code. And Fuego also has script for test orchestration, result parsing, and analysis for the test results record. Fuego can carry out specific tests automatically. That is triggered by software updates. In this case, I notice it is necessary to write a script to link both of them. There is a screenshot of Jenkins. You can see a part of the test in Fuego. Right now, Fuego has about 150 different test suites and some of them are not Fuego-specific tests, but most of them are wrappers and existing tests like drystone, bony, hackbench, and so on. Fuego has a wrapper for building, deploying, learning, and correcting the results for each test. And it also has a processor. So whatever the test has wired out of it that can correct information and put it in a standardized output format, sorry, a web control interface for starting and monitoring and checking their results is doing vitriolization on it. And you can also check the historical results on the web interface. So next related to is cyclic tests. It is included in RT test and benchmark 2 to measure internal timer latencies. If it is also open source software, you can get the source code from those URLs and you can also compile and execute on your target. You can see two graphs. Left one here is a time series graph. x-axis is a number of events and y-axis is a latency time of each event. The other graph is a histogram. x-axis is a latency time and y-axis is a count of events feature same latency. So third related to is F-trace. F-trace is one of Linux kernel futures. It is a generic tracing system to get maintained in 2.6.27. We can trace an internal flows of the kernel without recompiling. It is useful for data gathering, debugging, and performance tuning. If you want to know the detail, you can read the documents. I think there are a lot of sessions in ELC also. So this example is a result of function tracer. The function tracer is one of futures of F-trace. You can see FAT process was executing. FAT CPU it was executing on. Then it was executing. And FAT function it was calling. And you can also check the kernel status like that. So let's move on to issue. I'm sorry, so this is a you might you could not see all the results, but here is a result of cyclic test benchmark. And the graph shows the historical results, maximum latency. And you can see a big problem here. And I guess I think FAT change has occurred that delay. Was the delay occurred by our changes or by potential performance issue? So as my suggestion, I think it is necessary to trace during measurement. And to save a tracer to isolate the problem. So I'll show you our approach. Fuego can execute a script test on target. So I have made a test script to the following things on the target via SSH. Firstly, it set up function, sorry, it set up F-trace configuration. For example, it's clear link buffer and set current tracer and so on. Secondly, it learns stress programs. It make a kind of situation as same as our application. And after that, it learn cyclic test to measure the real-time performance. And finally, it gets a cyclic test log and F-trace data from the F-trace entry on that target. So recently, cyclic test has an option, break-trace break-time, that we stop tracing when the latency is over the break-time. So the script use the break-time option and we try to get a trace data in the worst case. But unfortunately, the break-time option is stopped not only tracing but also testing. Here is a function, trace mark, that we stop tracing. And the shutdown variables will be used later if the cyclic test should stop or not. So we can get the result and the F-trace data, but the record, the result might not be the worst case. So actually, F-trace has a good future to remain the log in the kernel buffer as we want. It's a snapshot future. It is easy how to take a snapshot, writing value to the snapshot entry. F-trace log is saved in the snapshot buffer in the kernel quickly. And the snapshot log will be overwritten when the program takes a snapshot again. So I have modified the cyclic test to save a maximum latency value and take a snapshot each time when the maximum value latency is updated. And after measurement, the snapshot log in the target, in the worst case, will be remained and we can get the log and the data. So I'd like to share our actual use case. This is a detail of our evaluation environment. Our aim is to make the influence to real-time priority process by other non-real-time process clear. So we have used cyclic tests. I have modified a stress energy program to make several stressful environments. A target board I have used is MinoBoardTabot. It has an Intel processor with two cores. And DOS is Debian-Gun-Linux 9.5 core stretch with the latest LTSI kernel. So stress energy at two to load and stress a target system. It has a stressor for a lot of components. For example, CPU loop and fork, timer, sync, create directory entry, file lock and unlock, and so on. I notice in you that extreme scenarios feature made by stress energy are likely to happen in real life. Anyway, the number of stressor of stress energy has roughly 200. I guess so many. And this slide shows the result of the measurement. So you can see the maximum value is 76.463 milliseconds. And this is a snapshot log at the same measurement. You can see the value here. 76.467. This is the same value of measurement. And it's sure it is revealed that the snapshot log was recorded in the worst case. So evaluation, we got through the detect the factor by doing a testing, doing a test and tracing at the same time. We all helped repeat execution of both. Therefore, we can effectively figure out the reason of the GA, which using the snapshot log. I think that's conclusion my session. It is important to endure, educate, performance before releasing products. It is necessary to repeat tests for reproducing the rare case, which does not meet real time performance requirements. Fuego is useful to us for not only measurement, but also tracing at the same time. Future works. I will discuss with Fuego community for adding our test script, which I have shown you. Here is a list of related projects. So thank you for listening. Do you have a question? Yes. Thanks a lot. One question. Do you have published your version of Cyclic Test, which you have modified? I maybe 1.3.0 Cyclic Version. Cyclic Test. You have published the source code of your modifications? Yes, I will do. You will do. Yeah. OK, some link or? But I think it needs, I need to discuss our tests or Cyclic Test maintenance. And we will need to show the effect or merit to add those features in Cyclic Test. So I will do that. But I'm not sure when my modification will be accepted. OK. Yeah, thank you. Can I comment on that? Yes, please. Just by way of commentary, so you can actually put the patches, though, into Fuego and apply them in the build phase. So you could, if upstream won't take them for a while, you can submit them. You can give them to me, and I'll take them. Exactly. So anyway, yes, please. Did any of the timing tracing that you add add to your latency or add to the timings you were seeing? In other words, did tracing contribute to any performance issues? I say the tracing feature is, what do you want? So I'm just realizing your question, but let me see. OK. I'm good. So the influence to enable tracing feature is, so make a performance loss or something like that. But in my opinion, if the results of measurement is OK, it meets our requirements. So after that, we disable the feature. So those environments may be OK also. So it is important to remain low, because the worst case, the rare case, especially rare case, is difficulty to result or to detect the factor, find the delay is what occurs. So I know the balance. But my suggestion is enable a trace feature. Thank you. Thank you. You presented one snapshot for the worst case. Is there the possibility that multiple snapshots are something like this for, let's say, the 10 worst latencies? I see. You can test several times, so you can get several snapshots also. But you only fetch the worst, I think, or are they all stored? Art snapshots per one testing. And so if you need several snapshots, you try to repeat several times. OK. Thank you. So if you have many more questions, let's discuss after this. So do you have any? Yes, please. Could you share some data about what were the typical reason to increase the maximum latency and what was your resolution to resolve the maximum latency issue? I see. But unfortunately, I have no low enough. Sorry, so sorry. Actually, you can use that. So you will get the same situation and same resolution. And anyways, OK. I'm going to finish my session. And thank you very much for joining.