 We're always running out of time, so well, wow. A lot of people are not going to Timbert. Sorry that you miss him. I miss him too, but it's great that you're here. I would talk a little bit of one of my favorite tools I'm using now really for decades in the looks and also for embedded things. And picked up a few special things, what edge stress is doing seriously almost every day for me in both software development and also in the lab working with real hardware. Who has never used S-Trace before? Okay, not too many, so I would have said, this is mostly command line based. I will give a lot of examples. The slides are only giving you some key points what you should read up in the main pages if you're interested in. After the talk, I will save my history about the commands and also put that upload to the conference page. And also the slides are a little bit updated in the meantime, but they don't have any serious content. So it's just giving examples and ideas. And so a little bit outline what it is very few sentences to me. And then I just want to show where you find the documentation and ideas about S-Trace. And then giving some examples about how you use S-Trace at all, what it is all about and how you get the output, how you shape the output, get more or less information and making it useful for whatever you want to do. Giving examples about filtering system calls that you select on what you're really interested in. And then, should I go closer to the microphone? Then you can lower it a little bit, but I always use small echo. I thought it's quite interesting to see, because I'm using that a lot, what you can do with timestamps timing and what you can get from your examples, for example, from any data sources or from your computer. And if time is enough, there's a small example of how you can replay things and why you want to do that maybe. So about me, very quick, I did very first open source things already in school, but we didn't know that time because there have been two high schools in a row in one street. And so we built hardware and software and handed it over to the other two high schools and was playing around when I came to university. The first thing I got as a student as my first job in computer service department, oh, there is this real Mac tape. So these large things, which you see in old FBI movies. There's some software that was then the type setting system latex for typography, type setting math and things. That is also open source before that was a term because Donald Knut all put it in the open source, public domain, whatever. With the only thing, you have to pass very extensive tests that you can call it tech if your binary runs there it's not passing in the tech, you should fix that. And yeah, then it was working early with VMS for the tech systems. Later, we got the first Unix machine in university, hooray. But that was all closed in Germany. The problem was we didn't get Unix source license and things. It's really big iron. So whenever something did not work and you wanted to extend it, sorry, you lost. Until then in the early 90s when really the source licenses and the AT&T lawsuit went apart and the first three BSD, BSD 386 at that time, sources came apart and then I bought an Intel machine again. I was playing with Intel machines and even before it did assembler on the 65 for two in a Commodore and did an assembler on the first 80, 88 IBM PC for hardware stuff. But it was all as gravity architecture and I hated it. So I liked other architecture way more because they are more clean. But then I had to buy my PC because it was an opportunity to write Linux which I started already in 92. So more than 30 years ago. And then I bought a graphics card which was supposed to be supported. And I noticed, oh, it's not supported. But also I stepped into writing support and correct style for my graphics card and did it a bit longer and was comment in of the S3 chips for a while in the x86 project. 2001, I finally left after a wonderful time at university and earned some money and was working in my hometown and giving for 13 years altogether, giving Linux support to customers or whatever they want to have getting in the R&D environment of automotive mostly. And until I had a contract for two years from Bosch SensorTech, just writing a driver for a YouTube which is upcoming and it took a little longer. I'm still working on that project and still working on those drivers, how that changed a lot, drivers changed a lot. And 2013 after two years of external development I joined Bosch SensorTech for Dream Consumer Electronics and a project which will be announced the next years. Let's see. And Bosch SensorTech is actually subsidiary of Bosch Automotive and they are doing all the consumer electronic sensors. So there's acceleration, gyro, magnetometer, whatever gas sensors which you have in your smartphones and all these things. It's all everything but automotive. So about documentation for that all. The most important documentation for me being a very old Unix guy is still man pages. And yes, you have to read the man page to get all the options, command lunch option, how they work. I was looking up because some of them I never can remember. There are new command options from time to time. It's an active development that's great. In the lower part, I found some nice talks from Dimitri Livin is the author or one of them. So he's the guy always giving talks at Foster in Brussels and he's very exciting to listen to what are the new things and I had a lot of bits and bits learn from him and typically always forgotten very fast because if I don't use it, it's not in my back brain and doesn't work out. So it's very basic what I do but even that very basic features, S Trace is very cool and there are more options which you'll find in the mostly demand page or also in the talks from Dimitri. S Trace is all set up or is using the P Trace system call. This is what the Linux kernel and also other Unix things offer as a debugging port that some debugger can control some other process that's process tracing and with that thing you can stop a process, single-stab it, make breakpoints and all these things and S Trace also uses S Trace that's for system call trace. So it's interfering in their call from your program from user level to system the kernel. This is the POSIX API. So you see all the POSIX calls, how you open a file, read a file, do these things, how you do network connections. Always on the POSIX kernel level and this is why what we will see in a minute from the S Trace output, this is always kernel calls and if you don't understand, it really will output, you're doing right with these parameters and if you don't understand a parameter or want to know the exact details of those calls, you always can read the manual and Unix manuals are intersected in sections and section two are the system calls that's the POSIX API. So whenever you see something in that thing there must be a men section two, whatever. Most of them you can just do men whatever but if you do that with right, you get in section one because there's also command line, command line, command right. So if you do men right, you get section one, so always men. And this is not what you're looking for then you can write to some other users because it's section one. So if you really want to understand what right does then you get this information that says right has a parameter file descriptor, blah, blah, blah. Who never was programming or reading any line of C code? No, hand up, that's great. Yeah, the question is just because the kernel interface, the POSIX kernel thing is plain C interface and so you just have to get a little bit of an idea what right might do and what the parameters are all about. If not, there are the men pages, documentation R and just a hint on the webpage for this talk. There will be, there are not only the slides which were the update after the talk but also two older articles from 2014 and 2015, a little longer papers to read, getting more ideas how you can do system tracing and also a long article how you can use S trace for analyzing the bash, really the command line and with coding and globbling and wildcards and things like that. Okay. Oh. How to start and use S trace. S trace has two modes. You can say S trace some program and it will output things and you can do S trace dash P some process and you can attach to some process. I would use both now just for giving a lot of examples. So if the thing is running, sometimes your start off your database takes long and S trace will do a lot of tracing and outputs and can slow down your processes, then maybe it's good or it's already running the SSHD is always running in your system and now you want to trace the next login what's going on and things. Then the second part is good. If you want to have all the important things, then you can S trace just start your command. The first very important option which I have here on the slide and we'll use that a lot is dash F, follow all childs. So typically S trace will only trace this single command or this single process you're specifying and if that's going a fork and exit and doing new child processes, it's not being traced automatically. You have to say dash F, follow all those things. For the second example, it's important because time is the command which will be S traced and if I only do S trace without the dash F, it will only trace time and time will then exit as a child echo because the time wants to figure out when echo is done and say this took a few milliseconds and if you don't have the dash F, you don't see echo but now let's, if, ah, okay, and the first example, I do not see my task list, that's bad. Why can't I see to which window I'm switching to, I don't know, somehow that's gone. So it's a little bit tricky getting the right window, please. Get that, so that's second number two and this is number three. That's what I would, maybe I have to close that. Okay, now here's a very small shell which I can use as site testing thing. So this bash has process ID 11455 and now I can do the very first example which I always say, hey, is S trace installed at all? And I do a, for example, S trace PWD. Unfortunately, most of the Linux distributions don't install S trace by default and you will only notice when it's a little bit late and you have a big problem, you want to use that wonderful tool and notice it's not installed. So if they have not installed it, do it now. And so here you get a lot of code now from S trace, it starts with exec, the old was the forg and exec thing, now it's virtual environment, whatever Linux thing, you see which path it is and how the command was called and a lot of other things. And whenever you look into that and you have no real clue what it's all going on, just keep it and look for the important key points and focus on a few of these things. But you can see it's opening libc, you get all the file paths which are loaded and which are necessary to start up a file at all. Very first one is the system loader cache and you can see how it's reading the shell library and blah, blah, blah, blah, blah. Here, the only thing I would be interested in what is get PWD print working directly doing is it's giving a kernel call, get current working directory and this already returns some magic string. And it says R, this has been 32 bytes. If you're not clear what exactly it's doing, then the other man pages. And hope this is man page on section three. And if it's pretty good, yeah. Section three is the C library, just directly calling the kernel call. It should be almost identical. And it says, yeah, this is the get current working directory. You have a buffer, you have to specify how much bytes are the buffer, how big is the buffer space that it's not doing override and memory leaks and a buffer overflow attacks and it will return, if you read the text, how many characters are returned back. So this is all how it's working with S trace. But yeah, just looking, oh no. So it said we have a buffer of four kilobyte. Feel free to write up to four kilobyte of path. And the kernel then returned with, now this string is 32 bytes long. If it would be larger than the 4K, it's not writing everything and you see it at least in the return value. And the second important thing, what now the most of the talk is about, then it writes to file a script on a one standard output, exactly that string, which you got from the kernel with a nice line ending that you see it from this next page. The bright writes exactly thus 32 bytes. I'm wondering how that works with a backstash end with this has been 32 bytes and this should be 33 then. Interesting. Oh, the current working directly maybe counts the zero byte. How am I to know that now? So the full string is 31 bytes. So the get current working there in return code also counts the zero byte, which also has written into the buffer space. And here you only count the number of characters, which you really wonder have been written because zero bytes don't count for write. It's just buffer, right? It's binary data without zero bytes. And then here you see that the string is coming out of the console. I will show you in a few minutes because that's quite annoying that now exactly it's getting called in the kernel. The kernel does that output. This is why the string shows up here. And then when it comes back from the kernel, S-Trace takes a second. It catches calling the kernel thing and it catches again in when the kernel returns. And only then it catches, okay, now I'm back from kernel. And the kernel said, I wrote 32 bytes for you. So if the disk is full, that could be less or whatever. So this is how it's looking all alike and you can watch all those system calls and can do a lot of nice things with that. That should be, yes, that's my browser again, so there should be those three pages. So calling it directly. And it's option one, just do as, yeah, it's doing well. It's even less sophisticated because it's not, it doesn't have to do a kernel call. It just, it's really write the string to the console and it's always a little bit garbled. This is why typically I do, yeah, just, I know what the echo thing is doing. I also write it to the regular output from echo to def null and S trace always outputs to standard error by the script or two. This is why I can throw away the standard output and still get the S trace output on the console. And then it's a bit easier readable. It's all in one line that please write the character. Nice. If I now want to see how long this takes and I very often type timing in front of commands. How long does the compile work? How long does this and that work? How long does the echo work? No, that's never tried it before. It's quite fast. And time also outputs on standard error because yeah, that's important information you would have a console even if you throw away the hello greeting. And if I do an S trace now, and I look what's really output, then it's a top correct and some timing and then if you analyze that, you see that this is only the output of the time command. And so 83% blah, blah, blah, whatever. And it outputs a lot of single characters, don't ask me. And this is because it only traces time. And if I want to see the echo thing which is more interesting to me, you have to say follow those child processes, please. And if I do that and scroll up a little bit, then first we should see somewhere that it really does exec, blah, blah, blah. So this first is all time. And now since it's more than one process, it shows me the process IDs up to somewhere, the fork and exec, you can read that in the first paper. It's executing whatever process ID. I should be saying, ah, the first process is something 63. And somewhere there's the clone of, in the old time it was fork process, now it's called clone. And the clone says, this is the child process and now since it's now tracing two processes, it starts attributing every line to, this is process this, this is process that. And yeah, the process blah, blah, blah, 64 is going to the echo command and the echo command finally somewhere are here. That's right, hello, Brock. Hi, sir. I can do the same now with the touch process. This is what I want to show, oops. That way, so this is, I have a process running already. This is this special over here. And now I can do a sjs-p-process ID. Ah, on my distribution typically this doesn't happen, but I know from Debian, Ubuntu and others that you are not allowed to trace your other processes which are not in your process group and which have been executed from the term security features. Which is quite annoying if you want to trace something, as root you can do, so I could do now sudo, whatever. But if you do that a lot of times and you don't want to always do that as root, there's this magic command here, it's in Proxys kernel something, you can check what your currently security scope is. There's a main page for that and it's, is it extra in the sjs, I don't know. And you just select this command, please. Yeah, reading is fine as a normal user of course, but if you, right now this is state one. And so if I want to trace all my own processes which belong to my regular user, then you have to do that once. And if I try the strace again, now I'm allowed to do that. And so now it's connecting there and says, oh, I'm in pselect, whatever that exactly is. And if I now start typing here and type an h, and you see now the pselect returns that, ah, there was an input and now it can read a character that, don't look at all the parameters. If you're only interested in data, it doesn't matter how pselect and select and poll and these things work, then it just notifies you can read something, you can write something and then see, yeah, this is the read, this is the write. And now if I type, you get for every character something. If I press enter, a lot of things happen because then it tries to execute the hello command which does not exist and it gives a lot of output and then it's waiting again. So this is the easiest, oh no, let's trace again. If I do that and there's a lot of things which I'm not interested in, we do that in a minute. Getting back to the slide, so this either call it with S trace or attach it. The second part is how do you control the output of S trace because it's very important for the usability I think. The easy way and I always start with that, it's just get the S trace output on the console itself then you see what's going on. Also you want to figure out either just get an idea of what the process is doing at all, just S trace dash P, what is happening? Is it doing anything? Are there a lot of kernel calls? And if you then want to do an analysis, then you can more fine-grained and do things, write things to output files. But then you don't see interactively how it's working, you can do a tail dash F on that file that you see what's going on. It always depends a little bit what exactly you're interested in, how you start looking typically. First with S trace tool, then on the next slide I will show how I select to dedicated system calls and I choose all my options which timing, the time stamps I want to have. When everything is clear, then I do the tracing or to starting again with all the gory details and options and then I put it to an output file. And then the output file can do with follow all processes and a lot of these options, some of the options of S trace can be multiple times. One, two, three times, I think no more than three times and they do different things. So here just follow all the child's that put everything in the output file. If you do that dash F twice, no matter if you do that separately or you always save dashes, save all characters and recycle, it's really possible. Dash dash dash F F just do it twice, which means that you don't get an output file but you get output dot process ID for every single process. We'll do that also. There are advantages and disadvantages depending on how you want to analyze that and what you're interested in. So let's try that all. What I claim here. So I want to hear the dash F first for following. No, this is on the next slide. Okay, but with dash E to make things a little bit, I can say I only want to see read and write system calls for example because otherwise we'll just spam it with a lot of things then it's easier to see what I'm typing. So this is the hello, read a character, write a character. And if I press enter, you only see that it's writing to standard error or there was an error, nothing else happened. Now if I do the in, oh no, what's that back? So just return. If I do echo, hello, and press enter, we see the string hello from echo and it returns and does my prompt. Also understand, yeah, just the echo is executed. If I do, backslash echo. Oh, I forgot to mention. If you have any questions, just shout up. I will not look for hands if anything is very unclear. Maybe I can explain immediately because we are running out of time anyway. If I do this. Oh, okay. Just jump in. So someone online and Bean is asking, is it possible to get the file name through FD and write FD with S-Trace? The similar answer is yes. The second answer is look up in the file. I also only learned that two days ago when I was looking at Manpage. Maybe I will show an example later. Oh, it's doing still echo. If I do in echo and don't do any tables and I press enter, there is no output anymore of the hello. What happened before is echo is an internal bash command so it can execute the thing and write hello to standard output in terminal and S-Trace sees it because it's tracing the bash and all the internal things which bash is doing internally. The backslash I thought might also influence not going to the internal commandos. It's not. So it's only disabling allies and things like that but because I'm also showing that type. I have to know which types of echo we have. Echo is a shell built in or you have it in the path. And the second example was now if I run echo with an explicit path, it's not tracing that because I did not start S-Trace with the dash F and I only see what the bash is doing. So if I have the full S-Trace, then I would see the clone. So I'm starting a new process. This is what S-Trace will output but I don't see what the new process is doing. I don't see that it's accept something, write something. If I wanted to see that, oops. Then this was the first time where the dash F option comes in. And now I can do the same in echo. And now I see this is from the prompt and what the command line is doing but then somehow I should be seeing this is the right from the new processes ID and if I do it a second time, there's a second right from again another process ID where all the other things have been hidden because I just trace only read and write outputs. And here you can see it's reading again, lib scene, whatever. You can focus even tell S-Trace to go only for file descriptor number one but this has other side effects which I don't like. The other options to getting these things selected because I'm a command line guy and Crap was invented 50 years ago, you always can do, for example, I know I only want to see you write opening brace or opening brace is also a magic character. It's much easier doing it like this if you do it. So filtering and using command line on S-Trace output is quite common to me and then I see it's a new process. It's always writing boring. Okay, if I'm not interested in keeping that but I want to have the file and date things then I can start with output files. Typically, time is short, my output files are always code, oh, output file, O-1, O-2, O-3. But do now this on the same example then it only tells me how. I'm controlling several process things like that and if I stop S-Trace again with control C now to it first then it got a small, very small output file and it's more or less the same output I had on the terminal. This can be nice. What I also can do is now using that two times dash F option so follow the child's but give me for every process separate output file. Doing this three times now and looking which output files I got. This is the first one I got for. Ah, the first output file is for the shell itself and these are for the three echo things. While this can be useful, you will see in a few minutes when I come to the timing options. Time is the topic for this talk. So, okay. Yeah. It's on the slide, I need that only for... I mean, yeah. If I type sudo and it prompts for the password in terminal, will this monitor also that what goes through there? Yes. Okay. No, ah, okay. Now I got your question. If I would S-Trace sudo whatever. Did I get it right? If I do S-Trace sudo. If you S-Trace the terminal and in the terminal somebody types sudo. Now sudo is still different in the early days. The point is the sudo is a set UID script and it's switching to sudo. If you're not in sudo mode, let's try me without S-Trace. I thought maybe I still have the password. Ah, okay. Then it was working, did I get the right output? What typically happens if I... I think my idea was if you could use this to spy passwords. And if I would do the S-Trace on the sudo ID, something asking for the password, it's not doing that anymore because it set UID root and you cannot as a regular user trace debug whatever the data root programs. That was a habit 30 years ago and it was a nice hacker attack to send systems. Oh, I can debug that and just trade it. They learned that it's not a good idea that regular users can debug and trace to things that's not allowed and so that way. Yeah, but you see the password, if I do SSH somewhere being asked for a password, it's in your log file. So be very careful, read the articles, be very careful. If you keep log files along when you trace things, you also can see that S-Trace is opening your local private key files if it's trying to connect to remote. A lot of things which can be sensitive and you read your own files. So log files have sensitive data and many means if you do that on user processes. Oh, but back to the real topics today. Shaving of output. There are mostly two or three important things. You can change the string size if you do S-Trace, S-Trace, cut, ADC pass, W, whatever. I don't want to see the things really, definitely none. Then you would see, ah, I'm reading something from the password file. It's trying to read up to 128K. It got 4,000, 5,800, whatever bytes, but you don't see them all. S-Trace is limiting strings out that the output is more readable. It's also doing that for file name, path name. So even if you cluster S-Trace, which files are being opened if I run my favorite edit or whatever? Because one of my very favorite things is I want to figure out which config files are used by tool. And then all the path names are chopped. And this is quite bad. And if you want to see the full content, the full file names, so it's good to giving some larger strings, depending on what you're doing. Triple nine is good sometimes, as you have seen. This allows 128 kilobytes in a single read, and you want to see the real 128 kilobyte in the read call in the S-Trace output. Just give a few more nines, and if you do that, and then you can see here, blah, blah, blah, blah. It starts somewhere up. This is the read call on file descriptor three, reading a lot of bytes, and so on. It said, yeah, I tried to read out many bytes, but it's the last entry password file. And then, surprise, it does a write call with exactly the same data. So, increasing the string size is always the first thing I do, and the second one, which might be interesting, quite a number of system calls provide additional information. How is this trace? And for that dash V, being more verbose can be quite interesting. Let's do it like this. I don't want to, then you get a lot of things about the, not yet the file descriptors. This will be the next option. How can I show you the, if I see the X, oh, right. Where it's mostly useful for me just on the exec command itself. The exec command, before we have seen it, it's calling bin pw or bin echo, whatever. And it's the parameter, which is on the command line. If I only do echo, it will not give the full path. But then you get the full environment, for example. And also the full environment can be chopped out. That's a lot of dot, dot, dot, dot, because those environment variables are a little bit too short. So then again, as I want to see the full environment. And this is then a nice thing, if you trace a full bash script and whatever and have a lot of xx, you can see all the details, how the path looked like and whatever. And a lot of other things, the IO control you see, which IO control things are being done and other options for that. Oops, there's a three, five minutes, that's very bad. Just to show that there's an option x, we don't need that anymore. Giving this track strings in hex output, I was looking for that for a while. That's only printing hex output backslash x, whatever for strings or byte sequences, which are unprintable, but still my write and read and things are all readable. And if you double that, then everything is being in hex. Can't be quite useful for non-printed characters, if you're quoting, if you want to reuse that output. Read the slides, oh, god. So, giving hex output as 99. Now it's about what I just thought a little bit, selecting what I'm interested in. So typically what I try to show for the timing, just reading and writing, for example, from a serial device and see when data come in, you can say I'm interested in all the open calls. Long time it was just easy saying minus e open, or you can do that with comma, I have a lot of these things. These days, if you do S trace the open calls, typically you don't see anything because most of the time it's now open 64 and whatever. And if you don't know, and in stud it's even more difficult because you have the stud, stud 64 and stud whatever. And long time ago, this is what I learned in a talk from Dimitri and Fostam. He made regular expressions. You can say slash open and any call with open, if there's a, I don't know, socket open, it would also trace that one, with trace all the whatever is calling stud. So this is quite useful for me if I want to have things, e slash stud, I don't know how much. It's a new FS stud, not what I was looking for, but if I can do the LS, for example, if we see some stud LS dash L, it has to do a stud on the files and you can see stud X to get all the file attributes. And you will see some of those things that are still, you get more information about file attributes and everything. If you do the dash V now, a single file attribute is giving more, all the bits and bytes are now in ASCII version, if possible. That's quite useful. The most common use for me is dash E file, give me all file operation, all file access, all system calls which have file names, reading, writing, no, sorry, not really, but opening, moving, things like that. So if you're interested in what does this process that script do with any type of file which is trying to access, you also see which files are not existing, this is very helpful. So typically I use that if I try to figure out where a tool, where I don't have docs are, where is the config file? Because you will see an open or access attempt to a file name which you have not on your machine and then ah, then I can figure out what to write in there. Timing, this is now what I planned to talk about. There's only three options, it's quite easy. That's lowercase t, uppercase t and r. And let's do that. S trace, t gives you a timestamp for every system call. By default it's not too exciting, it just gives hours, minutes, seconds and if you run it for a long time you don't even know which day it was. And as the slide already say, you can double that. Since before it was all the same second so it's on a very coarse scale with double t you get at least the seconds and microseconds and what very often I use is triple t. Then you get seconds since Unix time in front with microsecond resolution, that's quite useful. Since really the Unix seconds since 1970. You have an absolute time, you can compare that also with other timestamps from other tools in your computer, I'm just on a quest where something like a web interface on the embedded controller doesn't work anymore. And then you can compare, merge these timestamps with TCP timestamps and if you have all computer connected to network time synchronization then you even can go for comparing that with the remote host and share that, merge that in, you just have to sort all the files and you get where it comes from. In the old time, the dash R does relative timestamps, it can be quite useful. We have to extend that, sorry. Then you get just relative timestamps in the last call now in that thing it doesn't make a lot of sense. And in the old S trace version it was not possible to trace both absolute and relative time. It was very surprising to see if you now combine those it's possible to see absolute time and the relative time. And even uppercase t always was possible. Nothing happened here. You get this timestamp is when, because it really prints from left to right when the kernel is called and S trace nodes are we're calling close, this is a timestamp when the call starts. And in the very back, the uppercase t gives you how many microseconds the system spent in the kernel plus minus a lot of overhead for all this S tracing things. So this is not too exciting for these examples, but we'll have to show that. This is why I brought these wonderful embedded devices to have some real sensor data and the FGTY is just at least one. So okay, this is the wonderful white thing which I got yesterday evening. It's an ESP32 running RISC5 rust binary from the wonderful field as rust, whatever. And I noticed it's really taking only very minute. I don't know, I haven't seen it stop sign. Caps lock is fighting against me. S trace do just relatively timestamps for example. And then you can see one of the time steps are the green thing is because the output of that thing is green and we don't like the output of what's coming out here. And then we see it takes 1.7 second from the read. It's between read and write because it's taking a while and we can also do some other question. I haven't plenty of time. It's your lunchtime. I will start talking until the switch of power supply and the life that happened in one conference one time because that you, what's going on. Now it's getting interesting things here. You can figure out what you can see here is for example, it's staying 1.7 second in the read call waiting, give me data from the zero interface. So if you want to know when the data really comes you should do absolute timestamp plus that. Use your favorite scripting language, whatever. I don't know an option where I can get here an absolute timestamp. It's always, I state that sometimes one of the gadgets is in GPS receiver you really can do that's, don't know which one. ACM zero or one, don't know. This is the temperature sensor, acceleration sensor. This will be the GPS thing and the GPS will provide timing data. It doesn't have any data. So no, it's not receiving any satellites here that's a bit pity because then you would be able to see if your clock is very well synchronized. You either can check the timing of your external things and see with your computer clock if you would have seen any satellites I would see at least the day of time here but it's not doing that. And but we can do it for the other sensors and now if you start the thing again. But satellite again, ACM one. So this is very simply giving a single line of some acceleration sensor thing. It's doing reading and writing providing that once per second that was contributing once per second. You see it's not once per second but every 0.64 seconds. And then you, shall we stop? Yeah, yeah, I want to start. It's all about you. If you have questions, they can answer. Yeah, I'm still here. I won't run away. Just a little command of fun of the last slide replaying process. It's also what's possible with S trace. So it's not only looking like C code but for the counter called you really can take. Let me back to my show, come on. So the tailo things, hello. Oops. So that right hello something, can you see it? I can't. Why do I do it with time? This is really real C code except for all we are returning something and what you can do is cut right that in. I've prepared the download. What did you tell me about hello.c? If you just take this single line, remove the, oh, we're returning 13 by to make a semicolon and do main and whatever and say make hello, compile it. I think even that is prepared. Then you execute exactly that color call. And if you do the same, now with a larger example, you get all the writes and S traces and whatever and then you can do something like PLS trace, oh, this one for example. This is the S trace of PDF latex running exactly my PDF slides for that talk and I'm not really sure how this is now going to, oh, should pl.c, not sure. And you can just include something like that. Just do the whole write things, remove everything at the end of PL. No. Oh, there's one line with the equal whatever. Oh, it's only the S trace PDF. What is it really called? Something with P, C and C2, C2. How do we take C2? And yeah, I wrote it small. If you do the relative timestamps, you can say, who sleeps this relative timestamp in microseconds. I thought that I have an example without you sleep. If you did all together and I hope I didn't mess up too much now I have to do that back. And make PL and you now can run this program. It's always doing a you sleep from the relative timestamp and doing the write and doing the you sleep and write and this is exactly more or less in real time plus minus a little bit of overhead that you can see doing that. And long time ago I was doing that for fun, for colleagues in my old company because they look for some opportunity having an hour or much hour compile. Just keep it running and look like compiling that your machine is busy while you're not sucking up CPU time. But you also really can do in real time what you trace, have the same time pattern again. And I had once a use case where with a specific time pattern things happened with other specific time patterns did not happen. You run a tool multiple times you realize, oh, there's something going wrong on a remote server thing. Then you can look up all the timestamps or whatever and replay more or less the same thing and doing nice test cases also with that one. Lots of option and I'm very sorry that this didn't work out better. I think I have to write a second paper about all the timing fun, how you do these things, how you do timing analysis now, how could your clock in the embedded devices if you have the GPS receiver with very precise timestamps coming in you can see how put your clock is in here. In both cases you will see that you have a temperature drift and especially for the embedded things it's getting warmer and the clock is, even it's a quartz, it's not really good. And tons of interesting things just playing around with time as a physicist. Sorry for taking too long for the introduction. P3 come forward and ask for questions and I'll answer around them.