 There it goes. Hello everybody and welcome to our second talk in the 101 Essentials for Embedded Track. The next talk is going to be by Mike Anderson. He's a longtime speaker at the Embedded Linux conference and other Linux Foundation conferences. He often ends up giving far too many talks but we've managed to get him in to give this talk today on GDB on embedded devices and make sure that when you have many questions that you write them in the Q&A and he'll answer them at the end. Otherwise take it away, Mike. Thanks, Ben. Let's see here. Let's get going. Obviously, I hope that everybody's had an opportunity to at least download the charts. I think they're available online. If not, they will be available shortly and with any luck you will be able to get those and then have them for your very own copy. This is me, Mike Anderson. You can use my mail to there, mikeandersonptr at gmail.com. At the time I wrote these particular charts, I was unemployed, actually between jobs. I just started a new job last week so brand new on the job with the Aerospace Corporation. This is me. I've been in the industry for about 42 years, 20 or so of those with embedded Linux. I used to be chief scientist of the PTR group and many of you who have been in my presentations in the past probably know me from that particular role. I'm now an embedded systems architect at the Aerospace Corporation and I continue to be the mentor or a mentor for First Robotics Team 116, Epsilon Delta out of Herndon, Virginia. If any of you are interested in robotics at all, I highly recommend getting involved in First Robotics. It's a great way of being able to help the kids come up to speed and I look at it from my perspective. It's one of these days the software that those kids wrote may be running my ventilator and believe me, I want to make sure they did it right. So that's an important thing for me to make sure that they're all up to speed and embedded and are doing the right thing for us. LinkedIn, obviously if you want to get in touch with me on LinkedIn, you're more than welcome to do that as well. But enough about me, let's move on to talk about what we're here to all chat about, which is GCC and GDB and all the tools that go along with debugging and profiling inside of the kernel. Now, obviously part of the problem that we have with Linux is it's not that we don't have enough tools, it's that we have way too many tools, oftentimes with an overlap, a considerable overlap between some of the tools. So if I for whatever reason leave out your favorite tool, I appreciate that you can be upset about that. But unfortunately, we only have so much time to work with here. So we will press on and hope that we can cover most of the things that most people are interested in. We'll talk a little bit about the GNU project and GDB and where it came from and what we need to do in order to compile for debugging. We'll get into some special modes of GDB, in particular one that not very many people are aware of and that is the text user interface mode or the TUI mode. We'll also talk about some of the GDB front ends. We'll then dig into how we get help, some of the scripts and macro mechanisms, what we have to do to go about launching and kind of running applications within GDB, attaching to a running application, how we set breakpoints, watchpoints, trace points. We'll get into GDB server, which is a very useful tool for being able to debug code over on an embedded platform. The embedded platform that I have for use today is kind of a special one. It's the Udo Bolt, which happens to be based on the new APU from AMD. It is an X86 and it also has kind of an Arduino built into it as well. So it's kind of an interesting little beast, but that's what I'll be using for my embedded platform discussion when we get to GDB server. Then we'll kind of switch gears from the debug perspective over to the trace and profile perspective. We'll look at S-trace and L-trace, GPROF, GCOV, Valgrind, LTT, F-trace, and we'll talk a little bit about Perf, which is relatively new-ish. The issue that I have with Perf is that it only runs over on the Udo Bolt because I'm running a custom kernel on my platform. And as a consequence, it doesn't like running Perf out of the box, but then's the way it goes. So let's move on and start talking a little bit about debugging tools. Now, when we look at debugging tools, they basically fall into one of two different classes. Either they're a debugger like LLDB or GDB, live recorder, etc., or they're some sort of a checker. Now, in particular, checkers and profilers are focused on trying to find bad things, stack overflows, misuse, double freeing of memory, those kinds of things. How we're using the caches, address sanitizers, and even some of the static code analysis tools like Covarity kind of fall into that category, because they do have the ability to look at threading possibilities, the threading interactions and go, well, there could in fact be a problem here. And they'll try to bring it up and try to deal with it at something a little bit before you actually get into the debugging tools, before you actually get into the runtime stuff. So they tend to be a little bit on the static side, but that's okay, because we need to be able to do those too. Certainly, tools like Covarity have really helped tremendously with the Linux kernel, so that we could in fact get some, eliminate a lot of the potential race conditions and things of that sort that have existed in the kernel in the past. Now, with the GNU project, of course, GNU stands for GNU's Not Unix. It was originally developed in 1984. It was actually put out there in 1983, September 1983, it was proposed, and then the work didn't start until January of 1984. But one of the first processes, one of the first things they wanted to do was to build the tools that were required to build an operating system that would be a completely open source, free operating system that was very Unix-like, but was not Unix, and that's where the GNU's Not Unix came from. At first, there was just the GCC compiler, and then that was focused largely on just C. Then later, we got G++, which was C++, and now that's kind of finally manifest itself as the GCC with a capital GCC, and that's the GNU compiler collection. Now, the way the GNU compiler collection is built, it's got a front end that allows it to interpret the language and convert that into a back end, which will then actually do the compilation. We see front ends for C and C++, subjective C, Fortran, Ada, Go, etc., etc., and of course, libraries like libstandard C++. Those are all part of the GNU tools, and it's architected so that we do have that option of being able to work in terms of kind of a front end that's built to run on one machine and a back end that's built to run at a separate computer. For instance, we would have something that was x86 cross arm or Windows cross Linux. All of those kinds of things are different possibilities that exist with the GCC tool chains. Of course, as opposed to GCC, we've got LLVM, so there's lots of great compilers out there. I'm going to be focusing largely on GCC and the GNU compilers at this point, and GNU debugger rather at this point, because that's pretty much the topic of this particular talk. Perhaps I'll do an LLVM debugger at some point in the future, so that would all be kind of cool. Now, let me just go ahead and yes, the slides are available for download. They should be on the website. If I had failed to mention that earlier, if they don't show up on the website shortly, you can always email me, and I'll be able to get a copy as well. Now, the GNU debugger, GDB, it was built as a source debugger for the GNU compiler collection, which means it actually supports many, many different languages. This is one of the things that most people are not quite familiar with with GDB. GDB actually works with ADA and C and C++, obviously, but it also works in D, Fortran, Go, Objective C, Modula 2, Pascal, Rust, and several others. It is a very flexible debugger. It is not simply limited to C and C++. It actually works across the GCC, across the GNU compiler collection, into quite a nice debugger that works across multiple platforms. Now, when we get ready to compile something for GDB, we'll take a look at the command line debug options. When we get ready to build the code, we will pass a special compiler flag. This compiler flag is kind of ubiquitously known as the dash G option. However, there are more than one dash G option. We range anything from dash G0, which means don't do anything, which, I guess, makes more sense if we're going to make file and we're passing this as an environment variable. Because why would you go to the trouble of actually typing dash G0 if you didn't have anything out there to do if you weren't really trying to compile the code? Dash G1 includes minimal information, basically backtraces, but no information about local variables and line numbers and things of that sort. Dash G2 is the default debug level. If you just specify dash G on the command line, you will get G2. G2 produces symbols, line numbers, basically everything that's needed for symbolic and source debugging. And this is, like I say, the default if you use the dash G option in the compiler. But that is not all. It turns out there's a dash G3, which includes additional information such as all the macro definitions that may be present in the program. This is particularly important when we're looking at places where we're doing some large substitutions of macros. Like, for instance, in the Linux kernel, there are a number of commands that look like C commands, but they're actually inline assembly language commands. So all of those would certainly be something that could potentially be a macro expansion that we'd be interested in knowing. And we could see all that using the dash G3 option. But, you know, all of these options, these first three options, G1, G2, and G3, all produce output in what's referred to as dwarf format. And if you look at the output from the compiler, it's called ELF, embedded in linking format, or executable linking format. And then dwarf is debugging with attributed record formats, or so that's dwarf. So we have ELF and dwarf, very much a kind of Middle Earth sort of flavor there. But the mac daddy of all the options is dash GGDB3. This is like dash G3. However, it actually generates debugging information specifically for GDB, rather than the normal cough or X cough or dwarf to format that you come out with the dash G. So dash GGDB3 does in fact produce more information in the debug output, and it's specifically targeted at being used by GDB, rather than one of the other debuggers that are potentially out there. So here's an example of a compile for GDB. We have this ARM Linux GNU ABI GCC. We're using the dash GGDB3 option here, dash oh hello, and we're doing the compiler, the ubiquitous hello world. The example, if we were to take a look here in the ELF header, and we were using object dump as an example to show us all these things, we see all these things that begin with a dot debug. This is the code or the sections rather that get output when the dash G option is turned on to the compiler. Now, the nice thing about all these things is they are kept completely separate, and they're kept separate in the executable. Therefore, the code itself, the text segment in the executable does not get adulterated with debugging information. This is really handy because what this means is I can actually debug code on the target that has not been compiled for debugging as long as I have a version of the code that has been compiled for debugging online. So I will load the code compiled for debugging into the debugger, but the code that's running on the target does not have to be that code. It just has to be, well, obviously it doesn't have to be compiled for debugging, but it does have to be the same code base, obviously. Now, that's an important factor here because when we're dealing with extremely low memory footprints, and I have run Linux on a two megabyte machine before, it was a MIPS processor where they had already purchased all the RAM chips, and you couldn't get any more than the two megabytes that they had. When you're trying to run Linux in a very small memory footprint like that, you do not want to put anything into memory that has any of the debugging code in it because the debugging code adds about 30% to the size of the executable. So that's a very important thing to keep in mind, and we'll see an example of this when we get into a little bit of a session that I'm going to show you a little bit later here. So now we can run GDB from the command line interface. Now, obviously when GDB is built from the source code, we have to specify the host and target information, and the host and target information will specify whether it's Windows cross Linux or Linux cross Linux x86 to ARM, et cetera, et cetera. So here's the kind of output that we would expect to see from this kind of a compiler. We're going to output here, we're just going to run the GDB interface, and we'll see that this was particularly configured as x86, 64 Linux GNU. Now, if we had a cross compiler built on here, then we would see that it was configured as x86, 64 as the front end and potentially ARM as the back end. We did have a question come in, what's the advantage of G1, G2 if G3 does even more, especially if you can strip the executable from running on the target? The issue is that the more optimization, the more stuff you turn on here, especially if you turn on optimization, optimization rearranges the instruction stream. So if you turn on a very high level of debugging and a very high level of optimization at the same time, the debugger can handle it fine. Humans can't handle it too well when you see the instruction pointer jump backwards. So it's a little weird. It doesn't mean you can't debug it. It just simply means that you have to be mentally prepared to see things go backwards. Also, if we were to turn on all of the debugging options and we're running the debugger and we're actually interacting with the debugger, then, yes, having on higher levels of the debugging flag turned on will in fact slow the process down a little bit more because it's starting to keep track of more things. That does not necessarily mean that it's going to be like even three or four percent difference. It's going to be relatively small. So you're not going to see a whole lot of difference here because we really don't need to do anything to the target as far as being able to turn on the debug flags. We just need to make sure that the debug information is available to the debugger. So the text segment and the data segments and all the other important running segments of the application on the target don't change when we turn on the debugger. But it does change the size of the code. So when you start turning on debug level three, then the code is going to balloon up. So if you are just, if you're not really thinking in terms of being able to run the strip command and do all the things that you would normally do before you deployed the code, if you're just in a quick debug, compile debug test kind of model, then people don't like to have to go through the extra steps of doing the strip command and then doing the copy. It's just a little bit easier for them to just simply do everything inside of one make file. Now, personally, I would then put the strip command inside the make file, but we really do need to have the fully debug-enabled version of the code for the debugger. We don't need it to be the run on the target, but we do need it for the debugger. So it does get a little bit complicated and oftentimes it's just easier to not mess around with it. But fortunately, it really just changes the size of the executable as it exists on the disk, not so much the change in the runtime. All right, so if we wanted to run GDB in text UI mode, running GDB in text UI mode, the tui mode, uses curses. So obviously, if you don't have curses installed on your platform, it's not going to work all that well. But what it does is it then gives you the advantage of being able to actually see the code interact and operate almost like you had a full up GDB GUI. So for instance, let's go to the screen share here and let me see if I can get the right screen selected. So with any luck now, you're seeing a screen that shows my command line here. So let's go ahead and we'll run GDB. First of all, I had mentioned that we have the option for building GDB with different compilers. So if we took a look here at this output for ARM Linux, you'll see that the GDB was configured as the host is an x86 PC Linux GNU, so it's a x86 based host. But the target is an ARM processor running with hardware floating point turned on. So that's the kind of thing I was trying to show you a couple of pages ago, where we will configure a front end for GDB and a back end for GDB. Now, inside of GDB, I can automatically load commands. Let me just go ahead and I'll quit out and show you here. So we do GDB and I have a command. Let's just go ahead and compile it just so that we've got it all there. We'll do a GCC dash oh hello and dash G will turn on just level two. That's okay. We don't need to turn on any more than that for most cases. And we're going to compile hello.c and then we will run GDB and we'll run it in tui mode and we're going to look at hello. And now we see what it looks like when you're running in the text-based UI. So we have the upper window here, which is the source code and we can scroll around in the source code. We have the lower window, which is the command line. And inside of this, we actually have several different options. We can use the layout command. And if we just see layout next, it will then show me the next possible layout. In this particular case, I have the assembly language output. I can do layout next again. And now I see the source code on top and the assembly language. And then let's see here if I can get it to do this again. Registered values. I'm not running it right now, so there are no register values available. And then let's see here. I think it I want to see I think it's got a couple of oh then it goes back to the original one here. So if I wanted to just simply do a kind of a quick run of this, I can just tell it to run. And if I tell it to run, it will run to completion because I don't have any breakpoint set yet. But we see hello world and we see the value of i and now value i has been incremented. Notice one thing here. It says inferior one has exited normally. That is not a comment on the quality of your code. That is a comment on the way gdb sees your code. Whereas gdb is running it from outside of the code. It sees your code as an inferior process is something that is subject to the control of gdb. So in this particular case, we just ran it to completion and then it exited and that's the kind of message I would normally expect to see. There are a few other things that we can actually take a look here. We can do an info on the number of windows that are out here and this will show all the different windows and their respective heights. I can then specify the height. Let's say win height. Let's do source and we will subtract two from the source code. There it just changed the amount of window displayed by the source by two. And if I happen to then be looking at things like 2e, let's do register. And let's do register. Let's take a look at the floating point registers. And obviously it's not running. So let's do this. Let's do break at line 14. And then run. And now we see the floating point registers. So we can also see the general registers well. 2e, reg, gen, I think will allow us to our general, rather, general. And we can see the floating point register. I mean, we can see the general purpose registers here. So we see ax, bx, cx, etc. We also see the magic additional eight registers that were added to the x86 instruction set some time ago that you still don't see if you're in a 386 or 486 world. But it kind of gives you an idea of the relative flexibility associated with what you can see inside of just looking at it in the text UI mode. Now, as we move forward, we start looking at full up GUIs. Now, there are a lot of standalone IDE based front ends for GDB. The one that is the official supported front end is the data display debugger or DDD. You see a picture of that in the upper right hand corner there. It is very much like 2e mode except that it is an actual window that floats there and you can move things around inside of it and you can resize it. And it actually has quite a bit more flexibility for being able to display pieces of information like being able to see your watch variables and data structures and things of that sort. So DDD is a really nice little interface. Considering its age, it still holds up pretty well. Obviously, you can also use front ends like Eclipse. You can also use Microsoft Visual Studio Code. Microsoft Visual Studio Code is actually open source. It is not Visual Studio. It's the editor for Visual Studio, but it also has quite a few plugins that are available for it, including a GDB and an LLDB plugin. So we can actually do debugging using either the Clang compiler or using the GCC compiler. There are other ones, of course, KDevelop, SlickEdit, Code Warrior, many, many others are out there. So it really becomes a question of what your preference is in terms of which debugger interface you happen to like best. With the DDD front end, DDD supports JDB, Python, Perl, Tickle, PHP, and we can also specify a different back end. So for instance, if we are getting the dash debugger option, we can specify ARM-Linux, GNU-EBI-GDB and MyApp. And with that, we'll be able to run a different back end with the DDD front end that's running on the x86. The back end was running ARM and we'll then be able to reach out and contact, reach out and touch a different application to run here. And we'll show you how to do that when we're using things like a GDB server. And I'm getting some reports on the audio. Let me see what I can do here about trying to turn the audio down just a little bit. Maybe we can back it off a little bit. Hopefully that will help. Cool. All right. Excellent. Okay. So let's move on. Here's an example of the help output from GDB. So GDB has a very complete help inside of it. If there's any piece of information that you need to be able to get about how to run something within GDB, the help is actually quite complete. So we can ask for, for instance, help on break points and it'll then tell us everything we need to know about break points, trace points, watch points. It is very, very complete. It's like having the entire manual online and being able to pull it up at any point in time. So that's really quite handy at being able to see all of this. Now, as far as commands and definitions are concerned, one of the things that a lot of people are not quite familiar with in GDB is its ability to create your own commands and your own scripts. If you have a series of commands that you type frequently or if there is a long series of commands that you have to use in order to get started on a particular debugging session, you can actually define those as a command sequence within the debugger itself. And what we'll do with that is we will then define the name of the command that we want to define. And then once we hit return on that, it'll then prompt us for every single line that we want to type in. When we're finished, we'll type the single command end. And once we have done that, that will become an official command. So we can also then turn around and document those commands. And of course, all of us are low to do documentation. But as it turns out, we can in fact document our commands so that somebody else comes along and we can then check these commands into git. And then somebody else can check them out and have the same debugging tools and the same debugging scripts that you had used when you originally developed the code so that everybody kind of understands and they're all in the same, you know, basically all the same playing field. We can also go in and define c and c++ preprocessor macros. So these get into the macro define command. These are going to be visible to all of the inferior sources files. So if I'm trying to debug an application, and it has a pound of fine inside of it someplace, we can actually go in and manipulate that. So let me show you an example of how that's done. Let me go back over here to my screen. And we have an application sitting here. We can define a new command. We can see let's define a command called Fred. And then it's going to ask me to type in a whole series of commands and then type in as the last one. So let's just do a print of I, that's our variable that we're messing around with in this particular program. And then we'll type in. And then from this point forward, all I need to do is just type command Fred, and it will then tell me the value of I. If I then specify that I want to step over, so let's do it next. And control L by the way automatically refreshes all of this. I will then do next again. There we go. So now if I type in the word Fred, I have the value one because I just incremented it up here online 16. So by being able to define your own commands by able to define the documentation for those commands, then if somebody said help Fred, they would then be able to see whatever documentation you had provided for that particular command. So it's kind of a clever sort of option here. Now one of the questions came in, can we set VIM to be the editor of choice in 2e mode? I've not seen anything that allows you to do that, but I wouldn't be surprised if there was some mode that you could put it in that would either be Emacs or VI. But I haven't been looking for that quite frankly, because I don't use 2e mode except when I'm communicating to a target. And the target doesn't have any GUI capabilities on it. So if I have just an arm target sitting out there that just uses a serial port, for instance, or just uses SSH over the ethernet, then 2e mode is really handy to be able to do debugging on that platform when I know I don't have the space, for instance, for having all the X-Windows libraries and things of that sort. So that is there a way to do it? There may be. I don't know what it is right off hand, however. All right, so let's go back to our code, or back to our charts here. We also have, in many cases, when we're trying to do a long debug session, we'll go home, we've had dinner, we come back, we then want to sit and we want to pick up where we left off, or we want to pick up and start everything over again, but we want to be able to key in a whole sequence of commands to get us in the right place, having loaded the code, having set the breakpoints, everything else that we need to have. So this is where the .gdbnit command comes into play. If you have a .gdbnit in the current directory, then basically what it will do is it will execute all the commands that it finds inside of there before it gives you the gdb prompt. So for instance, commands like setting the history save on, setting pretty printing on, pagination, confirmation, turning those off. These are things that you will normally do inside of a gdb session. So rather than having to do this time after time after time, or just copy and paste between your environments, it's easy just to put this in a .gdbnit, and every time you start gdb, it will automatically load these commands, and you're off and running at that point. There is also a command, the dash x command line option that allows you to run scripts at gdb load time. So you can then specify, if you don't want to use .gdbnit, if you had your own script, for instance, then you could specify dash x, name of the script, and the application that you wanted to run, and it would automatically start gdb, load the application, and then execute the scripts so that everything was all set. You'd have your breakpoint set and everything else. Additionally, inside of gdb, if I need to be able to get out to the command line, I can use the shell command, and the shell command will then shell out to gdb out to the command line, and then execute whatever command you decided to do in the shell. So that's a really handy thing as well. Now, one of the things that is kind of a surprise to a lot of people is that gdb actually has an embedded, integrated Python engine inside of it. It doesn't actually just shell out what it does. There's actually a Python engine embedded inside of gdb. So we can execute Python expressions if we just simply prefaced them with the command Python, space, and then whatever Python expression we wanted to use, we can actually then execute Python commands. We can also write Python programs inside of gdb using the Python interpreter. Now, we see here a few examples, Python gdb.execute, that's going to execute some gdb commands. We can parse and evaluate things that's basically getting data from the inferior. We're basically going to reach out to the command line options. If we wanted to just get all of the possible things that Python can do, we can do Python help gdb, and that will then allow us to see all of the commands that are possible inside of the Python gdb package. But let me jump out here and I'll show you another example. And let's see if we can do this. So here at the gdb command prompt, what we can do is we can execute a command like Python print hello. There we go. So we just ran the Python, that Python command, nothing particularly magic about that command obviously. But we can then go into interactive mode. And whoops, let's not do that. Okay, so let's type Python. So now I'm in Python interactive mode. I can import class libraries. And then I can execute commands that require those class libraries. And type end. And there, it just ran it. And it said I am PID and it did the os.getpid command. So it's a way of being able to even get to the point of defining Python shortcuts that we can define as commands inside of the debugger. So just as we talked about the define command, we can also define commands that allow us to define Python commands and then execute extensive Python operations inside of the debugger. Very handy for us to be able to kind of mix and match. And many of us are very familiar with Python anyway. So it really helps to be able to pull that information and be able to get access to the Python interpreter without having to jump out of the debugger environment in order to get to the Python commands. So terribly handy. Then if we take a look at, of course, the bane of a lot of developers is the signal mechanism. And GDB is built on top of Ptrace, the process trace command. So when inferior gets a signal, the inferior is then suspended and the tracer gets notified. So what's going to happen, if we are interested in what signals we have, we can use the info signals command and that will print a table of what signals are out there and how GDB will handle each one of them. We can also then do an info handle and that will come back and tell us what those info signals, what that actually meant. So we can specify no stop, which means don't stop the program, but print that signal actually occurred. Useful for things like sig user one, sig user two. We can tell it to stop the program when the signal occurs. We can tell it to print a message when it occurs or no print a message. We can keep track of whether the program is going to pass it on. So this becomes important when we're dealing with threads and the ability to deliver a signal to the parent, which then gets replicated out to the individual threads from within the parent. So pass and no pass, that deals with being able to pass those signals on to threads that may be running in the same process address space. And we can then change a particular signal handler. We can do handle signal keywords. We can then turn around and deliver a signal on demand from GDB. So if we wanted to send the segmentation violation, basically the bus error signal, we could then signal sig seg v, and that would then send the signal to the main application. And then obviously if we had the pass turned on, then it would turn around and pass that signal on to the threads itself. So we can generate signals interactively from within GDB and test our applications that way if we happen to have signal handlers. Well, let's assume that for whatever reason we decided not to load the application when we ran GDB. There is a way of getting around that particular problem. It's called the file command. The file command, we specify the name of the file that we want to load, and it will then turn around and load that into GDB just as though you had invoked it from the command line. That is an important feature when we talk about debugging kernel code. If we're using KGDB, KGDB, when we run a KGDB of say the kernel, the kernel may not necessarily have had your dynamically loaded module in it when the kernel was compiled. Obviously it's not going to have dynamically loaded modules. So if we're debugging using KGDB, then what we'll do is we'll use the file command to load the driver into the debugger so that we can then start setting break points inside of the driver. We won't get into the details of doing debugging in the kernel in this particular session. We just don't have time to do that unfortunately. But understand that this file command also has some additional options in there, which tell me where to load the text segment, the data segment, read-only data, etc., etc., BSS. Those are all important things when we're trying to mess around with either code that has to be loaded in a specific location in memory, like for instance, if you're using U-boot, if you're trying to debug U-boot, U-boot has to have its text segments loaded at very specific locations in RAM for the board. So the file command here will then tell GDB where that code was actually loaded. And that allows GDB to reach out, generate the right addresses, and be able to debug that code as it exists on the target. So that's a very important command. Also, we've seen the run command. I did that a couple of times here. But we can set the arguments to the run command. So we can run the application, we can pass arguments that way, or we can use the set args command and set the arguments so that when we use the run command, it automatically passes the arguments we're interested in. If we did a show args command, that'll let you see the argument so that you can then go, oh, well, yeah, that's correct. Or no, that's not correct. Let me go ahead and change them and do another set command. So you can run it, and you can change the arguments, and you can have all of that set up inside of your .gdb init so that you don't have to type all that stuff all the time. Now, one of the real powers of GDB is the ability to reach out and attach to an already running program. So when we attach to an already running program, that program may in fact not have been compiled with debugging turned on. This is where you combine this gdb-p option and the ability to use the file command so that we can then attach to a running application. When we attach to it, it will stop the running application at its current execution point. So if it's say it's a server, it's running in background, it's a demon, we attach to it. When we attach to it, we will stop it. It will then have a breakpoint, basically, set at whatever place it was when we executed the gdb-p command. Then we can use the file command to load the executable with the debugging turned on. That is the executable that has the dash g option that was compiled with it. So again, this is usually targeted at applications where we have a very small memory footprint, either a small storage footprint or small RAM footprint on the target. And we still want to be able to do debugging on that target. Now, this is usually going to be focused on cross debugging and not necessarily native debugging. If we're in native debugging, we typically have enough memory that we can work with it without having to go through all this trouble. But the nice thing is that when we use the gdb-p option, we can then attach to any running application and it will stop it. And then we can start stepping through it. Now, if we don't have the source code for that application, then what we will see is everything in assembly language. So when you're looking, certainly you're doing debugging of U-boot or some of these low-level pieces of code, it helps to know the assembly language for the application that you're working with or for the instruction set architecture or the application processor that you're working on top of. So it's definitely handy to be able to have that kind of information and, you know, you don't necessarily have to speak assembly language, but at least be cognizant of it so that when you step into a piece of code that you don't have the source code for, you don't wig out when you see nothing but assembly language out there. It's okay. You can keep stepping and it'll eventually turn back into C or C++ or whatever language it was that you were actually using. Now, when we get ready to examine code, once the program gets loaded into the gdb, you can basically list any of the source files. So if the code is made up of multiple .os that are all linked together, that's fine. All you need to know is the name of the file and the line number or function name that you're interested in taking a look at. So when we use the list command, we can specify a line number, a file at a particular line number, a function name or a function inside of a file or a specific address. So all of those things work when I'm using the list command. Now, obviously, if I'm doing a list at an address, I'm typically going to be looking at assembly language code at that point, but that's okay. Again, that's not at all unusual when you're starting to do low-level debugging at this point. So you can specify the number of lines to list as a second parameter. So if I set the list size of, let's say, 15, then when I typed list and main, it would show me 15 lines instead of 10 lines. 10 lines is the default, by the way. You can also change things like the radix. Oftentimes, we'll be working in hexadecimal, but gdb defaults to decimal. So if I put in my .gdb init, if I put something like set output dash radix 16, then all the output will be in hex. Terribly handy to be able to translate all this stuff into something that makes more sense from a computer's perspective, maybe not necessarily from a human perspective, but most of us who've been programming for a while, we can equally think in hex as we do in decimal. It's like, how do you count in binary on your fingers? That's kind of interesting. You get to 31 on one hand. It's really handy to be able to think in those terms so that when you're looking at output code, you can do it in the radix that you need to have. Something still use octal, and of course, you can also switch that over to binary as well. If you want to know what options are available, then we can use the help set, and help set will then show you all of the potential options that are out there. It says the screen share is out of sync. I'm not sharing the screen at this point, so that would be why it's probably out of sync. Now I should be sharing the screen. Let's do something like this. Let's do a show. It shows me all the different options that are available to me here. Of course, I have to then do this layout. There's another thing that we can do. We can actually specify the focus, so we can specify focus and command. Now the focus is set to the command window, so now I can scroll around inside the command window. Those are the gdb command. Let's do a set output radix 16, and now it's set to hex. That would of course be something that we would typically want to be doing if we were trying to dump data structures and things of that sort that would all be done in hex. Let's go ahead and stop that screen share and go back to the actual charts. There's another thing you can do, which is kind of interesting, and that is once the code is loaded, you can actually call program functions from within gdb. The function will be called and the return will be printed and saved in the value history. It will use the same calling sequence that the language uses. Let's just go back out here to the screen share. I probably terminated that a little too soon. If we go here and let's just do a layout next, and there we go. All right, so now we're back to just the screen, the text itself, so let's do a list. You'll notice I have a function up here called sub, which is just a subtraction function. It's nothing particularly magic, but it's just a way of being able to show you how the command works. Hopefully you can see all that. Let's do this. Let's call sub, and we will pass it two parameters. Let's do, we take the first and we subtract the second. Let's pass it five comma three, and it returns the value two. Again, nothing particularly magic about that, but the nice thing is it's like, well, okay, just call this function. What is this function's inputs? What is this function's outputs? You can call it interactively from the command line. That's a really handy thing to be able to do, especially if the function that you're trying to call is in another file someplace that's been linked in. You can then list the function, so we can do list sub, and it would then show us the sub, the subtraction function up there towards the top. We can do that across any file that's been linked into this particular executable, so we can find that as well. All right, so let me stop screen share there, and that's the calling function. Setting variables. If we want to change the setting variables, any function, for instance, we can do set i equals five, or whatever language you happen to be using, and it has, let's see, I just got a message here. So it seems to be good for some people. Some of you are seeing the screen share fine, some of you are seeing it with a little bit of an issue. A lot of people are seeing it fine, so I suspect that may be a local problem. Refresh rates and things of that sort, depending on your individual data feeds and how much bandwidth you've got going out there. So I'll try to make sure that these things are as clean as possible. I'll just jump in and do a control L periodically to refresh the screen so that everybody has the latest and greatest, and hopefully that will help. But we can set any variable to any expression, and obviously we can use any syntax that's equivalent for whatever language you happen to be using. We can also, let's say that I have a function called call, a variable called call. Well, that overlaps with the call function inside of GDB. So how do I get to the call function, how do I get, how do I distinguish between the variable and the function? If I want the variable, then I'll use the set variable call equals expression. So the variable expression there, if I use the variable designator in front of it, then that says find the variable called call, not the GDB function called call. So if I happen to pick on one side, if I happen to pick a variable, and obviously, when we're writing our code, we don't necessarily know all the reserved words in GDB, we may actually pick a reserved word as a variable. It's not a reserved word in our language, but it just happens to be a reserved word in our debugger. If that's the case, then we just use this variable designator here, and it will find the variable in your code by that name rather than the GDB function by that name. Let's see here, printing expressions. Use the print command, obviously, that will print variable for me. I can then use, and this is a really interesting capability. Let's assume that I just updated the variable I, and what I want to do is I want to look at the previous value of I. I can then use dollar sign and the variable that I'm interested in, and it will jump back one version. I can also use multiple dollar signs, and I can continue to go backwards in all of the different values that the variable has. Obviously, there's a limit at some point, but it does give me the ability to take a look at the value of a variable before it changed, which is terribly handy. It's not quite the same thing as running backwards, which GDB does have the capability of doing in some cases, but in this particular case, we're just saying, give me the value that this variable had before I changed it. We can also use the at sign symbol. That's a binary operator for looking at consecutive data objects. Let's say I'm dumping a FIFO out of memory. I can use foo at number, some address, for instance, and then it will show me all the variables that are at that particular location, everything that's adjacent to foo. It's also very useful for being able to dump data structures, things of that sort. Definitely something of some use. If we have a known memory address, we can also use basic casting here. We have the X command, which allows me to show memory at a specific address. We're going to display it as a byte. It's a decimal byte, and we're looking at the address of test. That will then jump out, show me the address of test, and what it is as a byte. Definitely useful if I need to be able to cast it as a d-word, or a half-word, or a quad-word, something like that, then very useful to be able to specify that from the GDB command line. Breakpoints, of course, one of the big issues with GDB is our ability to set breakpoints. We have to understand that there are several different types of breakpoints inside a GDB. We have the standard software breakpoint, which is just the B command. That will go into the code and substitute the trap instruction for the breakpoint at the location that you just specified, and then it will store the original piece of code off in a buffer someplace, so that when I reach out with the B command, I'm going to actually reach into the executable, and I'm going to change the location at that variable, or at that line number, and make it the trap instruction that traps to the debugger. For instance, in the case of the X86, that's an int three, which is the trap to the debugger. This requires you to be able to actually modify the code. What happens here, if I'm running out of flash, then the normal software breakpoints don't work, because in order for me to set a breakpoint in flash, I would have to read the flash, change the variable, and then write the flash back out there. This is one of these cases where that becomes very problematic, because I'd be rewriting flash every time I wanted to step in instruction. That's not going to work very well. In order for me to be able to debug things in flash, I would then be using the hardware breakpoint. That's the H break option. Depending on your processor architecture, not all processor architectures have hardware breakpoints, and not all of them have a lot of hardware breakpoints. For instance, I think in the X86, I think there are three hardware breakpoints. In some cases, a PowerPC, there are five, ARM has three or four, I don't remember which ones, but in any case, the number of hardware breakpoints is dependent on your processor architecture. That's a useful thing in the sense that when I want to be able to set a breakpoint at a specific location in flash, I can do that, and what will happen is when the instruction pointer matches that hardware breakpoint mask, it will automatically stop the processor at that point and drop you back to the debugger. Extremely useful being able to debug things like Uboot, for instance, or an application. We'll show you here an example where you know that some piece of code is stepping on a variable, but you don't know which piece of code is doing it, so you can set a watch point up, and we'll talk about watch points in just a second, but you can set a watch point up that will then automatically stop if a piece of code writes to that location, and then you'll know exactly which piece of code is doing the writing, so if you've got some case where something is being stepped on, you're not sure what it is, then use a hardware breakpoint and you'll be able to find out exactly who's doing it, at what point in time, and where that code is located. Very useful to have. We also have temporary breakpoints. Temporary breakpoints, they'll stop once, and then they'll be removed, so if you just want to stop it as it's coming into a particular function call, but the function is a loop and you don't want to set there in loop forever, so we can absolutely then take into account, stop it once, check to make sure that everything's good going into the function, and then allow the function to continue to run and it won't hit that breakpoint again. There are also conditional breakpoints. Now, when we start dealing with conditional breakpoints, things can get a little wonky in the sense that you're basically after every instruction, you're looking to see whether that breakpoint has been triggered, so this can substantially slow down the execution of code. If you're trying to debug a race condition, this is not the way to do it, because by turning on this conditional breakpoint, you're going to slow the code down enough that the race condition may not happen, so use this sparingly and only if you need to have a particular, if it's a short duration run and it's not going to cause you any specific timing related issues, then obviously you can use conditional breakpoints. If it is a timing related problem, don't use conditional breakpoints, use hardware breakpoints instead. They're a much better way of being able to do some matching. You have to be a little clever, because obviously you can't say break on line 55 if x is greater than 5. Yeah, that's a very handy thing to be able to do, just understand that that's going to change the execution speed of the code. The more of those things that you have in there and active at any one point in time, the slower the code will run, so use conditional breakpoints sparingly, obviously use them when you have to, but don't use them if you just because you want to use them only when you really need them. We can also issue a series of commands to be hit when a particular breakpoint is set, so if I do, for instance, here after the breakpoint in main, then execute the following commands, and then we'll type all the commands in and we'll end that last command with an end statement, and then every time it hits that breakpoint, it will automatically execute all of these commands. Absolutely handy for being able to hit a breakpoint and displaying a data structure, for instance, every time you hit this breakpoint, saves you a lot of typing. I mean, that's really the key thing here is to try and save people from having to do a lot of typing. Now, if I'm using one of the GUI front ends like DDD, I have this little dialog box that pops up over here that comes up. I see here the run command. I have the interrupt command, so if the application is running and I press the interrupt button, it will stop the application wherever it was when I hit the interrupt button. I have step next, so step will step one instruction. It will step one C source line, for instance, one source code line, but it steps into functions, so if I'm calling print and I issue the command step, it will then step into the print function itself. That, of course, you probably don't have the source code to, so you're looking at assembly language, and that's a fairly common mistake when we're just tinkering around with GDB and we're not necessarily used to all the functions. People will use the step command because it does step from one instruction to another from one source code line to the next, but it steps into functions, so if you're stepping into a function that you know is working correctly, then you'll want to use the finish command, so the finish command says run till you get to the return. Next, we'll call the next line, but then step over it, so if I was calling a function, my function sub, for instance, then if I used step, it would step into the sub function. If I used next, it would call the sub function, but not step into it, so we just simply say, yes, I know that works, go ahead and execute it and just continue after you've called that function. We have others, the continue command, if we've hit a breakpoint and we go, okay, I've seen everything I need to see here, press the continue button, it will continue running until it hits the next breakpoint. I can kill the function as it executes, so I can kill it off, it's doing something that I don't want it to do, so just go ahead and kill it. I can also use these up and downs, these allow me to go up and down through stack frames, so let's say I have main calls sub and I'm in sub's stack frame, I want to look at the value of a variable up in main, so I would use the up button, the up command that would go up into main's stack frame and I would see main's variable and then I could use the down command and it would go back into sub's stack frame, so this allows me to move up and down through stack frames, especially if I happen to have a variable that has the same name, but in different stacks, not a good programming practice, I would remind you that, but sometimes it happens, somebody defines a variable called debug and another team is also working in another portion of the program and they define a variable called debug, well when I'm in a function and I do print debug, which am I looking at? Am I looking at the sub function's debug or the main function's debug? This up and down give me the ability to control which of those stack frames I'm looking at and if you're a real glutton for punishment, you can use the step I and next I, the steps instructions, so one assembly language instruction at a time, either step or next, so definitely if you're into assembly language instructions, this is a great way of being able to see those. One of the things that I talked about earlier is the hardware breakpoint and one of the basis that makes the hardware breakpoint what it does is this thing called a watch point. A watch point allows me to watch either an address or a symbol name and stop the application when one of the several things is happening. I can tell it to watch it, if I watch it, then if it gets written to, it will stop. If it's red, it doesn't stop. If I do an R watch address or symbol, then it will stop even if the variable is red and then I have some other variants here that says stop watch if thread three is modifying it. So if I have multiple threads inside the application, I can be very specific. I'm only interested in stopping if this variable is being modified by thread number three and then we can get into conditionals as well stops when the contents of a particular variable is greater than five. This would be another option to your conditional breakpoint, but in this case it uses hardware breakpoints and therefore doesn't have the same overhead that a conditional breakpoint would have if we were using standard software breakpoints. Here is an example of this. I have a breakpoint at main. I'm going to tell it to run and it hits the breakpoint and I then tell it to watch the variable x and continue. Well, when I modify x, it's going to immediately hit the breakpoint and show me the old value of x and the new value of x. So it shows me what it was before it got modified and what it was after it modified, but I'm still setting now at this location where it got modified. I can now do a little bit more debugging here in terms of figuring out, okay, what happened? Is everything okay? Why did it do this? Et cetera, et cetera. We can also debug threads. We can do an info threads. That will show me all the active thread IDs. I can then select a thread by ID and I can specify set a breakpoint in a particular thread. Other threads can reach out and touch that same breakpoint and it won't fire unless it is the thread that I specified the thread ID for. And then if I have a case where I've hit a thread breakpoint, I can then also tell it to show me all of the backtraces for the thread. So when I hit the breakpoint, I can have it set up the possibility of stopping the entire process and all of its sub threads. That's the default, by the way. Or I can specify using this set scheduler locking where I can then limit the execution to a particular thread. So I can either stop when I can stop the entire application when a thread hits the breakpoint. I can stop just the thread that hit the breakpoint. I can do either of those things by messing around with a scheduler locking mechanism. Now, also, we have the ability to generate core dumps. Now core dumps are kind of a lost art, if you will. The idea behind a core dump is it's going to take a memory snapshot of everything, the register sets, the instruction pointer, the stack pointer, everything that was executing at the time of the malfunction. By default, this is turned off inside of Linux. We can use the you limit command to turn them back on. And in you limit, we have to specify the maximum core file size in 512 byte blocks. Why 512 bytes? Well, because that used to be the sector size back in the old days. So this is normally turned off because if you have an application that terminates, it will leave a core dropping, which is a big chunk of memory that's basically been written off to disk. And people will forget about those things. And they have a tendency to collect. So the default case in Linux is to turn off core dumps. We can turn the core dump back on. And then when we encounter an application that terminates, it will then turn around dump the core. I will then be able to use GDB to load the application and the core file back into memory. And it will show me exactly where the application terminated. And I will see the reason that it terminated. And all of the memory and the register sets that were associated with the application at the time of termination will then be visible to me. Now, understand this is postmortem debug. This is not something that you're going to be able to then continue execution from that point. It's going to stop the application. The application is dead. But it does give me the ability to then turn around and look at what was happening inside the application at the time of termination. And this also works with most of the GDB front-ends. So whether it's DDD or Eclipse or Inside or any of those front-ends, it all works with all of them. You can also use it in 2D mode if you happen to be limited in terms of your graphics capability. So all of that just, it works the way it should work. Now, remote debugging with GDB and GDB server. This is another thing that happens. And this is a case where I don't necessarily want to be running the debugger on the target. The reason I don't want to run the debugger on the target is because the debugger is very intrusive to the target. What I want to be able to do is I want to be able to run a particular application under the auspices of the debugger, but not run the debugger itself on the target. That's where the GDB server comes in. Now, in the case of the GDB server, the GDB server is a very small executable. I think it's like four, six, actually maybe six k bytes in size. It's really, really tiny. But what it does is it allows me to attach to an executable piece of code, I can attach to a running piece of code, or I can run a piece of code under the auspices of the GDB server. When I run that code under the auspices of the GDB server, I can then connect to it from the host via, say TCP IP or an RS232 link. It has several different mechanisms for being able to do this. And then I can set and debug the remote target. So as long as I have a network connection somewhere to the remote target, I can actually debug code on the remote target. And here's how something like that would work. I have GDB server. So this is on the target itself, GDB server, the IP address of the host, and a port number on the host. I then have the application that I'm going to be debugging. And in this case, I put it in background mode just simply so I get my console back. You don't really have to put it in background mode if you've got multiple SSH sessions going to the target for something along those lines. Then on the host, I can do target remote and the IP address of the target and the port number that it's sitting on. And these port numbers are just numbers that we made up. 1929, I have a good friend of mine from the old days who always used the date of the stock market crash here in the United States back in 1929 because nobody else uses that particular port number for any of their real applications. So he would always just pick that one. So it kind of hangs over from old times. But then at that point in time, as soon as I do the target remote, I'm connected to the target and I start seeing source code. I can also use the file command. And from the file command, I can then load the executable with all the debugging code in it into the debugger. And then once I've loaded it into the debugger, then I can start stepping, setting my break points and things of that sort. I can attach to a running program. So I'll specify GDB server, the host IP, whatever that is, and a port number, like 2345, just pick one, dash, dash, attach, and the process ID. That will then connect and wait for the target remote command to happen. And then you'll be connected in the debugger. If I wanted to do this from within DDD, then I would do DDD dash debugger, specify the debugger and the application. That then tells the DDD front end that I'm using the arm back end and the application that I want it to load. And then I would issue my target remote command, and I would be connected and debugging the application on the target from the host. So definitely a very handy thing to be able to do. Obviously, now all of that was related to debugging functions precisely. There are a ton of additional things that GDB can do. We just don't have enough time to go through all of them. So we're going to switch gears a little bit here, and we're going to take a look at some of the live tracing capabilities that are also useful for doing debugging. Now, functions like Strace and Ltrace, there are a lot of tools that are available to us, and Strace and Ltrace are ones that I use frequently because they give me the ability to take a look at all the system calls or all the library calls that are coming out of a particular application. There are no special compilation flags. You do not have to compile the code with the dash G option to use Strace and Ltrace. So what Strace does is it hooks into every system call. So let's assume that I'm running an application that's doing something, and when I click on this button, the application dies. But it doesn't give me any message, doesn't give me a core dump, doesn't give me any details, nothing that I can debug against. It was executing something when I pushed the button, but I don't know what. So what I can do is I can run Strace and attach to the application just before I push the button on the UI. And as soon as I push the button, I will then see all the interaction that application has with the system calls. Now, I can also put a timestamp on this if I want to have kind of more information about timing. And we see the example Strace output here. This is every function call that's being executed and its return status. So this, of course, very handy for being able to take a look at, I sent this function, what did I get back from it? These are all calling into the system call. Now, you can also use Strace to do things like, where am I spending all of my time? Here we see the dash C option, which counts the number of invocations. And we see that in this particular application, most of our time is being spent in the right call. So if my application is slow for whatever reason, then I know that this is probably the culprit that I need to look at. And I need to be able to figure out why it's so slow. We see the number of microseconds per call and the total number of calls. Now, obviously in this case, most of the time is consumed in the right call. But there's also a fair amount of time being, you know, we're seeing the select call being called frequently. And it's probably the next highest contributor to this overall execution time. Now, Ltrace does exactly what Strace does, except it does it for libraries. And in the case of Ltrace, we can Ltrace specific libraries. So let's assume that I have a third party library that was provided to me and I don't have the source code to it. But I want to look at the interaction between my application and that third party library. Well, I can hook an Ltrace up to it and watch every single function call going between my application and the library. The output looks just like Strace, except that now it's at the library level rather than the system level. I can limit the libraries that I'm interested in looking at. And I can also limit the functions out of a particular library. So maybe I'm only interested in the update me function in library XYZ. I can then specify an Ltrace only show me accesses to update me function in library XYZ. So I don't have to see everything. I just see a few things as I go along and I can limit what that output looks like. Now, PROC file system is a representation of the kernel's information. It's basically a dump of the system control block, the struct task struct that's associated with every application. And each process ID has its own unique directory information associated with it. So for instance, if I jump out here, go back to my command line and hopefully it will update for all of you. So let's do a clear here. sudo-i and then we'll cd into the PROC file system. Every one of those numbers that you see is corresponding to a PROC file system entry. So if I did a psax pipe to grep bash, and that would show me all the bash applications that are sitting out there. Let's take a look at PROC 22149. So 22149. This then tells me all of the information that the kernel is keeping about this particular process ID. So for instance, I can take a look at the memory map. This shows me the executable. That's the text segment. There's the read-only data segment. There's the read-write data segment. I have heap memory here, so I know exactly where that heap memory is being stored. I have locations associated with libraries. And again, we have the executable and read-only data, read-write data, etc. We also, this p here indicates that this is private versus an s, which would be shared. So if I have shared memory segments, then I would see them show up as an s. These are all private. I don't think I have any up. There's an s right there. So there's a shared one. And that's with a modules cache file of some sort. But we see the stack. We see the virtual system calls. So all that information, this is the virtual address that it shows up at. And this is the offset into this particular piece of code as to where this actually got mapped. So that's a terribly useful thing. I can also take a look at the status. And this shows me things like it is currently sleeping. There's its task ID. Here are the number of file descriptors that it has, the owner who is running this particular application. This is the useful stuff down here. How much data is in the virtual environment, the virtual memory segment? How much is stack? How much is executable? How much are libraries? And how many page table entries? How much is being taken up in page table entries? So we can also see signals, things of that sort. Again, all very useful things when I'm trying to figure out exactly what's going on inside of an application. One other thing that I will show you is, if I go back out here to, whoops, I want to be in slash proc, I can look at interrupts. And this shows me all of the interrupts. Let me just do this all. Let me shrink it down just a little bit here. It shows me all the interrupts across all the processor cores. I have a hexachore on this platform. It shows me how they're routed. It shows me what kind of interrupt they are and what they're attached to. So this is very useful. I have fast EOIs. These are level triggered interrupts. These are edge triggered interrupts. These are message signaled interrupts. So these are PCI Express interrupts. So I can see exactly how my interrupts are happening and which processors are being used most often for servicing a particular interrupt. Let's see here. So let's go ahead and close that one and see what we can do about wrapping this particular discussion up. We have Valgrind, and it's Valgrind, not Valgrind, because it is the gate to Valhalla. There's Helgrind, Valgrind. There's a whole bunch of these little tools. We have Cashgrind, which basically looks at locality of reference inside of caches. I have heat profilers, and Helgrind deals with POSIX threads. So if there are race conditions inside of threads, Helgrind helps me find those race conditions. Valgrind is a complete of software. It is relatively complex, but it does work on ARM and it does work on x86, as well as PowerPC. So if you're trying to get into real fine grain debugging to find out exactly what's going on inside of the application processor and how it's interacting with your code, Valgrind is the key to being able to do some things like that. We have GPROF, which is a profiler. In the case of GPROF, we just simply have to specify the dash PG flag in order to turn on the profiler. And then we will run GPROF and the executable file. It will then produce an output that we can then take a look at in some detail. It'll tell us little bits and pieces. Let me show you here an example. So here we're looking at code coverage, which is also out of GCOV, the new coverage tool. So GCOV and GPROF, they're kind of old tools, but they're still very effective. Things like this, for instance, if I'm trying to get 100% test coverage, I now know that this particular line 13 is never being executed. It's never executed at all. So that says that my test case isn't exercising 100% of the code. And when you're doing safety critical types of applications, you want to be able to handle all 100% of the code. We can also take a look at branches, which branches have been taken, how often they've been taken, which ones were never executed, and how often those things were executed. So definitely a very useful tool in being able to figure out how all of that works. We have the Linux Trace Toolkit, Next Generation. This has been around for a while, but it's still a very useful tool because it gives us the ability to instrument the kernel and see the interaction between your host and the target and what's happening inside of the target is it executes a piece of code. So it's useful for debugging not only user space code, but also kernel space code. Now, LTT and G has been around, as I say, for a long time. It does have a user interface to it. Here's an example of that user interface running inside of an Eclipse plugin. So I can then see individual applications and see what their execution time was. I can then zoom in. Lots of very cool things for LTT and G. O Profile is kind of long in tooth these days. It is a statistical profiler. What it does is it allows me to sample the execution of the code and find out where it was running most of the time. So again, this is useful kind of like GPROF and the S Trace-C option, where I'm trying to figure out exactly how much time is being spent in what functions. Now, F Trace, this is Stephen Rosdett. He and a group of other folks were responsible for creating F Trace. It's a function tracer inside of the kernel. It was actually added way back in 2008. And it was one of the weird cases where the documentation was accepted into the kernel before the code was actually accepted into the kernel. But with F Trace, we have the ability to set up static trace points for event tracing. And we also have dynamic kernel function tracing. So we can look at the interaction between user space and kernel space or between two individual elements inside of kernel space. Very much like O Profile, it has to be enabled in the kernel. So you have to turn on F Trace in the kernel before you can actually use F Trace. F Trace has an output, a GUI front-end now called kernel shark, which looks very similar to what we were just looking at with LTTNG. It does give us the ability to kind of jump into a piece of code. And I can zoom in. So you see these little vertical bars here, these are kind of like a oscilloscope sampling areas. So I can then say, well, I want to look at just this piece of the code and it will zoom in on you. And then you can continue to zoom in and then you can zoom back out again to see exactly which pieces of code you're interested in. So you see the execution here across all the CPU cores. And you also see the individual events that were happening inside of this. And this is a nice little GUI. It is available from Stephen's website. It's also available in most of the major distributions like Ubuntu, for instance, has this available. You can just download it and install it using your app yet. Now, the problem, as I mentioned here, isn't that Linux doesn't have enough tools. It's that Linux has too many tools. Debugging in user space at least is mostly done with GDB or it's equivalent LLDB if you're using the Clang compiler. Gprof and Gcov, Strace and Ltrace, they're really tools to provide you additional insight as to what's going on. Then tools like Ftrace and LTT and G give you extremely detailed interactions between the kernel space and your user applications or between kernel space entities for kind of a system-wide view. So there are a lot of execution information that's available to us. One of the questions that came up here, let's see here. Can we see the execution time using GDB? I don't remember anything in GDB that allows you to do that. I would then probably pop out to Strace. The reason I wouldn't do timing inside of GDB is because GDB, because it uses the Ptrace functions, it's going to affect the execution time of code. So this is not something that you can, if you're running something within GDB, you can guarantee that the execution time is going to be perturbed because GDB is sitting in there at all. So I don't think it would actually have a whole lot of meaning to you if you were to be able to see it inside of GDB. Let's see here. Oh, one other thing. I'm just about out of time here. So let me see if I can actually make this work. We've been talking a little bit about Ftrace functions. So let me cd here to, I always go a little bit long. Sorry about that. So let's cd to the tracing functions. So I've got Ftrace turned on. We see all the different options here inside of Ftrace. Let's take a look at the available tracers. So I have block IO tracers. I have wake up, real-time wake-ups. I have IRQ off time. I have function calls so I can get into detailed function trees. Let's just do something real quick. I'll turn on the IRQ off tracing. I will take a look here at some do a couple of things just to generate some interrupts. And then, okay, so let's take a look at the current tracer first. So it's IRQ is off is what's turned on. So let's go ahead and turn the tracing on. And then I will do a couple of ls's. And then I will turn the tracer off. And then let's take a look at the trace. And this shows me my interrupt off time and how long interrupts have been disabled. So there we go. That kind of gives you some idea. I wish I had a little more time. I'd show you Ftrace with kernel shark running. It's kind of cool. But unfortunately, I've run over my time again. So there we have it. We will open it up for Q&A. If there's any additional questions here, let me just make sure I've answered most of the questions. How does line editing work in TUI mode? Yes, so if you've selected the command line, then you can up arrow and it will allow you to scroll through the history. So you can then edit the histories there. You can just type over them again. Let me see here. Anything else? Okay, so we talked about execution time in GDB. Any other questions? I know that it was kind of fast here. Hopefully, everybody has been able to find the... I see I'm still doing screen sharing here. Everybody has been able to find the charts. I have here an application called Migrate. Let's go ahead and do a make on it. I'll just show you real quick what happens when we run kernel shark. This is going to run the application and it's going to keep it limited to just four different CPU cores. And then I'll run kernel shark. And it helps me if I can actually make toast plus and now go back and run kernel shark. And there is the application as it ran. This is actually one of Stephen's test cases where it actually deliberately kind of mashes the CPU to see how quickly things can migrate across. So it kind of gives you an idea of what it would look like if you were then trying to drill down into the actual execution of something using something like kernel shark. And you can get really, really detailed values here. I mean, you can tell exactly how much time, what time things took to execute, how often they did context switches, things of that sort. Very useful tool and comparable to what you see with LTT and G. But this one is a little bit easier to set up. LTT and G requires some additional drivers be loaded to the kernel. So it's fairly automatic these days. But F trace is a much faster and easier to work with. And that's my personal opinion. And it's probably worth what you paid for it. So all of 50 bucks, right? All right. So I think there's no more questions. And I don't want to run into anybody else's time. So if you have any questions, let me just go ahead and stop the screen share. And let's go back just to my information in case you want to try and get ahold of me. I'll get there. There we go. So there's my mail, mikeandersonptr at gmail.com. Or I'm on Twitter at at Hungar. Don't pay as much attention to Twitter as I should. But you can definitely reach me there. Or you can reach me over at LinkedIn. I'm on LinkedIn as well. If there are no more questions, then I will turn it back over to the powers that be. Thank you very much for your attention. I hope you had a good time here and got something out of it. See you later.