 Hello, I'm Chris Simmons and I'm here to talk to you about debugging embedded devices running GDB This page is just showing you the licensing for this slides. I'll leave you to read that at your own leisure This is a little bit by myself I have been using Linux as an embedded operating system for over 20 years now I've written a book on the subject And you can follow me on Twitter at to net software and you can find me on LinkedIn And also you can see my blog looking after the inner penguin at to net to co co UK So this is all about debugging so bugs happen we need to know how to find them and In this case, we're going to be looking at interactive debug debugging using GDB There is a lot more stuff about this in my book go read chapter 14 And before move on I want to just highlight the the quote at the bottom from Brian Kernigan Who says debugging is twice as hard as writing the code in the first place Therefore if you write the code as cleverly as possible, you are by definition not smart enough to debug it Yep, I've often felt like that Not that my code is in is in any way clever. So make your codes straightforward Don't obfuscate Do things in the best way possible so this Tutorial is going to be some slides me talking, but also some demos Yep, I know demos. We'll see what happens So for the live demos, I shall be using my laptop as the as the development machine So this is running version of Ubuntu 1804 in fact and then for the target I'm going to be using a good old Raspberry Pi and I have prepared a Yachto project build for it Because that is makes it look more like a typical embedded system. I have prepared a set of instructions Which you can download download from that link. I Encourage you to do so and to follow along with the video So this is a Raspberry Pi. It is an ARM based device Actually as a quad core ARM v8 architecture processor Which is a 64-bit processor, but we're actually only using it here in 32-bit mode It has a micro SD card slot, which is what we're booting it from and It has an Ethernet port which we'll be using for networking Everything's going to be command line. I'm afraid so we will not for example using be using any HDMI videos So the operating system is built using Yachto project go follow that link there and This is the common. This is one of the most common ways of building embedded Linux systems and The their instructions on how I set this all up in the workbook. So let's talk a little bit about tool chains First of all before we get on to the debugging But before we get on to the real debugging Let me define what a tool chain is So a tool chain is a C compiler specifically GCC the GNU C compiler collection Plus a package called bin utils which we need for the assembler and the linker and some other useful tools as well plus a C library the C library allows us to Access the operating system underneath which is going to be Linux and Of course GDB which is what we're talking about here in the first place Now there are two different ways of developing code for a system. You can do it native or cross With native development then you develop on the target you run on the target and this is what you do on a PC for example And also on say a Raspberry Pi running Raspbian But it's not the most common way of doing this for embedded devices The reason being that most embedded devices are not really powerful enough or not configured in the right way to do compilation natively It requires fast processors equal requires lots of memory. It requires lots of storage So it's much more common to do cross development. So with cross development. We develop on a on a host system Typically running Linux and we run on a target and That's what we're going to be doing here in this tutorial and I've used Yachto to create the cross development environment for us so we need a tool chain of course and We're going to be using the fourth bullet point on that slide. We're going to use the build system Yachto project to generate the tool chain for us now looking at cross tool chains the first thing you may notice is that they have Peculiar the tools have peculiar names So they have a prefix of the form arch vendor kernel operating system for example here we have MIPS EL Minus unknown minus Linux minus canoe. Oh And the prefix always ends in a final minus So in this case the architecture is MIPS EL which is little Indian MIPS processors not very much used these days But there they are a thing The vendor is set to unknown which means that it's either unknown or more likely. It's just not important in this case The kernel of course is Linux because we're always talking about Linux in this context and The operating system is normally set to GNU. So you take the prefix such as that and then for each tool you want to run you just append it to the prefix So to run the cross compiler you type MIPS EL minus unknown minus Linux minus canoe minus GCC That is your cross compiler I want to take a little bit of a detour and just mention a few things about arm cross compilers Since it's quite likely that you'll be in an arm be using an arm processor So on tool chains have gone through a couple of generations The first generation Followed the format I've just described so you would find typically something like arm dash unknown dash Linux dash canoe dash Then there was the second generation of Interfaces of tool chains using what's called the extended a bi and a bi is the application binary interface Which is a set of rules for making function cores and such So with the new incompatible a bi the tool chains acquired The extra string e a bi so now we have arm unknown Linux GNU e a bi Then later generations of arm processors acquired floating point hardware So HF was added to the operating system part And so on there are even more complicated Combinations if you look carefully so arm tool chains extend the concept of the Toolchain prefix and they add in to some hardware components as well as part of the operating system Next I want to mention the CIS route So the CIS route of a tool chain is the place where the header files and Library files are stored plus some other configuration information You can find the CIS route From a tool chain by asking the tool chain itself So you can use the GCC command and minus print CIS route and it'll tell you where it is If you do this on your native tool chain on your PC if you just type GCC Minus print minus CIS route it will say forward slash So the CIS route is the top level directory For embedded tool chains, they are going to be something different So in my example here, I have Another example of a tool chain. This actually is a 64-bit arm tool chain AR-64 from build route Then it's going to GCC and it says its CIS route is in that directory So this is the directory essentially where I configured the CIS route in the first place So the CIS route is always for for cross tool chains is always someone else not in the top level directory And we need to know this stuff when we're cross-compiling, but also when we're debugging as we'll come to So within this route you will expect to find The libraries so they'll be in lib and user lib and If this is a 64-bit architecture, you may well find that lib64 is there usually as a symbolic link back to the lib directory And the other important thing here is the include directory So from the CIS route because CIS route slash user slash include that's where all the header files should be From the debugging point of view the header files don't really matter, but we really need to know where these libraries are So they'll be in the CIS route slash lib and in the CIS route slash user slash lib So you can get information about your tool chain from the tool chain Prince this route is one thing You can ask it the version with a minus minus version and you can get the configuration information with a minus V So I have created a tool chain using Yachter project I typed bit vague minus C populate underscore SDK and then the image I was creating which actually was called core image base and That created for me a tool chain with this name Well, actually it generated for me a shell script a self-insulting shell script which would install the tool chain So I run that shell script notice it ends in sh and It extracts itself and puts it into the default install path Which for this is going to be opt Slash pocky slash three dot one dot three that is because this is Yachter project three dot one dot three They're done fell release and then to actually make use of the tool chain. I must source a shell script so I type source opt pocky three dot one dot three and then this the Setup script is called environment minus setup minus and then the the tool chain prefix Well an extended form of the tool chain prefix really There is one little oddity with the way that you use the Yachter project tool chain That is that if you ask this tool chain to print its this route it prints out slash not slash exist Which doesn't exist So this is telling you that this is Not the way to run the tool chain. You don't actually you're not meant rather to run the tool chain like this You are instead used meant to use a variable called cc which has been set up by the shell Environment setup from the previous slide So if we look at dollar cc, we see These are the parameters we should use if we want to compile and one of the parameters is this this route So to compile a little program. I don't type arm dash pocky dash linux dash guinot eapi dash gcc That will not actually work. I type dollar cc program on a compile minus o the executable Incidentally the this looks like odd on the slide I use a dollar sign at the beginning of a line to indicate that you're at a shell prompt and Then space and then the thing you actually type So in this case you type dollar cc Not dollar space dollar cc. I hope that's obvious The tool chain is not just gcc. There's a whole bunch of other things Amongst them is AS This is the GNU assembler so that can take assembler code and produce machine code We have the linker that can link objects and libraries together We have GDP which is a thing we're going to be looking at and a whole bunch of other things as well Okay, so demo time. Let's have a look at the tool chain for the Raspberry Pi and it's also boot at the Raspberry Pi Of course and see how it works So here I have Two window two terminals rather the right hand one I'm going to be using for the target for the Raspberry Pi the left hand one is going to be my development machine so I'm going to begin by booting up the Raspberry Pi and I have a serial terminal attached to it So that as it boots up, we will see the kernel messages Scrolling onto the screen. So this actually is a Raspberry Pi Be a 3b we're booting from the SD card and it's up there's the DHCP configuration come in and it says login so I'll log in as root No password So this is a fairly small Yocto project built If I run the ps command, for example, there isn't very much going on here The most important thing from our point of view is going to be There is a an SSH demon running called drop bear and we're going to be using that to talk to the Raspberry Pi To get a shell and also to copy files to and fro Okay, so I'm actually going to go write one thing I need to do if I'm going to connect to it via Via SSH or via the network. I need to know what the address is so I'm going to type I have config and I noticed that ether zero is that address So now I can quit from mini-com. I shan't use this again and I can instead use SSH Want to log on as root and And give the IP address, okay, and then we're back where we started Okay, so over here I Have a copy of the well-known hello world program and I want to compile that and run it on the Raspberry Pi So I begin by sourcing that setup script In art and Something or other So now I can access the tool chain. I can I can add something arm hockey Do something GCC I didn't put a T in there that was better and We have GCC version 9.3 Just to prove the thing about the sys route. I think it's not so good today Then it says not exist, but if I do the echo dollar CC thing Then it gives me the actual sys route. It's going to use okay, so Here's my hello world Program. It's a slight variant of the hello world program. It actually has a little loop and it says hello world four times and I've got a little make file Just to make it easier to compile So I'm just going to type make that is that's cross-compiled everything very very quickly nicely and I now have a program called hello world Okay, then it's been cross-compiled if I try and run it it won't run and If I type file hello world It confirms to me that this is an arm executable It has been compiled with debug info which has not been stripped Excellent, so to try this out. I'm going to copy this now to the Raspberry Pi So I can do scp Hello world We'll copy it as root and I've actually set up In my host file because I know that my Raspberry Pi will always come up with the address 192.168 42.2 So I've set a name in my host file called our pie 3 So that should work. I know it wouldn't I need to say where to put it. So we're going to put it into user Been Okay, so that is copied it to the Raspberry Pi over here in Raspberry Pi land. I can type well if I look in user World There's the same program and I can run it and it says hello world zero two three So fantastic everything set up we can compile things we can run the results Now we can talk about debugging mmm So when we're going to debug things we naturally need to build debug information in And you control this with a minus G option to GCC So if you're not going to be debugging anything you set it to zero and Then you can set it to increasing numbers up to three for more and more debug information and As well as a minus G. You can also use minus G GDB and then the numbers zero one two three Using minus G GDB means that it creates the debug information in a format specific to GDB Otherwise it uses a generic format called dwarf Um in my experience it doesn't make a huge amount of difference certainly for the things we're doing in this tutorial It doesn't matter at all. So all the things that we're going to be you know them all this examples I'm going to be running I should say I'm going to be compiling with minus G to Which actually is the default source level debugging that you would expect Next issue next issue. Pardon me is going to be code optimization If you've ever tried single stepping through optimized code, you will know it's not easy Once you start optimizing things once the compiler starts optimizing things It starts doing all kinds of crazy stuff and the code that actually gets compiled is not exactly the same as the code you wrote So it is usually a good idea Especially if you're going to do single stepping through code to turn off or minimize the amount of optimizations that you need Now you can do this explicitly by setting Optimization flag minus capital O zero Sometimes that will cause problems. So there is another common option minus o g So these are gdb compatible optimizations. In other words optimizations that won't cause gdb too much confusion The examples I'm running I've gone for the first option and I'm actually compiling without any optimization at all So if we are doing cross compilation, we also need to do cross debugging In other words, we want to run the debugger on the target, but we want to control it from the host So we can do that using a program called gdb server Gdb server is actually part of the gdb package So we run gdb server on the target and we have it load The program we're going to debug Gdb server doesn't care at all whether there are debug symbols there or not so usually the The binders we put on the target are stripped so they don't have symbols It means they take up less space in the storage and then we're going to talk to gdb over a network connection and We're going to control it from gdb from our tool chain and That copy of gdb needs to load the same program, but it's going to read the symbol tables from that program So that's the basic setup. Let's see. Oh Not quite Yes, I've said that stuff. Oh, yeah, so before we get to actually doing the debugging we need to think a little bit about this roots so when The gdb that's running on your host when it comes across a library reference It needs to know where to find the library in order to find the symbol table for that library in other words, it needs to know where your sys root is and Generally speaking it doesn't have any way of knowing this so you have to tell it So this is why we need to know the sys root for our tool chain We type sense this route And then something like opt pocky 3.1.3 sys roots and then the architecture which is Something we'll come to in a moment. So here's a sample of a simple debug session I'm going to be running gdb server on the target on the on the Raspberry Pi I need to tell it which port number so we're going to be using TCP TCP network for the debugging I needed to give it a TCP port number to listen on I Tend to use 2001 and then the program want to debug is going to be hello world So when we do that Gdb server will load hello world into memory and then set a breakpoint at the first instruction and then sit there waiting for a connection Okay, so hello world is not running at this point. It's actually waiting so then on the host computer I Can run My copy of gdb Gdb loads hello world and Then that will give me a gdb prompt Now I need to set the sys root as in this particular case. It doesn't really matter But if I wanted to step into some of the libraries, I would need to set the sys root Then I need to make the connection between gdb and gdb server So I type the command target remote The target address 192.168.42 2 and Then the port number 2001 and when you do that these two Programs will start talking to each other and you should see this message on the Raspberry Pi Remote debugging from host and that's the IP address of my laptop Now I'm in charge over here So I can now do a regular debug session so I can set breakpoints I can continue execution And so on and so on. Okay. I said all that stuff. So what else can we do? So we can set breakpoints To set a breakpoint you type break and Then you can type a line number or you can type a function name You can get a list of breakpoints you have currently by typing info break and You can delete a breakpoint with delete break we can control execution So in the example a couple slides back I type continue which I can abbreviate to see That means continue execution from a breakpoint. It means continue continue running or For a breakpoint. I may decide to step So if I type step which I can abbreviate to s that will step one line of code and If it is on a function it will step into the function If I want to step over functions, then I use next which I can abbreviate to n and If I'm in a function and I just want to continue until the function ends until it returns I can type finish Which I can abbreviate to thin There is also another command called run which you may be familiar with from other tutorials Run does not work with When you're doing a remote debugging like this The run command essentially means load a fresh copy of the program into memory and then start it running from the beginning That's not what we're doing the with remote debugging We load the program using GDP server on a different computer and then we make the link with target remote So on remote debugging you want to continue not run Next then let's have a quick look at variables so We can print variables with print and the variable name We can change the variable to a different value. You can type set and then the variable name Sorry set and then var and then the variable name and the value you want it wanted to have Okay, so let's set up a simple debug session with hello world So over here on my Raspberry Pi I'm going to use GDP server Oops, and I'm going to use port 2001 and then program on to load is going to be user Okay, so that is now loaded into memory. It's waiting for a connection over here I'm going to load the GDP from the tool chain. I'm going to load the hello world program and next I'm going to set a break point Oh, no, not Next I'm going to link to the target. So I type target remote and Then the target is going to be at 192.168.42 To go on 2001 So now it says remote debugging from host and it knows who I am and Then over here. Ah Yes, so here. I forgot of course to set the sys root I'm just going to carry on at this moment and I'll come back and redo this with the sys root so you can see the difference But for a moment, I just want to set a break point in Main That should work I continue And we now at the main function There is hello world and I'm just going to single step through it with the end command next So it's going to print out hello world And the value of I currently is zero. So it should print zero So zero hello world and so on and so on Until we finish the program and we're done Okay, so let's quit from that and do it Slightly better. So once again, I set that up to listen I need to know what sys root is don't I so if I do Echo Here's miss this route. I really just want this bit now I do Let's redo the command So now I can set this this route our time set put in that stuff Now I can do the target remote Continue We're basically the same point to the word before The main difference I guess is that we don't get the rude messages about missing sys roots And you can see that it has red symbols from the linker And actually only from the linker But that's good. So I'm I'm just gonna set this running. Oh, no, there's a thing I could do Let's demonstrate changing variables. So let's just step through this a couple of lines So it's printed out zero It's about to print out again. So I currently is one I could be mean I is an integer. I think what happens if I set it to a negative number? So set that I equals minus three Minus three hello world and then it counts up to three. Yeah Okay, so that's all well and good so far Let's move on to the next thing So there's a lot of typing going on there to set the sys root and to do the target remote and so on and Even for me, that's a bit too much typing. So let's look now at GDP command files This is a handy-meat thing. It allows us to put a bunch of GDP commands into a file and then either have them executed at when GDP starts up or We can also on the next slide actually do do this By adding new commands to GDP So these command files we can create a file called dot and GDP in it in the current directory And then when GDP starts up it will read that file and run those commands And we can also create a dot GDP in it in our home directory and put a bunch of commands in there as well Except that it doesn't actually work all that well Recent versions for security reasons actually have disabled this functionality Unless you put into Dot GDP in it a line like this Water load safe path safe path and then the path to the jet dot GDP in it file This is becoming too much like hard work So there is the third option we can create a GDP command file and then load it explicitly With the minus x option. So that's what we will be doing So you can put into a command file any sequence of GDP commands anything you can type at the command prompt in GDP you can put into a command file and You can also do other things So for example, I can create a new command using define I use define The command name then a sequence of GDP commands and then I terminate it with the end keyword So I have set a Simple little thing here, which sets a breakpoint on main and then get prints out my breakpoints And then I can run it at runtime. I type be main and it sets a breakpoint And then it prints out all the breakpoints So this is really handy for stuff that you do over and over Okay, so demo time again. Let's try some of these things out So first of all over here on the rescue pie Let's set that up to be waiting So over here, here's my hello world and Yeah, what are things I have to keep on typing So first of all, I really get bored with typing this sets this root thing. There's really quite a lot of that So I'm going to create a file. I'm going to call it gdb dot in it And I just need to copy The thing that I typed in a little while back because it's a big long string that one so I can just put that in here and Whilst we're at it. So the other thing I have to keep on typing is the target remote thing I may have to do that Multiple times and sometimes I don't want to do it. So I'm actually going to set this up as a command So I type define I'm just going to call it our pie three because that connects to my Raspberry Pi and Then I want to do target I Want to 168 2 colon 2001 and Okay Save that so now I'm going to debug pretty much as before so arm Pocky Maybe I GDP Minus x so minus x is going to point to the dot gdb in it So the gdb init file and then program is going to be hello world Okay, I think that's everything so that looks good and If I type our pie three that should make the connection to the Raspberry Pi Which it does break point of main Continue Step okay good. So that's going to save me some time for the forthcoming Demos Good from that. We're done Next thing So command line gdb as we've been using it is all very well But maybe we can improve on that so I want to look at a number of front ends to gdb and I'm going to start off with one called the terminal user interface or just plain 2e This is a part of the gdb package Assuming it is enabled in your gdb cross compiler you can just add minus 2e to the command string So our popular Linux going to be I minus 2e hello world that will give us The 2e interface and it looks a bit like well it looks exactly like this So It's an improvement so the main thing is that we now have this window here Which shows us the code and as we set break points and steps through the code the cursor advances We can also see visually the break points this B plus indicates of the break point on this line as it happens the gdb that we've built using Yachter project doesn't have 2e enabled but No worries because there is another thing which I actually prefer to 2e in any case There's a thing called see gdb Which is a curses Front end to gdb It is quite like 2e but also better On Ubuntu it isn't available as a package or at least it isn't in the standard lists so I had to actually download a copy of cgdb and Compile it and install it and then having done that I can just use cgdb minus d and Then the same string as previously and it looks like this Well, it's a bit nicer than 2e to be honest. It's certainly more colorful So we have syntax highlighting highlighting which which is quite nice and Although it isn't particularly obvious this the 7 in red here indicates that there is a break point on line 7 and The 8 in green indicates that that is the current place where we're at In stepping through the code and then the third one I want to look at is ddd The good old data display debugger This has been around for ages It keeps on working. It's still the best standalone graphical front end to gdb So you install the package the package ddd And then you can launch it with a particular debugger with the minus minus debugger option and then point it to your tool chain gdb and It looks like this So it says building on what you saw with 2e and cgdb But now we have a a third window the one at the top, which is the data display window So we can put things here for example if I double-click Well, okay, when you run When you run into a function actually in this particular case Then it pops into here. I think called args Which are the arguments which are p and w in this case And if the argument is a pointer, I can double-click on that and it expands it And if that contains pointers, I can double-click on those and they expand and so on So it is really really nice for looking at structures Unfortunately, the the toolkit it uses is really old ancient. So it does look a bit dated But it works The other thing I should mention before I move on is this floating window here This gives us the the tool the the the clicks we can do use To for example a single step. So we have a step and we have next We have finish and a few others I haven't mentioned And it also has the run command here don't ever press the run button if you're doing cross Debugging that will mess things up. There are also front ends for gdb in a Fairly wide range of ide's including eclipse Visual code k develop and a bunch of others too Okay, so back to my demo So this time I want to debug hello world, but using cgdb So that out of the way That gdb server running So I'm just going to do the same thing as before, but I'm going to add c gdb minus d And that's it okay So now it has immediately displayed the the first bit of the program in the window here I'm going to make the connection to the Raspberry Pi I do with our pie 3 Now it changes because now it has made a connection with Gdb server it knows the address that it's at and it now knows that actually we're in the startup code for the C function So we see it set a bunch of assembler I don't particularly want a single step through this So I oh that's interesting So I just typed in and it was smart enough actually to say you probably don't want to do this so It actually has put me into The start of the program Oh, no, it hasn't it's actually I'm terminated. I wish I hadn't said that now. Let's do that again That's not what I meant to do Okay, there is hello world his Okay So now I want to type R5 3 to make the connection Good now I want to set the breakpoint on main and then I can continue and Now we're in In the C code that looks better so now I'm just going to step through this and You can see that as we step it advances the the cursor and it shows you the next thing it's going to do So that looks a bit a bit nicer to me and eventually we step we've Exhausted the count and we drop out of the function and we're done Okay, good so far. We're doing quite well Okay, next topic I would have a quick look at another way of looking at variables and These are watch points So a watch point is a kind of breakpoint that depends on a data value For example, I can type watch eye and that will then a break whenever the value I is written to Which is handy Notice, this is a hardware watch point In other words it this is setting up some hardware registers on the chip So that we don't have to single step through everything check in whether eye has changed on each Breakpoint or anything it does this at full speed until we hit the hardware watch point and then it stops now not all hardware can support hardware breakpoints, but Hardware watch points, I mean But actually most of them can That's a little bit crude. Maybe we want to be a bit more precise So we can set a condition We can say watch eye if I equals three or indeed any other condition that can be any variable that doesn't have to be I So in this particular case That means that will break when I equals three So if it changes to be zero one two, etc. It won't when it hits three will break Okay, so let's do that for For real same thing as before. Let's restart that And let's restart that by three Continue So now I'm going to set a watch point on I Okay, and then I'm just going to hit continue and it should break Whenever I is written to which it does of course on every loop Now when we hit the watch point, it says the old value zero new value is one and it gives me the line of code which made that Continue again now it is equal to two and three and four And we're done. So let me just do that again But this time I will set a conditional watch point So I'll try three And continue Now I want to type watch If Okay, now I type C So now it's gone around the loop three times and it's just going around for the last time and We have the break on I equals three and we're in the place that we'd expect us to be So, yeah, this is actually quite powerful quite simple. We can set breaks on variables to break when when they become some interesting value or just when they change at all Okay Right next topic Okay stack frames so Each function has its own stack frame the stack frame contains the local variables and also return addresses and so maybe some other things too It's useful to be able to look at the stack frames To see which functions have called which functions so you can see the call tree from where you are currently in the code to all the ancestor functions that called Ultimately to this function So the back trace or BT command does exactly that So here I'm in a function called at tree and I type BT So this then shows me that there are two ancestors for this Ad tree was called by itself because it's a recursive function So stack frame one the ancestor is at tree and the ancestor to the ancestor stack frame two is main So main called at tree at tree called itself We can use the info local command to see the local variables in each stack frame and We can switch from one fact stack frame to another Using the frame and the frame number So if I type if I'm in at tree if I type frame to I then see The variables that were in the main function at the point at which it called me so very handy stuff So let's get let's have a look at an example of this and see how it works So this I'm going to use a program that is not hello world because I need a program that has Some functions in so I'll use the word counting program that was in the slide a couple of moments ago So take a copy of that It's from the samples directory. It's called word count So there's a make file and a little test File with some with some words to be sorted. So let's build that And then I'm going to copy that to the Raspberry Pi I Three And I'm going to put it into let's say root root at by three and I put it into user bin I Also, we'll need the sample data to test it with So I do an SCP of test I Three and I'm just going to put that in the home directory of the root user Okay, then over here on the Raspberry Pi There's my little test program and it just contains a small number of words and So my word count program is going to sort them and also count them So I do word Count Okay, so now we have the same words, but in the right order in alphabetical order And we see there's one of each of them except the there are two occurrences one there and one there Whoops. So this seems to work quite nicely So I'm going to use the debugger now just to show you a little bit how it works and also to demonstrate the stack frame feature So GDB server GDB server Port 2001 User then Then over here, I'm going to debug it in the same ways we did previously So I'm going to take a copy of the GDB init file that we had with hello world So we're going to copy Hello world in it to here and Then I'm going to run this with C GDB as before Minus X GB in it and then word Okay, that should be everything launch that Make the connection to the Raspberry Pi break on main continue lovely So I'll just step through it a one time Just to see what it does and so it actually defaults to using test.txt Opens the file files there it gets a word What is the word we can print it? It is there. Okay, that makes sense because that's the first word from the file Make sure that it is a sensible word and then it adds it to the tree So this is it a tree sorting algorithm And then it gets another word Okay, so let's step into the add tree function now so I type s to step in So now I'm in a new stack frame if I type back trace So I'm now in a tree and it's got as far as the word brown and And The main function is still there I can actually step back to that frame one in this case So this is the place where we left the main function just as we're about to call add tree And if I type info local for example that will dump all the local variables in the main Function so it gives me a file descriptor it gives me a pointer to something and the word brown, which is the one we're currently processing Okay Switch back to the current frame. Let's step on a little bit further. So it It is now iterating through the The tree trying to find the place at which to add brown and it keeps on going left because we haven't found it yet So it's stepping in another one and now That trace we're now three frames deep So we have pretty much the example shown on the slide in fact. So main has called at tree And Then at tree has called that tree. Yeah Okay so I'm just gonna let that run up just inside to see if we continue and That will carry on and print out the messages so the main thing I want to show is just that you You can build up the the stack frames and crucially use you can Go back to other stack frames using the frame command We'll maybe come back again later on when we look at core dumps core dumps are especially a good place when stack frames are good okay so I'm quit from that and Let's go on to the next topic Okay, so to continue the story the next topic. I want to look at is debugging libraries So there are two things to consider here. The first is that GDP needs to find the shared objects For the libraries you'll do that your program links with and it does that through this this route that we set earlier But in addition to that if you actually want to single step through this stuff or set breakpoints within it You really want the source code for those libraries So GDP is going to look by default into two directories Set by the variables dollar CDR and dollar CWD So dollar CDR is the compile directory In other words is the directory in which the code was compiled and this actually is put into the elf header So long as you haven't moved that code to a different directory that will that will work and Then CWD is the current working directory. So it'll go look there as well We can find out which directors are being used by typing show DIR and We can add a directory of our own Devising by adding DIR and then a path and you see that when you do that it puts that at the front So it'll look in this case in my for my lib and then it will look in CDR CWD So most of the time that does the right thing however Particularly when you're using an SDK as we are with the Octo project SDK that we have installed When you install the SDK you inevitably install it into a different directory to the one in which it was compiled so for example if we use the obj dump tool from our tool chain and We look for the dwarf records the debug records in a program for example. Hello world Then we'll see that the compiler DIR is actually set to something like this So we have user source debug glibc and then some big long path Now in actual fact That is a sub directory from our sys root So the place we really wanted to look is not in user source debug because there's nothing there it actually needs to look in Opt pocky 3 1 3 sys roots and so on slash user source debug Okay, in other words GDP doesn't know that the source code has been relocated into the sys root We could go through it and add lots of dir commands as shown on the previous slide for every single Directory that contains library code But there are many of them it would be rather tedious to do that so we can do this thing here instead we can set set substitute path and I want to substitute user source debug which doesn't exist with the sys root and Then use a source debug from the sys root which does exist so Let's try this out So for this I'm going to use a program Which links with a library actually two libraries it links with the C library of course It's also going to link with the USB library Live USB one so that we can yeah, we can try a couple of things out So let me just compile that. I think I may have a style copy lying around. Let me just remove that And it wasn't okay, so I'm going to copy USB demo from The sample code I have a make file and the code itself So I'm just going to build it as usual and then I'm going to copy this to the Raspberry Pi Just so she can see what it's meant to do SCP USB demo to root Pi 3 and we'll put it in user bin as usual Okay, then over here on the Raspberry Pi. I type you s us demo and It is simply going to print out the IDs of the various USB devices plugged into my Raspberry Pi The exact details of what it prints out. I'm not particularly interested in just now I just want to demonstrate how to debug it So Let's do that Let's not do that So let's start gdb server as usual listening on port 2001 And we want to debug user USB demo and then over here on my Development machine. I want first of all, I want the gdb init file we had earlier So could be that from the good old hello world program and then I'm going to launch this with cgdb So I want to do first of all arm Pocky Gdb Front end that with c gdb that minus x Impractice we will write a script for this, but I haven't For the purpose of this demo to make it clearer. I'm doing everything from scratch almost everything Gdb init USB demo Okay, it's looking good Connect to the Raspberry Pi Set a breakpoint on main Continue excellent So the first thing it does is print out a message I'd quite like to see if I can trace into that. So let's have a go at that first So I type in And I type s for step and this takes me into the printf code Except that I don't see much code. I just see lots and lots of assembler So that is because we don't have the correct path to the source code for glibc So let's just quit from that Yes, because I want to sort things out So what I would do now is to and I've excuse me for a moment because there are some cribs I need to copy from over here So I need to Look for that Compiled here, which is here somewhere believe me. There we go Apologies for the non-professionalism here So I'm using the upstump and I'm looking for the comp dear and I want to look for USB demo So as shown on the slide we find the compiler there is set actually to something like this So the the C library at least is compiled in user source debug glibc that of that and As we now know that actually is actually is situated in our CIS route. So if we look in Pocky CIS routes Cortex and then if we look in user Src debug So each one of these directories contains the source code for the various things including Glibc which is here So I just need to put in the appropriate Substitute path which I will do right away So first of all, let's start to be server over here Rewon the debugger over here But I'm going to set the substitute path command and again I'm going to copy my crib because it's quite a long string So I type set substitute path user source debug and then the CIS route of the Pocky tool chain Exactly should expect So now let's see if that makes a difference So connect to the Raspberry Pi Main continue step one line and then step into printf and Now we see the source code Is it quite nice? So I'm just going to step through this I'm not going to dwell on this too much because to be honest This is quite scary code and I don't really want to look at it But the main point is I can now see the source code for the C library oops So next let's have a look at the the USB library. So this is live USB one So I should be able to do the same thing if I step into that Yeah, but it doesn't work So we see lots of assembler code and we see down here That it is unable to find this file Whoops that file and that's partly because it's looking in dot-dot-slash-dot-dot-slash Well, that's not going to work So let's go back to the drawing board and see if we can find that file. So quit Yes So I know it's going to be in my CIS route somewhere. So I'm going to do a find and We're going to look in opt Pocky CIS routes Cortex something I Suspect to be in user source debug and I'm going to find Core.c and There it is. That is the directory then that contains the file So it's just a copy of that Just to make sure we've got it right What went wrong? There we go So there's the corec file and the other files that comprise LibUSB So I can use a dir command to to deal with that Before I do that, let me just do one little thing and again I'm going to do this from the crib because I really want to keep that set substitute CIS route which we can Have here. Okay, so that will be run when I launch GDP now And that's exactly what I'm going to do. Oh, but I need to remember this path. So it's getting fun, isn't it? So launch the cgdb Connect to the Raspberry Pi. That doesn't work because I haven't started gdb server Now I have okay break main good So now I'm going to add that directory to the search path for source code Okay, and now I can step through my code Just to prove everything still working. I can still step into printf that still works So just like finished to get out back out of that and Now I should be able to step into LibUSB. So step Yes, it is now find found corec and I can step through this Okay, again, I'm not hugely Interested in exactly what this does. I just want to prove that I can actually look at This code backtrades Yeah, everything's all there. So I'm just going to type continue and let that program finish and quit so next topic So the next thing I want to look at is what I like to call just-in-time debugging or how to attach to a running program The situation here then is that you have a system that is running. It has a number of server processes Chugging away and you need for some reason to attach to and find out what one of them is doing Maybe it's misbehaving or Maybe as an example we're going to look at just now. It is a program that started at boot time and it becomes Practical to actually attach to it initially So you boot the system up and then you attach to the program and then you can go into a debug You can do these things using the attach Option of gdb server And what are you going to attach to you have to give the process ID of the process of the program you're going to attach Once you've made the attachment then gdb server will stop that program at a breakpoint So the program stopped then we attach and then we do the Target remote thing as normal from gdb and then we have a regular debug session And then when we're done we can type detach in gdb That will then release the program from control of gdb server and it will go back to running in its normal way So let's try all of those things out So let's clear up here Oops So for this I'm going to be using Well, we need to find a program to to to debug I'm actually going to debug the init program So init is the first thing that runs. It's this guy here and It has process ID one I need to find the executable for that because I'm going to be using that in a moment in the development machine So that it turns out actually we can find out if we look in proc The PID number which is one and CMD That's the wrong one. You know what I'm actually going to I know where it is It is going to be in Espin and if we look for an it L Then there is a slash Espin slash init and that is a pointer to slash Espin slash init SV init So that is the actual executable. So that's the thing. We're going to debug So bear that in mind for a moment So then over here I need to find the executable for that. So that'll be an opt Pocky And then I want to look in the sysroute and Cortex Espin init Okay, so that is the copy on within the sysroute so I'm going to load that into the debugger on the development side and then on the Raspberry Pi side. I'm going to attach to to that process to process one So over here then we're going to do GDP server minus minus attach 2001 and Then I'm attaching to PID one. Okay, that worked that does mean of course that The init program actually stopped at the moment That doesn't matter in this particular case But just bear in mind that you want to be careful what it is you stop and then over here then I need to run the debugger But the thing I need to debug now is not USB demo. It is in fact the thing On the line above Okay, so far so good. So make the connection so now when I make the connection the Source window here This is now put me in the context where the init program was when it was stopped by GDP server. I can type backtrace and We can see that We're currently in libc-ducis call. So it's made a C library call Which is called from libc-ducis call, which are called from something select, which is called from select.c Which is called from check init 5.0, which is called from etc etc So we're in the C library unsurprisingly. We're at a blocking system call called select and We're presumably waiting for something to happen So I'm just going to step out of this Because we're way down in assembly code right now There we go. So there's actually a time out there of some seconds and Where are we now? Let's step up another level Okay, so this is the selects function call that init had made and If you look up a little bit you see actually it does a time out of five seconds. That's why there was that Delay there and now I can just step through this and see what it is. It's doing Let's go to the start of the So this is the main loop within init and it is Make sure everything's looking okay This is the point where it calls check init 5.0 So it's seen if anybody is talking to it. We just step into that and it's doing some stuff with a pipe This is the point where it calls a select function. We just started out a few minutes ago That has a five second time out on it two three four five. Yeah, and then we go around the loop again Okay, so I think that's good enough For for now, so I'm going to let init carry on its way. I'm going to detach So over here we see that GDP server has detached and now everything's back to normal Excellent. So now we need to go on to the next topic Which is core dumps So let's talk about core dumps and why they're useful and what we can do with them So you get a core dump When a program goes wrong The core dump then is what we call the volatile state of the program So it's the data segment the heap anything that may be mapped into the map M map area such as Data from libraries and such and of course the stack So all of this stuff is when it generates a core dump is written to a file called core and we can then use that within GDP we can as we can Reestablish this this environment and then we can see what's going on. We look at these data values We can look at the stack you'll find however that by default you don't get core files There is a limit called our limit core, which is a Per user limit and In order to create a core file you need to increase our limit core to be something greater than zero You also have to make sure that you have the user that's going to create the core file has the permissions to write to the directory where it's going to write the core file and There's also restriction that if the program has the set user ID bit set Then it will not generate core files because that is a security problem The bottom line is that you won't get core files unless you type this command You limit minus C and limited that sets the limit for core files to be unlimited Instantly be aware that these core files can be quite big even on simple programs. They're likely to be a few megabytes but the bigger The data segment and the more stack you have then the bigger the core file So on big graphic heavy programs, you can easily have core files that are hundreds of megabytes Now, where is this core file going to be? stored So by default you will get a file called core CRE and It's placed in the current working directory of the program that crashed which created the core file This is not particularly convenient, especially on embedded systems. You end up with core files scattered all over the place and They may not be possible. They may be your your Your bin directory or your user directories are read only in which case creating a core file in one of those would not work So it's really really nice to have this feature here. You can write to this file a core pattern and This is a specification of where you want the core files to be stored and what you want them to be called The man page for core tells you the full details But here is a good example if we create a directory called core files and We can use specifiers like percent e percent p and This will determine the name of the core files the name of the files rather when it generates a core So percent e Expands to be the name of the executable that generated the core and Then percent p expands to the current PID of that program Having created the core file You will need to copy that to your development machine and Then you can launch gdb like this So now we're not going to attach to gdb server. There's no gdb server involved here We run our gdb from the tool chain We give the program that We tell gdb that the program that crashed so that you know So you can find the there's the symbol tables and Then we give it the name of the core file that contains the volatile state and It should give us something like this so it should show us the line of code where everything went wrong and then we can Look at the variables the the local variables the global variables and We can look at the backtrace So let's try that one out. So back to there and clear up from breathe previously So for this I have another one of my little demo programs It's called may crash In fact, it will crash So from the sample code and looks like there's some stuff in there. Let me just clean that up There's no core file there. That's better. Right. So if I now build that Copy it to the Raspberry Pi and as usual I put it into the user bin directory So now if I run that program It prints out one message and then we get a segmentation fault There is however, no core file So if I type you limit minus a that tells me the various limits for this user and The important one for us is this one here Core block side, sorry core file size in blocks is set to zero. So we're not going to get a core file So we change that we type you limit minus C and There we go. So now If I type may crash It says segmentation fault fault core dumped And there's my core file In this particular case the core file is not particularly big because this is a tiny little program The next thing is as I said having these files called course cat it all over the place is kind of inconvenient So let's create a directory. I may already have created this before And Then I want to tell the kernel to write stuff into that directory. So we do echo core files or slash percent e minus percent p and I want to write that to Slash proc all my time is getting worse proc sys Okay, I think there may be something already in that direction So I'm just going to clean that directory out. So we're in a good state and now I'm going to run my May crash program again So it says core dumped and now if I look in slash core files There indeed is the core file Should be about the same size about 300k. Yeah, that's good Okay So my programs crashed. I have captured the The core file. I just need to copy that to My development machine and run gdb and find out what went wrong So Over here. I want to copy from the Raspberry Pi I want to copy from core files and it's may 500 so 500 is the is the process ID the PID when it actually crashed So now we have that in the development machine So now I can run cgdb as before Almost as before all that stuff. I Don't really need gdb in it at this point Actually, I do need you to be in it because I'll need to find the We don't have gdb in it. I Do need that because it's handy to have the sys root set up So you'll find any library code such like So go grab this once again from the good old hello world Gdb in it and then let's do the same same thing again so Apologize for let's do this a smart way control R Instead of that The program I want to look at is called may crash and then I want to also use the core file which is called may crash 500 return Okay, so this now tells us What happened So core generated by program called may crash It was terminated by six seg V segmentation fault and Then up here You can see the line of code so that this is actual line of code where it terminated So we have P Which is null and we're trying to write a number to a null pointer, which gives us the crash Which would also be able to do things like print a back trace, which I think you can probably guess but there you go We can look at Locals There's only one I can change step frames to step frame one for example That puts this into the main function and I can type info local Yeah That contains Just one local variable I which is currently set to zero So cool now I can look at stat dumps so Coming back to the presentation then we're nearly done now I just have a few final thoughts and then we can sign off So first of all, this is has been a tutorial an introduction to Debugging with GDP There is some good resources In particular the first of all this Peter J. Seltzman book highly recommend that Goes into great depth of GDP of GDP and also DDD in eclipse This is really handy Arnold Reck Robbins book the pocket reference, which it's a little bit out of date But most of it is still up to date And of course there is this book written by this person called Chris Simmons Chapter 14 remember Talks about debugging and so that is the end of the tutorial and I apologize for the really rubbish Demos, but hopefully you'll get something from this If you have any questions, please go ahead and ask them and I will be live at to to answer your questions as best I can Meanwhile, you can contact me. I have a My blog is called looking after the Nipenguin You can follow me on Twitter and you can also link with me on LinkedIn So thank you all very much for for tuning in and I hope you found this useful Talk to you later