 Let me start by asking who's around here. So who's got any sort of embedded Linux experience? All right, that's pretty good. Android platform experience, a few. Oh, pretty good. OK, Android application development experience, a couple. Oh, still not bad. All right. OK, so my name is Kareem Yakmore, and in the next hour or a little bit more, we're going to essentially delve into how do you do any sort of platform debugging inside of Android. And obviously, a lot of people have some experience with this in here. So if you know better than what I'm presenting by all means, please do say so. There's quite a bunch of stuff I tried, and some of it I was relatively successful in getting to work, some other stuff, not so much. All right? So all right, just a little bit of background here. So I, in case you haven't seen any of my presentations, I wrote the embedded Linux book for O'Reilly 10 years ago. Just released the embedded Android one a couple of months back, most widely known in the open source community for getting in trouble on the LKML. I contributed to LTT back in 99. Fortunately, somebody else is doing a much better job at this than I was since 2005. When I'm not talking here, I'm playing with all sorts of open source stuff and helping customers putting in all sorts of wacky devices. The goal of this slide is one thing and one thing only, and it's to say that I don't know everything I wish I did. So hence, please heckle if you think I'm completely off topic. All right, so I just want to start off by a quick recap of the architecture, making sure everybody's on the same page with regards to what's an Android, how does the thing tick, and hopefully moving pretty fast on that side of things because I actually want to get into the actual debugging of the platform as fast as possible. So what you see afterwards is essentially a whole list of things that I think if you're doing platform development, you're going to be interested in, stuff like symbolic debugging, dynamic information, tracing, and things like that, how to get that stuff and how does it work in Android. All right, some very basic architecture. This is the hardware Android typically runs on. You've got yourself an SOC which is coupled to some kind of baseband processor that's talking to some tower somewhere and the SOC actually has everything almost connected to it. It's not one for one model, but it does a fairly good job. This is essentially what you're dealing with when you're dealing with Android. And the design of the stack kind of has that and this in mind, so that's what's in your SOC. Again, I'm not sure I'm teaching you anything new here. A whole bunch of stuff is in your SOC in addition to the core processor that you have in there. All right, so this is the stack. You've got the Linux kernel with a whole plenty of additions inside of it. You've got a native layer up here which has got essentially a different C library than the one you would expect in a standard Linux world. So it's not G-LiP-C, it's Bionic. The main feature of Bionic is, BSD licensed. Otherwise, it doesn't have full POSIX compliance. It doesn't support shared memory as a Sys5 IPC and it doesn't have a locales and a whole bunch of different stuff. It works just fine for Android, all right. Most importantly, there is a hardware abstraction layer. Okay, so if you're familiar with the driver model of Linux and you know nothing of Android, this is probably the thing that you wanna hear about and learn about. So essentially what this allows is, it allows vendors to actually take some proprietary stuff and put it in user space to talk to the drivers and then they don't have to distribute that and under the common licenses that you would expect in the open source world and because they can do that, all of them do it. And therefore, if you wanna get Android running on any commercial device, you need to talk to your sock vendor and have them give you the hell modules because otherwise, your hardware's kinda useless. All right, a whole bunch of native daemons, there's init and toolbox, init does the same role as it does in standard Linux with the significant addition that it maintains a set of global properties which can act as triggers. So if you change the value of a given property, it can trigger the execution of something in its scripts, all right. Toolbox is the same thing with Busybox except its main feature is? BSD licensed. If you know anything of Busybox, this thing sucks. But anyway. On top of this, you've got Dalvik which runs most of the important intelligence that has to do with Android that is in the system services. If you went back a few years ago, Google actually didn't mention anything about the system services until people actually started looking at the sources and these days, if you go to source.android.com, they actually tell you there are things called system services and those actually constitute the intelligence of Android and this really is the main important part that's in between the applications and the hardware that's underneath, okay. If you need to kind of debug some kind of hardware related problem, you're gonna go through one of these system services to actually figure it out, right. You've got the whole Java libraries that were there prior to Android, the Android API that app developers get and the actual apps up there, okay. This is how Binder works. It's a little bit complicated. So Binder, okay, Binder's purpose is to provide, it's actually, sorry, let me start by saying this. Binder is the core communication mechanism in Android. If you don't have Binder, you don't have Android, all right. There's approximately one single human being on this planet that actually understands the Binder driver and the kernel. So the mechanisms of it are somewhat arcane and it is not for nothing that it stays in the staging directory and we were having a discussion about that last month at the Linux Plumbers Conference and it's probably gonna stay there for quite a long time or at least the foreseeable future. So the Binder driver's purpose is to allow a general purpose operating system to have on top of it implemented a object-oriented operating system and that's what Android is, an object-oriented operating system. It's essentially just a remote method invocation mechanism. Everything in Android uses this, the system services amongst themselves use that and the applications when they talk to the system services use this, all right. So one thing that you wanna do when you're doing any sort of platform development or debugging is find your way into this and start talking to the system services directly through Binder, debug them through Binder and that kind of stuff and I'll show you a few examples of that, okay. These are the system services anymore, kind of true fashion. The system services are not one single block but actually quite a bunch of blocks. All of them are actually talking to each other through Binder in some way, shape, or form. The thing that makes a system service, a system service is the fact that it's registered with the service manager which is kind of like the index or directory of all system services which are running in the system. I'm speaking very fast here, I'll promise you, I'm slowing down a couple slides. All right. This is the hardware abstraction layer. The only thing that Android or the upper layers of Android understand about the hardware is the HAL API, okay. How the HAL modules actually talk to the hardware, how they interface with the driver, that's something Android doesn't care about, okay. The only thing that's important is that the manufacturer provides the device manufacturer with, sorry, the SOC vendor provides the device manufacturer with HAL modules that are respected for an API. So long as those HAL modules are there, the upper layers can actually talk to the hardware, I think they can talk to the hardware. Okay, so this is the more important part. If you wanna do any sort of platform development on Android, you wanna essentially achieve the same thing we would have been doing with embedded Linux 10 years ago, all right. You wanna have essentially a host that's providing some forum networking boot for the targets. You connect the target, I keep bumping into this thing. You wanna connect your device also in addition to just serial and ethernet, which is what would have been standard 10 years ago, but also over USB because Android debug bridge operates over USB typically, although it can operate off obviously off of IP, but that's the standard way in which it typically comes out of the box working on. All right. In terms of booting, so you've probably seen this elsewhere, it uses DHCP to get an IP address, it then uses TFTP to get a kernel and then uses NFS to actually get itself a root file system. If you're doing any sort of platform level development, it's nice to have this because you can press reset and you get yourself an updated system with whatever you just compiled, all right. In terms of IDEs, okay, you can use whatever you want. It can be anything that's not on the slide, so just for kicks here, who's an Emacs user, all right. VI, all right. All right, who in here believes that the editor he uses has got the best interface? Come on, let's be honest. So let's entertain the thought here for a second that this would be the rule of the day. Right about now, Steve Jobs is turning his grave. So, this was just following a conversation we were having yesterday at the party, so I just thought I'd put this cameo in here. Anyway, all right, so how do you set up Eclipse to actually browse the sources with it and have it deal with the AOSP, all right. How do you actually use Eclipse to step inside the sources, okay. So there's quite a few steps that are involved in this. Some of them I found to be less obvious than they should have, sorry, they're less obvious than they should have been, okay. Some of the stuff should have been straightforward, but it actually isn't. So let me actually start doing a little bit of demoing here. So the first thing you wanna do is actually prepare the terrain. So you're gonna have to have obviously the AOSP, which is the Android sources, and you're gonna wanna get obviously Eclipse with the ADT plugins. So what you can do is you can just get the ADT bundle from developer.android.com. That's the same exact bundle that an app developer would get to develop applications. There's absolutely nothing different from the environment you would use to actually use to develop the AOSP or debug the AOSP with, okay. So the first thing you gotta do to make Eclipse kind of be able to deal with your AOSP is you're gonna go get that class path file that's in the AOSP itself. So let me actually show you this. I've done it for one tree and I'm gonna kind of try to do it live again. We'll see how that works out. So let me go for development, what is it again? Development IDE, Eclipse, and there's a class path file in here and copy it right here. What does that look like? Let's actually take a look at that. I told you I'm an Emacs user. Here's the class path file, okay. XML file, hey, where'd you go? It went somewhere. Hello. I think you're gone. Ha ha ha ha ha ha ha ha ha ha ha. Point taken. There we go, here it is. Ha ha ha ha. So XML file with essentially the classes that are part of the AOSP, okay. So let's just keep this as is for the moment being. And the other thing that you need to do is actually modify the way Eclipse itself is set up. Okay, so there's an any file that comes with Eclipse. So I went and got myself an Eclipse version and I extracted it in here and installed it and let me go there. So I went to local, where's that? ADT bundle, Eclipse. And in here, there's eclipse.ini file, all right. So you can look at this file if you want to. Nothing really original, okay. A whole bunch of configuration flags and the flag that we wanna have. Why is my interface not responding as it should be? Yes, I'm using Ubuntu AM. That is correct. What the heck's happening here? Okay, let's do it this way. My, ah, that's the problem. You see, when I, my mouse click is actually not working. My presenter? No, maybe if I do this, let me see. Let's stick the presenter out. Oh, that doesn't change anything. Give me one second, let's try this. It doesn't set it up. I have a problem here. You give me one second, all right. I am going to do something like this. Wait a second. Well, he's back for this, but it's not gonna last too long. Anyway, I'm gonna have to take a second, wait two seconds and actually shut the thing off and come back. But anyway, so what did, what, this is the default stuff that's in here. At least the snippet that we're interested in. If you read their documentation, they tell you to adjust those things to these values. So somewhere along the lines, they kind of up their own defaults. So the only thing you have to change here is this XMS 128 megs. Don't ask me what this does, I have no idea. Then the next thing you wanna do is actually import the project into Eclipse. Now, if my mouse doesn't work at this point, I'm gonna, oh, actually it does work. No, it doesn't. It selectively works. Okay, let me do this, okay. You're gonna, we're gonna have to go for a reboot for a second here. Any questions so far? Still this guy actually reboots. No? I'm just wasting your time then. Ah! Good question. Do I recommend setting up Eclipse instead of using VI, shame on me. Or Eclipse, or Emacs or whoever. No, actually I recommend having whatever default editor that you're used to using still around. It's just that if you're trying to step in the sources, it does a pretty good job of doing it, okay. And if you're trying to browse the sources, it does a pretty good job of actually interpreting Java and kind of following the classes and tell you who's calling who and that kind of stuff. So I wouldn't kind of forfeit the use of whatever default editor you already have, but I would definitely consider the inclusion of Eclipse as part of whatever it is that you're already using. Here we are. Okay, so let me go now and grab Eclipse. So this, nope, this ADT Eclipse. Let's start this guy up. We can probably go for a coffee break. It's not the fastest thing out there. It doesn't get any better. Yes, well I, what'd you say about? Come on. Alrighty, so one thing you can do here is, if this thing ever responds, yeah, we're gonna have these gray screens off and on and this is actually a pretty good laptop. So we're gonna go for a new Java project, right. What I'm doing here is just telling Eclipse about my EOSP, right, so that it can actually index the sources and allow me to do whatever I want with it. So let me actually get, don't use the default name for some reason. Still getting funky behavior here. Okay, so let's call this EOSP 43 ELC. All right, don't use the default location because he'd create a new project here. What I'm gonna do is I'm gonna essentially give it a new tree which is over here and I can say next and he's gonna now start loading the project, give it a second here and he should give me kind of like the list of all the classes that are available and I should be able to do finish and then on the left-hand side here I should see that new project that appears in the list of projects that are available to it. Come on. My primary reason for recommending Eclipse is for class browsing, it's partly but the other part is that this is the only way you're gonna be able to actually symbolically debug the Java code that's in the EOSP. So if you want to debug any system services, any system library, anything like that, you're gonna have to go through this. Yes, it is building something in the background. It's because I just have a whole bunch of projects open so it's kind of attempting to build those in the same time that it's trying to load this new EOSP. So that's definitely a challenge with Eclipse. There is nothing fast about it, unfortunately. I guess the upside is once it's all loaded it is actually better but while it's loading like this, at least the CPU is kind of getting used up or something. Oh, there we go. Okay, so next screen should come up and you won't now, there you go. So it shows me all the classes that it was able to spot. Then I say finish and then over here I have essentially the project that's been created and if I scroll down here I can see all of the, not all of the sources because it hasn't actually included any of the C stuff. These are just the Java, this is just the Java code. So one of the other reasons why you want to still keep your existing editor around is for dealing well with all the C stuff, okay? Getting to that, so that still doesn't solve the issue of I can't step from my Java to the C and kind of following from here to there. That's entirely true, okay? There are things that you can do that can alleviate the pain. There are things that I found that haven't been solved yet and which are major pain as I'll show you in a couple of slides. So anyway, this is gonna go on for a little bit. It's trying to actually build this thing so I'll show you what I've got. So essentially it's the same thing. This is another tree which I had actually imported earlier and the trouble is, and let me actually go back to the slides to explain this to you. So the trouble is you have to actually go and fix the class path file that's given to you by default, okay? Because there are some classes which are missing and there are some classes that are added that you have to remove. There's some stuff that you have to comment out. I'll put the slides up there. It works with four, three so you should be able to just kind of copy and paste what I'm doing here and doing it on your side. There's essentially things that you have to build by hand, that kind of stuff, but eventually after changing a couple of things, you can right click on the project and say refresh, all right? Let me actually show you this here. If I go back to Eclipse, I don't know if it's actually had the chance to take a look at that thing up there, it doesn't seem to. So let's try to do refresh here, see if that actually does anything. Well, it seems to be kind of occupied with other stuff, so I'll let it do its job, but essentially when you expand the project that you will have newly loaded, before you do those fix ups, all right, and you refresh, what you'll see is you'll see a bunch of items with the X on it, which essentially means that it's not compiled, right? But once it is compiled, you'll get this thing with a triangle with an exclamation mark and that's fine, that's exactly what we want, all right? Because we can actually start browsing classes and see call chains and that kind of stuff. So if I go here, for example, and let me actually show you what was I looking at earlier? Yeah, let me show you this. So if I go to a framework space called Java and then go to Android app, there is a context impulse and here, where is he? There he is, context impulse, all right? Context impulse has got this function called get system service, anybody know what that does? Get a handle on the system service, that's the API that app developers actually use. So I actually can actually right click on this thing and I can see open call hierarchy, okay? And here I'm gonna, it's gonna, those three dots that it's showing there means that it's actually looking for who's calling this, and it will give me a full list of everything that's calling on this method, all right? It takes a little bit of time because this one's actually pretty popular. And then you can do the reverse, which is actually right beside it, which is I wanna see who it is calling and then we'll show you a whole list of methods once it's actually done with that. Come on, don't disappoint me. So yeah, while it's doing this, let me show you on the right hand side here. So this is a small screen, obviously this benefits from having a large screen, but if you go here on the left, on the right hand side, you've got the class and you've actually got everything that's in that class that's being described to you here, okay? All the methods, all the variables, and so on and so forth. So for browsing Java sources, this is pretty useful, right? A lot more than trying to grep through the files and so on and so forth, okay? What else while this is doing? There's a bunch of shortcuts here that you can actually also use so you can have something like control E and that will give you the list of files that you recently opened and you can play around with that to actually open some stuff that you've opened not too long ago, come on, for God's sake. Can't you finish up so I can move on? Okay, so that being said, so it does build the actual AOSP in terms of checking for it to compile, but it doesn't actually replace the main command line that you have on the command line. You still have to go on the command line and type make if you actually want to build your AOSP and flash it on the device, okay? Eclipse in that regard doesn't substitute the command line and it doesn't act the same for this as it would for, say, an application, right? In the case of the application, it would build the entire thing and get you an APK. There we go, okay, it finally ended with some kind of visual quirks, but anyway, here are all the callers of GetSystemService in the AOSP. Alrighty, so let's say I wanna monitor the framework, okay? I'm kind of leaving Eclipse on the side here. I'll come back to it. The next thing I wanna show you is how I can actually step through the code. This is very interesting, but I wanna kind of go on here first. What are the tools which are available for kind of monitoring the system, all right? You've got a whole bunch of command lines which have to do with the native framework. So let me actually go here and got myself the AOSP that I already have built here. So obviously I'm using the emulator for simplification purposes. You could do this same stuff on a real Gizmo. This, this, and then this guy here. So this is for three. Pretty much all the tools I show you here are actually available in previous versions and will likely be in the next versions. So something like sked top, all right? So this is gonna start checking who's using the most amount of scheduling time. It looks like top, but it's not exactly the same kind of functionality. You've got LibRank, which will allow you to see which libraries are being used most often. Come on. Respond, will you? Now my laptop just froze. I'm gonna kill it. Hello. All right. We go again. One last time. While this is booting, questions. Please. Yes, sir. Sorry? That's a really good, really, really good question. High five. No. Okay. So the, that's latency top. There is, shows you sked top. There is obviously a PS. It doesn't have the same kind of flags as the busy box PS, but it's good enough. There is also, come on, let me do this. Hopefully this time we get better luck. You know, this God bless Lenovo laptops with Ubuntu. Going back to this guy. Show map is, I like show map. So let me show you this show map for the PID. So let's do show map for the system server for example, which is still booting here. So it gives you the whole list of libraries and you know, the amount of size of, you know, RAM and virtual space that they're actually using. So if you're trying to look at the framework, dumpsys will essentially poke all the system services and dump you the state of each single system service. You can obviously ask each one of the system services for its status independently. So I can do dumpsys status bar and that's going to give me the status of just the status bar. All right. You've also got service and most especially service list, all right, which will list you this list of system services which are available. There's a few things you can do with service that are really powerful. I have a few slides further down while I'll show you how you can actually use service to directly call a system service without using any sort of program or creating an application that does get system service and actually talks to the thing on the other side. Obviously overall you've got stuff like a logcat, very useful tool. If you've never used this, I don't know what you've been doing with Android. Very, very, very, very practical. Okay, so this maintains a buffer in the kernel. I'll show you a diagram that explains how this works and it just reads it when you do a logcat on it. You've got a dump state. If you've got a device in the field, this is very useful for bug reporting purposes. It goes and gets stuff from slash proc, from PS, from logcat, from a whole bunch of different sources and gives you kind of a dump of the state of the system at this point in time. All right. This dump state is actually privileged and requires root, but it runs, it started in the background listening to the sockets and there's another tool called bug report which essentially kind of connects to that socket and can essentially retrieve what dump state actually has. So you get any device off the market, even though you don't have root on the device, you can do bug report on it. Okay. Alrighty, stop. That's enough. Okay. Watch prop, get prop, all right. The properties that are maintained by it. You can see them with this. Watch prop will allow you to monitor the changes on those things, right? Watch props, there you go. So obviously there's nothing being changed. It's not going to do much at this point. This is how logcat works, okay? So logcat essentially uses the logger driver that's in the kernel. The logger driver in the kernel maintains a whole bunch of different buffers. When you do logcat, the one you're actually looking at is the main log, right? These are circular buffers. So essentially if you are writing too much stuff in the logger, you're going to overwrite what you had before. It used to be that the ASP itself wasn't that verbose, but these days with the newer versions, I seem to be seeing itself, not me without even adding anything from my side, it actually overwrites its own stuff. So there's a bunch of classes that will eventually map down to liblog. This really is the central point to the thing, and they have a hack where essentially if either those slog functions or log functions are sending anything to the radio, they have an if on it and they'll send it to the different buffer instead of having it kind of specified in advance. And obviously kind of logcat uses that to, it reads straight from that driver, okay? Of all the mechanisms that are in Android for dynamic instrumentation or dynamic monitoring, I found this one to be the most reliable. It's not necessarily the one that is the most powerful, okay? Or the one that can give you the most information, but it's the one that kind of always works, okay? And if you want to start interfacing with the framework, okay, so the framework is up, I can monitor it, but now I actually want to do stuff with it. There's a bunch of tools that allow you to do that. I'm not going to do start or stop, but essentially if you do start, stop, what will happen is it will actually kill the zygote process and that means that anything that's Java-based is gonna die, all right? Very useful if you're doing any sort of HAL development because you can actually stop the framework, do whatever it is that you're doing in the hardware and then do start and then the framework will come back up and then the HAL module will start talking to your hardware again, okay? So that's pretty useful. Service call, all right, so let me show you this. Who's used service call before? Okay, so let me show you what service call can do. So let me do this, always on top, let's keep this here. So if I type service here without any parameters, it tells you that it can list the system services, it can check them for them and then it has the service call thing. So let me show you an example of that. So service call status bar one, all right? So I've just expanded the status bar here. All right, service call status bar two, all right? And this can be entertaining, we can have fun here, all right? There is other stuff that I can do. I can do call status bar five and then S16 Bluetooth, all right, 32.1. You can see the Bluetooth icon up there, all right? Now it's there, now it's not, all right? Let's go for the alarm, alarm clock or alarm. I think it's alarm clock, alarm clock, all right? Alarm clock is up there and it is gone, all right? So what am I doing here? What the heck is he typing? The first digit that you see one, two and five, right? Anybody familiar with EIDL files, right? EIDL files are the files that define the interface of a system service. That's how they become callable through binder, all right? And EIDL files, what you do is you define an interface and you say the interface of the service is Fubar and whatever, Toto and all these other methods, all right? The order in which you declare them in the EIDL file gives them an implicit sequence number. That's the number that you see over here. I just happen to know that the number one and two don't take any parameters and I know that five actually takes a parameter and if you actually look in the EIDL file of the status bar, one is expand, two is collapse, number five is set icon visibility, all right? And essentially what I'm doing after that is passing the parameters by hand. This is very useful for testing a specific system service with by bypassing the entire app API that's given to app developers. So if I go back to my diagram here, okay, not on top anymore. If I go back to my architecture diagram, this right here, okay, when I'm using service call, I'm talking directly into this box. I am bypassing this thing up here. There is no Android framework involved in those calls, right, so that's very useful. And if you add a system service, you want to test a given system service, that's a good way of doing it. What else is on that list? Here. AM, okay. Anybody familiar with that? Community manager, okay. This allows you to actually send the intents on the command line, right? So let me actually show you something like I can never remember the command line of this thing so I'm actually just gonna grab my own notes. I'm gonna stop the sort, find that name, my LTT, whatever it is. There we go. Stop. Emacs, and then let's open that thing here. Middle click, which doesn't seem to work, but anyway. Here we go, this one liner. Copy, go back to the command line, do something like that. Obviously you can't do that. Okay, so let me actually type it and you'll see what happens when I do that. Actually, well, okay, the wifi's not on, it doesn't matter. So I do this and it brings up the browser and it tries to go to a certain location, okay? So what am I doing? If you look at the command line, I did essentially AM start. So I'm actually sending an intent to start an activity and what am I starting? I'm starting something of type view. I'm telling you to view something and the data is a webpage. So the components or the app that can respond to viewing web pages happens to be the browser. The browser comes up, right? Very useful way of actually sending intents on the command line. You can send broadcasts, you can send broadcasts, and you can activate services also in the same fashion. Another one that's interesting is PM. So PM allows you to talk to the package manager. So whereas AM is to talk to the activity manager, PM allows you to talk to the package manager. So for example, PM list packages, what's installed on my system? You can see that there, all right? You've got a WM for talking to the window manager, you've got some of the other guys over here for playing around the stack in various ways. So far so good? Yes sir, no? Question? All right, stop me if I'm going too fast because I'm just gonna, I just got a lot of stuff in here. All right, so if you are in the AOSP sources and you're working your way through stuff, there's a bunch of things that you should know about which are set up by building and vsetup.sh, all right? So if I'm in the AOSP sources, if you've never played with this, you really should. So I can do something like this, building and vsetup.sh, I do a lunch, and I can do something like go to your system server.java, and this just CDs directly to the directory where that file is located, all right? If I wanna build, if I wanna build that part of the tree without having to rebuild the entire tree, I can type mm, okay? That's gonna compile just this package. This is really useful for just testing the local changes that you've done. It doesn't regenerate the entire tree though, all right? It doesn't regenerate city images that you burn on the device, but this is useful, okay? You've got jgrep, cgrep, resgrep, which are greps, but for C files, java files, and resource files. So they only look for the string that you're looking for in those specific files, all right? And there's croot, which is really useful because like now, if you haven't played around with the AOSP at all, you'll find that the tree can be very deep, all right? It's not uncommon to be seven to 10 directories underneath the top level. You know, having to type cd.dot, slash.dot, slash.dot can be bothersome and CTS inducing. So what you want to do is you'll type croot and voila, you're back exactly where you were, all right? So that's really, really useful also. Okay, obviously last point kind of goes without saying the tree is big, okay? This 4.3 is about seven or eight gigs of sources. It takes a little bit of time to kind of get the hang of it. Oh yeah, that's where this thing is. So, you know, just have some patience. All right, some symbolic debugging basics. As the gentleman was saying earlier, the major problem you have with Android is you can't just use one debugger. That'd be too easy, right? So you've got different debuggers for different things. DDMS is used to interface with Dalek. Then you've got GDB and GDB server to talk to the native layer. And if you're doing any sort of kernel stuff, then obviously you're gonna go with JTAG, all right? The two last ones should, if you've done any sort of meta-linux work before, be kind of second nature, or you've played around with those things that I don't really need to explain too much about them. But the obvious thing is how do I actually debug the framework with Eclipse? That's the really interesting part, okay? So let me actually bring Eclipse up. Yes. Nope, CD local, ADT Eclipse. Hope for the best. So while this is loading, please ask a question. Now, any questions up to now? Yeah? All right. So if you wanna remember something from everything that I just showed you here, service call, very useful. If you're doing any sort of Hal debugging, it's a godsend. Come on. So do we actually have something load for, let me actually walk you through some of the slides while this thing is loading. So the first thing you wanna do is actually, and I actually just did a mistake here. I'm gonna cry. Okay, I gotta shut this guy off. Yes, absolutely. How do you find what? The arguments. Oh, the arguments. How do you find out the arguments of the AIDL file? That's a good question. Let me actually, so let me actually, while this is shutting off and restarting, because I actually have to do that, let me start DDMS. My emulator is still running in the background, because DDMS itself is gonna take a little bit of time to actually get started. So let me show you here. All right, so the AOSP here, okay, if you go to Frameworks Base Core Java, that's where all the API for, that's exposed to app developers is, and that's essentially where the definitions of the system service is located. So if I go, for example, inside Android OS, and let's say I wanna talk to the Power Manager, right? The Power Manager has a definition, which is ipowermanager.aidl. Here's the AIDL file, and essentially here's the interface definition. So number one is acquire wake clock, number two is release wake clock, so on and so forth. The status bar, and the parameters are right there. The status bar which I was just showing you is actually in a different location. It's in com, android, internal status bar. There's a status bar service, and if you open this guy up, you've got exactly what I just showed you earlier, expand, collapse, number five is set icon visibility. All these AIDL files will actually generate Java files, and you can actually go look at the Java file itself if you want to. And in fact, Eclipse is actually able to look at those Java files, and I'll show you an example of that in a short bit. So now that I've started DDMS, DDMS is actually the, so let me actually touch the system server here, and this is gonna take a second to load. DDMS talks to the remote target through ADB, and the language it talks is something called JDWP. Anybody know what that is? Java to bug wire protocol, okay? It's a wire protocol specified in the Java spec. You can actually use JDB on the command line if you want it to, and talk to the port number. So Eclipse talks to DDMS, which then talks over to the device. When you want to debug a certain Dalek process on the other side, you have to actually mouse and click, use your mouse to click the actual process. You can see all the Dalek processes here, and I just clicked on system process. That's the system server, okay? And what you see happening when you click on it is by default, there's a port number associated with every one of the Dalek processes, and it immediately gets 8700, okay? And 8700 is the port that's gonna be used by Eclipse to connect to that process and debug that process. So this is taking some time, but there we go. Is it coming up? Yeah, good. So now I can actually start Eclipse. Now that DDMS has started, which is essentially what I'm trying to show you here, some of my slides that I think I shut off. So let me go back to my slides here. Okay, so yeah, get port 8700. Oh yeah, so that's one thing that's important. I actually tried to use the DDMS that's in Eclipse because actually the Eclipse bundle that you get from Google has a DDMS inside of it, but for some reason, if I use that one to connect to the virtual machine, in this case, QEMU, but whatever, Target, then when I try to debug, it says connection refuse. I haven't figured that one out, so whatever. So inside Eclipse, which, there we go. We get this warning here because there's another clips already running. So anyway, there is, I have a debug configuration, debug configurations. And the debug configuration I have here, which is for a remote Java application, that's really important. It's not an app, the apps are actually up there. It's an Android applications. So the remote Java application is AOSP4 and it's got essentially connection type, a standard socket attached. There's a socket listen by default that's selected. Make sure you don't have that, otherwise it doesn't work. Localhost 8700, all right? And then I can do essentially now that this thing is coming up. I can actually tell it to debug, so let's click on the little bug here, whatever. Okay, scroll down to AOSP4, there we go. So now it's gonna connect, Eclipse is gonna connect to DDMS, all right? And what you're gonna see here, once it's done, on the left-hand side is the bug window, you're gonna see the actual Dalvik process that's selected. In this case, it's the system server. And I can actually see all the threads that that process is actually running. And I can actually also, I'll obviously see them here in DDMS, but I wanna see them here because that's what allows me to actually debug this application. And while this is loading, let me actually go on the command line and type what I'm eventually gonna wanna type. So remember what I did with the collapse and expand of the status bar here? So let me do this, let me keep this on top. Oh, it is already on top. So let me go back to the command line here. And I think I have another, yeah, there we go. Service call status bar one, okay? There we go. So now it's loaded, and I have a few breakpoints here. Let me actually show you those. I have a breakpoint here in the code of the status bar system service that actually expands the status bar. And what I do, if I actually send the call right now, it's now jammed, and I can actually step, all right? So I can actually step here in my system service, okay? This is a system service that's running. I can step here into over. And if I actually go look at what happens in the notification, let me step into that. Give it a second. I did click F5. Did I not? Where are we? Okay, let me let it continue and let me actually do it again. So let me do, I do this. Okay, so now it's stepping through this guy. Step over into, hello? Well, it's somewhat jumped over. Anyway, you can see where I'm going with this. I missed something here or another, but I can actually step through the code, all right? So, and I can do this with something else. So for example, here I have a breakpoint on the clicking of show all applications. So if I click here on show all applications, hey, oh, see, it didn't work. Why did it not work? Anybody know? I'm not connected to the right process. If I go to DDMS, you see here, there's this green bug on the left. That means I'm debugging that process, right? I need to select the launcher, which is this guy here. Now he gets 8,700 and I can go here and it's gonna take it a second, but it will actually load this. Just give it a second here. I think that's what it, is that what it just did or is it still loading it? Let me actually collapse a few things. Oh, I have to relaunch the debug. Okay, go here. There we go. So it's not yet connected. There we go. So now it's connected. And now if I actually go ahead and go back home, click on this, it should hit the breakpoint and it has, okay? I can step into that. For some reason my F5 is not working, whatever. So, or here, whatever. Continue running after that. And anyway, you see what I'm going to do with this, okay? Make sense? Yeah, okay. Alrighty, so that's for debugging with Eclipse. Make sure you got the sequence right. The slides kind of show you how to do that. I showed you this thing. Yes, this. You can debug multiple processes, I just showed you. Now, GDB, right? Because you should remember earlier I said, DDMS will allow me to debug my Java stuff. What about actually debugging the C stuff? Can I actually do that? The first thing you need to do is make sure that the Android MK of the application, or the C code that you're trying to debug, has local C flags equals, or plus equals hyphen g, or hyphen ggv preferably, and then make sure that the build system doesn't strip the C code, because it will do it by default. So you have to tell it, don't strip the code, all right? And then after that you can either attach to a running process, or you can run one, you can start one straight off the command line. So that's exactly what I'm gonna do here. I have service, remember that service command that I typed earlier? I've actually compiled that with the code that actually, sorry, the Android MK, which has essentially telling it not to strip the thing, and now it's waiting for GDB to connect essentially. Okay? The next thing I need to do is actually do some port forwarding. So let me actually do that, I did. So ADB forward, TCP, two, three, four, five. Come on, TCP, two, three, four, five. So my ports are forwarded, and the next thing I'm gonna wanna do is just essentially connect with GDB over to the target. So it replaced, okay, so if we've got pre-builds, GCC, Linux X86, ARM, ARM, EABI, seven, BIN, ARM, EABI, GDB. All right, we're inside GDB. I wanna tell it where the file is, so out target product generic system, BIN service, so far so good. The next thing I wanna do is connect. So target remote, local host, two, three, four, five. I am now, or should be connected. He doesn't know where I am, but I'm connected, all right? And then I can do some, I can set a breakpoint. So breakpoint in main, continue. Come on, okay. Next, next, and so on, all right? So I'm essentially stepping through a standard GDB session. That works fine enough. Where it really gets messy is multi-threaded, all right? I haven't been able to get that to work. So apparently the GDB server that's in the USB doesn't actually have multi-thread support. So what happens is you set a break, so it's a little bit funky because you actually have to do the math of telling it where the location of the library is, and then when you connect to the process and actually set a breakpoint, it sets the breakpoint, but when it hits the breakpoint, it gets a sick trap, okay? Because essentially GDB server on the other side doesn't understand the threading model that Android has, okay? So unfortunately I haven't been able to figure this one out. Some people seem to indicate that they've solved this by actually compiling a GDB server with threat support. Has anybody got this working? Kind of just being curious here. You have? Yeah, okay. I'm sorry? Not with multi-threaded. Okay, so you just used the standard GDB server that was in the USB and it worked for you? Huh, we need to talk. So just to give you an idea here, when you load a dynamic library, so like I'm doing here, I'm giving it the dynamic library which I wanted to use, you have to tell it where is the offset and you can figure that out by essentially doing a cat on the maps of the process and then calculating the offset with what the elf binary says in terms of the beginning, from the beginning of where it is and then you can set breakpoints with that, okay? What? Sorry? Why? You're right, on regular embedded Linux, this works out of the box. I'm not too sure exactly why I have to do this with Android but it might have to do with the fact that it's using a different linker than the standard linker that G-Lipsy has. Yes, but there's true, yes, I've tried that, but this didn't work for me, not in Android, okay? I set SOLib, absolute prefix or whatever, that didn't work for Android, at least for me. I don't know if other people have got other experience with it. Has anyone got that where it worked with Android? Just doing the set SOLib with Android, yeah? Okay, well I have. Okay, interesting. So JTAG, there's really nothing magical about JTAG in terms of Android, it's JTAG, okay? It uses the same stuff as before, okay? Now, in terms of dynamic data collection, okay? The Logcat stuff works out of the box, so that's, at least for me, always my fallback. F-Race is kind of interesting because they've actually went out of the way to add support for F-Race in 4.1, which is about almost a year and a half now ago. So they have this thing called SysTrace, which actually is a Python script that talks over ADB to an A-Trace binary and then it collects information and it generates an HTML file which you should be able to view in a browser, except, and it shows you something like this, okay? The trace, all right? Except I can't get this thing to work. I can't get their tools to work, okay? I can get F-Trace to work, but the moment I use their tools, F-Race goes dead, okay? I'm not able to collect anything with their stuff. And the other thing that I really had a really hard time with is their traces they generate. You can only read with Chrome. You can't read it with Firefox, okay? It is HTML, but it will not work in Firefox and that one actually had me lose a couple of hours and it's not set anywhere. It doesn't say anywhere in their documentation which kind of leads me to believe that they're all using Chrome over there, but anyway. If you want, but it is interesting to actually take a look at what's inside because essentially they have instrumentation inside the stack that's using the stuff. So the events are being fed. So in other words, so long as you don't use A-Trace and SysTrace, you should be able to get F-Trace to actually collect the information through you and actually display it to you, okay? So then just forget about their tools and use whatever that you've been used to using. Now with regards to, so yeah, there's A-Trace kind of, it gives you some information here about the command line it has and the categories of stuff it does. Again, I haven't been able to get this one to work. Has anybody got this to work on any device you have? Out of the box on it? Yes, specify. Okay. First try didn't work. Okay. On the emulator or? Okay, that's it. So it depends on the device. That's what I've seen. So for example, on my Nexus 7, I can't get this to get, because I thought, okay, maybe I'm screwing up, right? Maybe I can get it to work on the internet because I'm not doing something right. But on the Nexus 7, sorry, I wasn't able to get it to work. But from reading the posts on the internet, it apparently is very, very, very board dependent. So maybe the Galaxy Tab 2 actually has that fixed for some reason. Yeah. Now with Perf, unfortunately, it's a little bit more difficult, all right? Let's just say that it's not well supported on ARM, all right? And let's, sorry? There are some patches that, so for example, there are some patches for ARM for plain Linux, but not for say ARM for Android Linux. So for example, I tried getting it to work on Panda with ARM, with Android, sorry, and Beaglebone with Android. And in both cases, I wasn't able to actually do that. On which board? On Panda. On Panda? Is that the same AOSP that TI distributes, the same plain vanilla? Okay, then I'll, there's a couple of people I wanna talk to here. All right, benchmarking. Let's just say that some manufacturers have taken the habit of actually checking for which application is running, and then they spice their CPU up to actually make it so that the benchmark kinda gives better or more glowing reviews of what it is that the hardware is doing. There are quite a bunch of benchmarking tools out there. If you wanna take a look at those, I don't have enough time to actually run through those things, but there's a whole bunch of apps in the market that actually do benchmarking. So if you're into that, you might wanna take a look at those things. So it's a little bit primitive at the moment being. I mean, I can't say there's a silver bullet for doing everything. It's a really uneven terrain. So I showed you with Eclipse, I can step through the Java code without a problem. I can step through C code without a problem so long as single-threaded. Some people have actually had more success with that than I have. Some stuff has just not been working, or hasn't been working evenly on different boards. Your luck will vary depending on what kind of hardware that you've got. There's a few things I forgot to mention or kind of as I was preparing this, I completely forgot and I thought about it afterwards and thought I should mention. So there is this trace usually in the AOSP, you can just use it as usual, like a lot of Linux, nothing really revolutionary about this. There's something called the BuggerD which kind of runs in the background. And if you set a certain environment variable, if a program crashes, it will hook to that program and essentially you'll be able to use GDB to actually do a back trace on the actual crashing process. So that's not bad. There's something called tombstones. If you've got a binary that crashes, it doesn't really do a core dump, but it creates a file in data tombstones that kind of does a back trace so that can be useful. And the same thing with ANR traces, application not responding. So if you've got any applications that's doing too long latency, it will dump an ANR trace to allow you to check that out. Which pretty much is all I've got and I've passed a time of five minutes after speaking very, very fast. Any questions? No questions? All right, well, thank you very much. And I'll see you tomorrow for the next one.