 to see so many people here, really excited. This is my first time at DroidCon India, and actually my first time in India, so super excited about that. Right, so my name is Eric Andre, I work as an Android developer at a company called Badoo, and I'm here today to talk about memory dumps, and specifically how to make better use of memory dumps as a developer for debugging and also to help you develop your application. Before I get started with the actually interesting part, let me just say a few things about the company I work for, because probably no one, well, probably very few of you have heard of Badoo, we're not really very big in India, but Badoo is a social networking platform where you can meet new people as friends for dating and so on, really cool place, and we're quite big, we have more than 20,000,000 users worldwide, 190 countries, and our offices are in London and Moscow. Also, we are hiring, so if you're looking, please look me up later, we can have a talk. Right, sales pitch complete, let's actually talk about memory dumps. First of all, I should say that I am crazy about memory dumps. I think memory dumps are one of the most underused kind of techniques or tools available to us as developers, which has been around for a very long time, but we haven't really made very good use of them, and hopefully after this session, some of you will possibly like memory dumps almost as much as me, or at least you will know a little bit more about them. So I'm guessing that those of you in the audience who are developers, probably most of you, have at some point used memory dumps for one or another reason. So I'm not gonna bore you too much with the basics, but let me, before I get started, just say a few things about what memory dump actually is for those of you who haven't used them. So what is a memory dump? Well, a memory dump is basically a snapshot of your application at a certain point in time, containing a lot of information about the application state. So what is in a memory dump? Well, a memory dump, first of all, contains all of the objects which you have created during the run while the application is running, which are the instances, but there are a couple more things in a memory dump which are very useful to you when you're analyzing them. And first and foremost, we have classes which are like you're familiar with in Java, basically the definition of how to interpret the data in each instance. Of course, it doesn't contain everything you might find in a Java class. For example, it doesn't contain the methods, it's all about actually the data. So it will contain information about, say, the fields and so on, instance fields, static fields and so on. However, there's a third component to memory dumps which you couldn't really do without when you're trying to analyze them, and that is strings. Strings in memory dumps come in a lot of flavors, but there are a few main ones which are extremely useful and the first one would be class names. Without class names and field names, you couldn't really make any sense of all that data you have. There are also some other kind of strings in memory dumps which are not class names or field names more related to actually runtime-generated strings, which we will go into some detail in a little bit. And those strings could be stuff like maybe some JSON data which you just fetched from your backend. You could be some strings that the user entered or some strings that are shown in the UI, for example. Right, so that's the memory dump. What do we do with memory dumps at the moment? Well, it turns out that the current use case for memory dumps is extremely limited. Basically, the only thing we do with memory dumps at the moment is to analyze memory-related issues. And first and foremost, memory release and what is very connected with it, problems with running out of memory. And I'm sure that as Android developers, if you look at your crashes on Hockey Apple criticism, I'm sure there's gonna be other memory errors there. It's a pretty common occurrence. A lot of times, there isn't much we can do about it. So it's something that's always there. And for doing this, we have some very good tools available. First and foremost would be Eclipse Memory Analyzer, which is a very nice tool done by Eclipse, which allows us to dig into memory dumps. However, all of these tools which are available to us, they are very generic Java targeting tools. So if you're an Java enterprise developer or a Java desktop developer, you are basically using exactly the same tools as the Android developer to look at memory dumps. And this causes some problems. And these are problems that I've been trying to solve over the last year or so. And what I've been thinking about, because I've been using memory dumps for the last, I don't know, seven or so years to deal with different kind of issues on regular Java and Android. And the thing that struck me was that we have such a huge wealth of information available in memory dumps, but it seems like we're making very poor use of it. Especially when you look at Android and the way we use the tools currently, it seems like there is so much stuff in there which is very, very Android specific. And when we're using tools which are designed for desktop Java, we're missing out on a lot of cool stuff. So if you just look at the data which is Android specific in a memory dump, there is some basic stuff. For example, we have everything that's related to what a user was seeing on the screen when a memory dump was captured. For example, we have the whole view hierarchies which describe what a user is seeing, the activities which are connected to the view hierarchies. And we also have a lot of background stuff which you wouldn't see if you're just looking at the device. For example, we have the states of different Android services. And depending on what the application does, you might have some data related to the API that you're using. For example, you might have data related to GCM, you might have data related to whatever event bus you are using in the application to dispatch data or handle background operations. Also stuff like event queues, handlers, and loopers. All that data is right there in the memory dump. And sure, you can use Eclipse Memorabilizer, dig out all that data from in there, but it's really gonna be a pain and it's not gonna be worth it for you. And what more? Besides just looking at all this data, I want to be able to actually make use of this data to do something meaningful. And when it comes to memory dumps, we're already using it to analyze crashes related to running out of memory. But what if we could use memory dumps to do more? What if we could use it as a generic tool for solving all sort of crashes or problems in applications? Because, well, most of the time, the data we have available when we're looking at crash is gonna be enough, right? If you get a good bug report, you might have a stack trace, you might get some steps to reproduce, you might even get a video possibly recording what actually happened on the screen when the crash occurred. However, in most cases, we don't get so much data, we might just get if we're lucky a stack trace or so. But imagine instead, when you're analyzing a crash, if you had actually a whole memory dump to deal with, telling you not only it's stuff like what the stack traces were, but also telling you exactly what went on in the device at the moment when the crash occurred and possibly even some more data. So this was my starting point when looking at what we could do with memory dumps. And first, I wanna highlight one of the more, well, easy to show use cases here. It has to do with working with view hierarchies. So all of what I'm gonna show you is part of a package of tools we have developed internally with Badoo and we have now open sourced, which deal with memory dumps. And the part I'm gonna show now is part of something we just released yesterday, it's called hproviewer, which is a Java desktop application for dealing with Android memory dumps. And the first case, use case, is gonna be dealing with view hierarchies. So before we get started, just, what is a view hierarchy? Well, view hierarchy is basically the stuff that you would be defining in your XML when defining the layouts in Android. So it's basically gonna be all your views, your view groups. Well, yeah, exactly, what you put in XML. And when actually looking at this data in a memory dump, there are some very basic properties of all Android views which we can make use of to visualize this data. So for example, if I wanna deal with view groups, I don't necessarily need to know if this is a linear layout, if this is a relative layout, I just need to know where is it, which is defined by its position on the screen, left, right, top, bottom positions and so on, as well as which views are inside a view group or which view groups might be inside a view group. There's some other properties as well, which we'll look a little bit closer later, for example, stuff related to how it visually represented on the screen, which we might add as a nice bonus. But this is basically all you need to know to make, well, to interpret the data in a memory dump related to view groups, these very simple parameters. So what I started out doing was that, first of all, I wanted to know where are my views on the screen? So this was basically the first step in showing something at least. And possibly if you're sitting at the back, it might be very hard to see some of this stuff, although I'm gonna release the slides afterwards, so please do take a look there. So right, so the first step would be to actually visualize where stuff is on the screen. And what you're seeing here on the right-hand side is a view group. And you might say, this looks pretty useless. I can't really make any possible use of this. And you will probably be right. So sure, we can see that something is going on here. Sorry, so I'm gonna grid here, which might tell us something, not much, though. It's a starting point, though. But what could we add to make this more useful? Well, let's start with something super basic. So one of the fairly common views which are used in Android would be the text view. And it turns out text views are also very easy to deal with because the only important thing about them is that they have a text in them. And sure, they can do some fancy stuff. They might do some formatting or whatever, but if we just focus on a text itself and add that, let's see what this would actually look like. So this is actually the same view group now with text added. And okay, sorry, I can barely read this myself from here, but I can tell you what it might actually be. So we are seeing some stuff here. It says add photos, add videos. You have three private photos. And oh, there's actually, that would be actually my name right there. So based on this, we can now see that this is probably my profile. And if you had used Badoo, which actually is where this data is coming from, from our application, you would see that this is the user's own profile. So now we know a little bit more. It's still not super useful, but there is actually another interesting piece of information in here. If you could read it up here, it actually says connection loss. This you're using Badoo in offline mode. So now we're actually getting one piece of somewhat useful information, which is that whenever this dump was captured, the user was in offline mode and he had no network connectivity, which might tell us something about this data application as well. Okay, what could we add more to? This still doesn't look very impressive, so let's make it a bit more flashy. And what is more flashy than colors? Boom, let's add some colors. Okay, this is still perhaps not very useful, but it makes it look good, which is important when you're doing a presentation. So this is simply using color doorbells, which you have probably all used, which is simply a doorbell based on a color, not much more. Right, so what could we do more to actually make this a little bit more visually appealing? Well, there's another quite useful piece of information in memory dumps, which turns out takes up quite a big portion of it. That is bitmaps. Bitmaps, well, images in this case, might reveal even more what's going on right now. So if we add bitmaps to the mix, we are actually at a point where we are, we have almost recreated exactly what was on the screen at the moment that this dump was collected. And well, actually this would be almost exactly what the user was seeing at this time. Of course, there is some stuff here, like formatting of the images, which doesn't exactly turn out to be exactly that was showing, but still you can actually have a very good idea of what's going on. And what's more, as I'm gonna show you very soon, you don't only get to see what the user was seeing at the moment we capture the memory dump, but you can also see a lot of other stuff which the user wasn't seeing exactly at that point. So let's skip over to the very quick live demo, which probably as demos usually go, will be a disaster because demos never work when you're doing it at the conference, especially when you're in full screen mode and you cannot see the mouse pointer. Come on, escape should dismiss this. Oh, nice, that's not what I wanted at all. Let's see if we can do this instead. Right, yeah, okay, here we go. Okay, so I'm gonna switch over right now to actually the application. I've done HPropView, like I mentioned, which is all available on our GitHub page, the Badoo GitHub page, which is also gonna be linked in the presentation. So right, so this is the tool and it actually contains quite a lot more than what I just showed you because that's basically the flashy part to it. It does contain a lot more useful stuff for which we'll actually give you information about. For example, let me briefly show you information about, whoa, there we are. See, writes. So here actually you have exactly the same screen as I showed you. This is coming from a real memory dump. I captured it a bit earlier. Okay, actually it doesn't, actually let me switch over here. Okay, so besides actually just showing you what's going on in the screen, we can see some stuff about how this screen was launched. So you can see all the intent parameters which were passing through activity, which in this case is just some stuff showing you about how this screen was launched. Actually, this is maybe not a very interesting screen, but this is the root activity of our application, which is an activity which has no UI. So one thing you might not know is that if you have an activity with no UI, you still have the decore view, which is what you're seeing right here, which is showing you the action bar or sorry, notification bar, as well as the row there at the bottom, showing you the, where these software menu keys will be. Right, as you can see as well, we have several choices. And these are actually all the activities which were in memory at the time we captured the memory dump. So it's not only gonna be what the user is currently seeing, but you basically have all the activities which are on the activity back stack. So you can use this to piece together an idea of where the user was coming from, where he went from that screen and basically give you his path through the application. I'm not gonna go into details, there's quite a bit of functionality here, but you can also, for example, find out more about, in this case, we're looking at all the network connections which the user has made at a certain point in time, including the state of the sockets and so on, like where the sockets went and the state of them and whatnot. And one thing I should mention is that at the moment, this tool is simply a complement to using Eclipse Memorandlizer, but what we are working on is to create a fully-fledged tool for working with memory dumps which hopefully can actually cover the entire functionality of Eclipse Memorandlizer, including doing advanced queries to find certain references and so on and also to find memory leaks, which is still a very, very important use case working with memory dumps. All this extra stuff is more of a bonus, just to show you what's really possible. Memory leaks is still very important. Again, right. Okay, actually let's get back to the presentation if I can manage. Oh, right, we're back at the beginning. Okay, let me quickly skip back to where we were. Right, demo. Okay, so there is a small catch, though. So far I've showed you what a memory dump can do. However, I need to mention also there's a lot of things a memory dump cannot do because memory dumps are great or age-proof memory dumps are great except a lot of cases where they are terrible. And what I mean with this? Well, there are a couple of really bad things about memory dumps and one thing is probably gonna be obvious to you if you ever capture one and that is that memory dumps are really big files. And what I mean with this? Well, usually a memory dump might be if you're lucky you're dealing with files that are several tens of megabytes, but in a lot of cases on high-end devices you have files that are several hundred megabytes big. This might not be a problem in some cases. For example, if you have the device which you are interested in and you have it in your hand, you're able to plug it into your computer, then 200 megabytes is basically nothing. You can transfer it in a couple of seconds. Imagine, though, if the device in question is in your user's hands and they're out in the wild, maybe with a 3D connection if you're lucky, maybe with, well, if you're really unlucky they have an edge connection in which case you're screwed anyway, but let's imagine they have a 3D connection. Let's see, what happens if we ask our user to upload a 200 megabyte file? We, well, actually we looked at it, but it turns out that in most cases they're not gonna be very happy about it. In a lot of cases, 200 megabytes is a very sizable chunk of a regular user's data budget in some cases it's even much more than that. So, this means that these HPov memory dams are not portable. You have them on the device and they're gonna stay there as long as you don't have physical access to it. There is another major issue with them though which isn't technically in nature and that is that they contain a ton of very, very sensitive data which might be very private to your users. And I know as developers we always consider the privacy and integrity of our users is a very important thing. For example, if you wanna transfer some data of the users over the internet, we will always make sure that we use the best encryption available to make sure that there is no way for this data to leak. And of course the same must be applied for memory dams. We cannot just take these memory dams and send them around if they contain sensitive data. What kind of sensitive data? Well, in the case of bitmaps it's quite obvious. In Padoo for example, we allow our users to send private photos in the chats between users. And of course these images can be quite private and the users would be extremely upset if they were ever to leak. Strings though, they are not maybe as obvious but a very, very critical example of sensitive strings which probably applies to most applications is that if you're using any kind of library dealing with some kind of online service such as Facebook SDK, Twitter SDK, Instagram and whatnot, you're using strings for authentication. So what all these libraries have in common or all these SDKs have in common is that they use OAuth for authenticating the users. And in OAuth the end result of actually having the user sign in would be an access token which is a string. And as long as you're using this access token in your application, the string is gonna be in memory as well which means that it will be in whatever memory that you capture because strings are very hard to just unload from memory. Java is very, very clever about managing strings actually. So right, so this is the main problems with age proof. And when I got to this point where I saw this problem, I gotta be down because this meant that even though I've done some tools, it's not gonna be very practical. I can still only look at memory dumps I capture myself which kind of ruins the fun of being able to look at like view hierarchies because I already know what's on the screen anyway. So we, we set out to create something better. And what we wanted to do was to be able to create a memory dump format which actually is small and portable. But we still wanna make sure that we actually have all the data needed to analyze the files of course. So it needs to be small but it also needs to contain a lot of data which is quite a tricky trade off. And the other part is that it needs to deal with sensitive data in a good way. We don't want it to contain any sensitive data in a form which is not protected in some way. As a bonus, it should be easy to read and write because we are still stuck with age proof probably forever unless and well, unless Google comes up with something better. It means that whatever format you create, it needs to be converted on the user's device from age proof to whatever format you come up with. And if you're dealing with 200 megabyte file, just reading it on your average Android device is gonna be fairly intensive. So you need to make sure that you don't waste users resources, battery and whatnot dealing with a complicated file format. Right, so that was the starting point for this project. And what we come up with, and this is a tool we released earlier this year, is something called the BMD. So they say that there are only two challenging problems in computer science. One of them is cache invalidation, which is tricky. And the second one is naming things. Personally, I think naming things is probably the heart of those two, which is why I came up with this very unimaginative name, but do memory-dump format. I guess, well, it's but do. Right, so what is BMD? So BMD is a memory-dump format which uses some nice, clever encoding tricks to take a whole lot of data and store it using as little space as possible. However, there's only so much thing you can do with clever encoding techniques. If you have a 200 megabyte file, maybe you can get it down to half. 100 megabytes, though, is still way too much. So the other key to actually reducing the file size is to throw away data we don't need. And it turns out that in an H4 file, there's a ton of data which we really do not need. And the main reason for this is that, like I said, this file format was created for desktop and enterprise Java originally, which means that the files contain a lot of data which makes no sense on Android. For example, protection domains and code signing. These are all fine and dandy on desktop or enterprise where they might serve some purpose, but on Android, this is part of the legacy Java stack. There's stuff there which is there because Java demands it, but it's either not implemented or not used on Android. So how about we just throw away all the junk data? It turns out that if you combine that with clever encoding, you can actually get some really small memory dump files, but while still containing all the data you need. And when I say the data you need, of course, in this case, is the data I need. But the beauty with this, and since this is all in source, which is awesome, it means that it's also very customizable and for someone who might actually have different needs from me, it's quite easy for them to modify this to target only the data that they need, which might be slightly different from what I need. Also, it's an format which is very efficient to read and write. And the main key to this is making sure that you have a format which you can, for example, you can read or write an entire file in one pass without having to backtrack, without having to keep a lot of data in memory up to the user's resources, because the worst thing you could possibly do while dealing with this is to run out of memory while processing out of memory, memory dump, that would be very ridiculous, and possibly end up in an endless cycle, which is a bummer. Right, so without further ado, let me show you some numbers, because numbers don't lie, as we all know. There's a bunch of numbers up there, but I would like to draw your attention to the one circling red, 1.8%. So what's this? Well, this is the ratio of the output file to the input file. And in this case, we had a 210 megabyte H-profile input, and we ended up with a 3.8 megabyte VMD files output. However, this is not the end of it. So what we have done while using this is that we would apply a second pass of compression to it. So when you are actually processing it, you would then just wrap it in a deflate output stream which is basically a zip compression, and you would end up with an output file of maybe one or two megabytes. And it turns out that one to two megabytes is really a very crucial limit, because somewhere around one to two megabytes is where the pain threshold for most users are when it comes to uploading files. We know that from personal experience, I would do that users are often willing to upload that kind of data because that's the size of, for example, the photos that the user can upload in their personal profiles. But we know that if you go beyond that, it gets much trickier or almost impossible to get someone to actually upload that data. So with this kind of size reduction, it opens up a whole lot of doors to us when it comes to when it's not possible with memory dumps. And it means that we can actually now do all those cool things I talked about, which is that you can take a memory dump from the user's device, which you might have captured not just when they're running out of memory, but rather when a certain crashes occurred. I wouldn't do this for every crash, of course, where I would personally target it is for those very, very pesky crashes where you've already tried fixing it using your usual tools, your stack traces and whatnot, then you have failed and you need more data. And in this case, that is when you wanna target it. So what we do in that case is that you collect the memory dump when the crash occurs and the next time the user relaunches that vacation in the background, we would run this conversion from HPROV2 BMD. And this is still a fairly slow process. And unless you wanna hog up all the system resources, you wanna do this slowly so that the user doesn't even notice. And in that case, it might take five minutes. Five minutes seems like a very, very long time, but as long as you're doing it in the background and you're not bothering the users, it's not a big deal. I can tell you that if you actually do it in the foreground, you could do it in maybe one minute, but the user wouldn't be very happy if he could not use that vacation for a whole minute, which is an eternity in app time. So right, so you can do this in the background and then ask the user, something happened. Do you wanna help us make the app better? Blah, blah, blah, blah. Please upload a file, type a button. It goes from the user's device, poof, up to the cloud, where you can conveniently analyze it at a time you want. And all of this, capturing the dumps, doing the conversion, the integration application, it's also part of the library we have released. So you can quite easily integrate this in application. The only part you need to implement is actually the part where you transfer the file because that part is very application specific what you wanna do with the converted dump itself. So for 40 minutes, I've been kept on talking and I haven't shown you any code, which I know early in the morning is enough to put anyone to sleep. So I'm gonna show four lines of code and that's it, that's all you get in this session. Sorry, if I had two hours, I would show you a lot more code, but in 45 minutes, this is the best I can do. Right, so this is how you can actually use this, this library for capturing and converting memory dumps. And it's not the most interesting piece of code and it's basically exactly what you're probably already doing if you're using any of the crash analytics libraries such as HockeyApp, Crashlytics and whatnot. We have simply this call in your own create in your activity, you can put it in the application on create as well. And what it does is that it registers an uncoded exception handler, which would then capture the memory dump. And there's also a service part to this which simply scans the file system for memory dump the next time the application starts to convert them. And that's actually it, that's the whole thing. Doesn't have to be more complicated in that. And actually, I'm gonna skip this, add a short demo actually showing this, but I already talked about it, so I'm not gonna make you watch it. Instead, I believe I actually have maybe couple of minutes for if there are any questions. Whoa, I have 10 minutes. That's much more than I expected, which is good because we can be on schedule. So either you can ask me questions or you can look at this adorable sea otter licking a pane of grass. So let's see, got some hands. Let's see if we can get a mic right now. Otherwise you can scream. So how do I solve out of memory exceptions using memory dumps? Right, so if that is what you were interested in coming into this session, I'm sorry to disappoint you by talking about everything except actually running out of memory. Hard to solve that. Well, that would, at this point, because this is actually one of the future goals of the tools we are developing to automate this process of finding actually causes of out of memory. At the moment though, using Eclipse Memoranalyzer would still be your best bet for actually finding this kind of issue. I can go into more details, but it would basically be another session about just finding out of memory errors. So I'm not sure I can give you a short answer to that question. It's very specific to what you're doing in your application and so on. So, but usually it involves bitmaps. Bitmaps are the bane of Android developers which will make your life miserable. But the best thing you can do is not, I would say use, I'm gonna do a sales pitch for Facebook and say use Fresco and then you don't have to worry about running out of memory ever again. Not exactly true, but it will help you. Hello, let's say I build a library. In that case, do I need any dependency on the app? If I want to include this inside my library. Right, that is actually a good question. I haven't tried it, but yes, it should be fine to bundle it together with the, are you doing it like as an AR library or how, or is it like a standalone library? Which is a- A gradle one. A gradle library. Well, yes, you could include it in your library using gradle. Right now, this is actually a standalone library project containing a JAR file. We're hoping to soon release it on Maven Central so you can much easier integrate it into your library. Okay, so app doesn't have to do anything with that. My library will take care of it. Yes, basically. I mean, for this, the conversion part though, there is still some need for actually adding a UR component to it because that is not included in the base library because it's quite specific to how you actually want to implement the UI part, the part where you might request the user's permission to upload a file though. Thank you. So this S-Proof part, how much configurable it is, like if I want to have a S-Proof with BMD. So can I get it before having a memory leak because sometime it may happen that your app is on the edge of memory leaks. Yeah. It's lagging, but it doesn't have a memory leak or it's just an edge of going to out of memory but it's exactly not out of memory. So is it configurable that can I have this S-Proof before going to out of memory problem? Okay, good question. It's not actually part of the library at the moment because capturing the other memory itself is much easier. We have that method before going to out of memory. Yes, you do have some callbacks. Yeah, we have a callback, but can we init this library at that time? Yes, well, basically what you could do in that case is that you might not actually... Because the code you saw there with the init, that is simply for registering the exception handler. But what you might do in that case is you can use the method which is available in the Android SDKs. I think it's in the debug class. You can do debug.save memory dump. And actually what this does though, which is a problematic part is that when you call this method it will freeze the application for as long time as it takes to save the memory dump which means that... Can we move it to a background process and... We can init that in a background. Yeah, actually that's a good point. Well, no, sorry. The problem is that it actually needs to freeze the entire VM because you can only do this... The problem would be if it's caching the memory dump and you allocate more objects, it would put it in an inconsistent state. So it actually needs to freeze everything for one to two seconds. But if I'm going out of memory, I think at that time it's actually freeze itself. Ah, yeah. So basically you're in a situation which is quite bad where the user is not getting a good experience. So you're right. It's already gonna be freezing so it wouldn't really make much difference. Right. So should I put it in a background side or in an intent service kind of thing? I would say that no, you can still not actually capture the memory dump on a background thread as far as I'm aware of. Sorry, I could be wrong about this. So in that case I would still do it on the main thread and yeah, the user will notice it that it stops for a couple of seconds but it's still gonna be the best bet there. So there is no way that I can attach this thing with a stack trace plus stack trace plus a true combination of both. I can't get anywhere. Well, you, so the way I've been using is in conjunction with other tools like, for example, Hockey app where I still get all the stack traces from Hockey app and then I get the memory dump as a complement to that. So I could as well include it in the hfor file but it's not there at the moment. If we try to build a tool over it that can give us stack trace plus a true, so will it be a big fight or it can be a easy good to go thing? Sorry, I didn't cast your question. If I want to build a tool over it that can give me a stack trace plus a true, will it be a big fight or is it an easy go? That would be quite easy. I would say to capture all the stack traces, yes. As long as you can get hold of the threads and then create your stacks. So yeah, that should be quite possible. Thank you. So we've run out of time. You would have to take the Q and A outside to Andre. We'll just get started with the next talk. It's Sohum. In between I'm gonna make some quick announcements. So yeah, first up, thank you so much to InShorts for being a Platinum sponsor. They sponsored the Aura VR headsets which you will be getting with your goodies bag.