 Hold on. I got to get a couple photos. This isn't part of my presentation time. I just need some photos for everybody. Okay. No, I'm not doing it. Not yet. Not yet. No Friday hugs right now. This is just, this is Aaron's time right now. This is Aaron's time. You're going to have to wait for Friday hugs. There's a special portion of my presentation for Friday hugs. So let's wait a second. I'm choosing to stay behind the podium because I'm not wearing any pants. I'm just kidding. I am wearing pants. It's true. You can see. Okay. So this is the title of my talk. Mark Compact, GCN MRI. My name is Aaron. Hello. I am a 20-something hipster nerd who is now a 30-something hipster nerd. My name is Tender Love. If you don't recognize me in person, this is what I look like on the internet. Yesterday I was very jet lagged. I'm still a little bit jet lagged. But yesterday just, this is how I felt yesterday was, it's Friday, Jr. Yay, Thursday. But today, today, today is Friday. So let's weekend. Now is the time for Friday hugs. I want everybody to stand up and I want you to give the internet a hug with me. This is a tradition that I do and I'll explain why in a minute. Okay. All right, everybody. One, two, three. Happy Friday! Happy Friday! Okay, thank you. So I do this, I do this every Friday. I hug the internet. The reason I do it is because I work from home and I get kind of lonely. So the internet is my water cooler. So that's how I say hello, everyone. I was actually in, I was in Japan a couple weeks ago and I decided to go to some retirement homes and do Friday hugs there and this is a photo of one of them. We're doing Friday hugs there. I'm just kidding. That's not true. This is a real photo but I was out there. I'm on the Ruby core team and the Rails core team and I have a quiz for you today. Is this Ruby or Rails? Thank you. All right. I work for a company called Github. I don't know if you've heard of it. This is a small company I work for. It is, I have to make this a joke at every conference. It is the only legit company I've ever worked for. So this is not, thank you, thank you. This is not to be confused with Matt's presentation where he was showing off legit. But I thought it was kind of cool because he posted the link to it and it was on Github so it was legit on Git. But I actually was googling around and I forked the project so there's actually two of them so now it's too legit and those are too legit on Git. Yes! Those of you who grew up in the 80s know what I'm talking about. All right. So anyway, I love using Git but I will not force push it on you. So I work for Github as a Ruby and Rails developer and what I mean by that is I actually work on Ruby and Rails for Github. Fortunately, Github allows me to do open source development as my full-time job though I've recently become Mr. Manager but I still do open source development and Mr. Managing as well. So anyway, I get to work on open source code all day and I just want to say Github loves open source and I'm extremely grateful that they let me do this. I have two cats. This one is C-Tac Airport, Facebook, YouTube, Instagram. I didn't include the Instagram part and she's just choo choo for short. This is the more famous one his name is Gorbachev Puff Puff Thunder Horse and they or Gorby Puff for short. They love each other very much actually that's not true he hates her and she loves him and they just so she always snuggles with him and he gets annoyed and leaves but they're totally experts in getting in my way. For example, I like to sleep on my keyboard like this and this is like I'm constantly cleaning hair out of my keyboard because of these cats. I'm a mechanical keyboard enthusiast. I love mechanical keyboards and I'm going to show you this is my list of my mechanical keyboards. I own five Ergodox keyboards. Three of them are Infinity Ergodoxes. One of them is an Ergodox easy and the other one is the original. I own two planks and one atreus and this is very sad. This is thousands of dollars of keyboards right here. I shouldn't spend so much money on these things but I do. This is my daily driver one. This is the Infinity Ergodox. That's the one I use daily. All the other Ergodoxes look very similar to this one but this is my daily use one. This is my travel keyboard. This is what a plank looks like. I have this one here with me today so if you want to take a look at that afterwards, please come say hello to me and I will show it to you and you can type on it. It's very loud and I want you to enjoy how loud it is. This is the Atreus keyboard. This is an interesting keyboard as well. I like this keyboard a lot but I don't travel with it because this one has more keys into a smaller space and it's easier to pack although I do love this keyboard. I figured I was putting in these slides about my keyboards. I figured I may as well show off all the other crap that's on my desk. I also use an L-track mouse. This is a roller mouse and that is my cat lying there in the background. The reason I got this mouse is because I was having RSI problems but also the cats kept sleeping on my mouse pad so this is the situation I was living with. I could barely use the mouse pad and it would get really bad like that. I don't want to move them out of the way because I feel bad but actually I reviewed this mouse pad. If you go on Amazon I reviewed this mouse pad and there's thousands and thousands of comments on my review for this mouse pad because I think the title, the mouse pad was like, I really enjoy the 25-50% of this mouse pad that I can use at any time. It is a popular review on Amazon so that's actually how people know me is this review. I use a Norman layout. This is different than Dvorak or Quirty. It's actually based off of Quirty where they moved as few keys as possible but to improve typing. It took me about three weeks to learn that so I have a completely custom setup and what that means is that I'm totally useless on a laptop basically. Now the reason I did all this stuff is because at one point I had really bad RSI issues like I couldn't type at all. My hands were just hurting all the time and I realized the reason I was getting these pains in my hand is because I wasn't thinking about my work environment at all. I just would type on whatever and I don't care how I sat or it didn't matter even though I was spending however many hours a day at a desk. I never really thought about that much at all and once I started thinking about it and actually getting into this stuff that's the reason I spent so much money on these keyboards and things was to make sure I didn't get that pain again. What I want to pass on to you today is to think about your work environment and to invest in yourself. Don said this earlier. Yes, call back to you. So make sure to be thinking about your work environment so you don't get injured like I did. You don't want to get injured and then do this stuff. You should do it in advance please or so please invest in yourself or treat yourself as I like to say. I also love puns but sequel puns make me very upset. So no sequel puns please. Take a photo. So I want to talk a little bit about error messages. There was an awesome talk yesterday about error messages and I enjoyed that presentation and I want to tell you about my relationship with error messages and that's that I have an incredibly short attention span. I like to cure meat and this is some of the sausage that I make. I made some sausage at home and I promise this relates to error messages. So I like to make sausages. This is some pancetta that I made. I made this. I also enjoy making bacon though I couldn't find any photos of it for some reason. She's so cute. So I love to cure meats and stuff and I want to make bacon and when you go buy it at the store it was about 4.99 a pound and okay that's for expensive bacon and I apologize for the imperial units but it doesn't really matter because I've used the same units throughout so you can figure out the price. It's 4.99 per some unit. Now when I went to make my own bacon I had to buy raw pork belly to cure it myself and unfortunately when I went to buy that that ended up being $5.25 per pound and that really bothered me because the raw pork belly was more expensive than the cured pork belly and presumably they had done more processing on that so it bugged me that I wanted to make my own at home and I wasn't saving any money doing it. So I went around to several stores and kept buying these raw pork bellies and they kept coming in the same box and no matter what butcher I went to they would put it in the same box and I'm like hmm this is interesting so I figured out where the box came from and I found out that it was actually a wholesaler and I found out if I went and bought these pork bellies from the wholesaler they were about $2.25 a pound. So that was a great discount. Unfortunately if you wanted to buy pork bellies from a wholesaler you actually had to have a business license so you needed to incorporate a business and that's why I started my consulting company adequate industry is everything adequately. So I started this consulting business so that I could go buy these pork bellies. Now when I filled out the online forums it cost about I don't know $40 to start a business and I was filling out the online forums and like they kept asking me these questions. This is where the short attention span comes in. They kept asking me these questions and I didn't know how to answer them so I would just like I'd fill out the forum and it would give me some errors and my eyes would glaze over because I didn't understand the errors so I would just fill out keep filling out stuff like putting in stuff and checking stuff until it finally let me through and I'm not quite sure what I put into the forum but it let me through and the government took my money so I figured I must have done everything right they wouldn't take my money unless I did it right right so anyway fast forward like I had the business for maybe three years or so and I get a call one day from an auditor from the city and she says to me your business isn't set up correctly and I said oh really and so she explained to me what was wrong with the business and I had no idea what she was talking about and I'm like I have no idea what you're talking about and she said well yes your business is set up completely wrong and I said well yes but you took my money so why did you take my money if it's all set up wrong and she's like well I don't know but you're gonna have to pay some fines and I'm like oh jeez and so she says well how much money have you made with the business and I'm like I don't make any money with the business I omitted the part about pork bellies I don't make any money with the business so she said okay well then you don't have to pay any fines but we want to close down your business we want to close it and I said okay that's fine I don't want to pay anything you can go ahead and close it down so she says okay great close it down and then about two weeks later I get a letter from the city asking me whether or not I wanted to renew my business so I replied yes and so I may or may not still have a business not totally sure anyway so that is my saga of error messages so anyway let's move on a bit I want to say thank you to the organizers for having me thank you all of you for coming here today I love coming to Singapore this is my fourth or fifth time I think at this conference I love being here the talks are always amazing I saw some of the best talks this year I've seen so far I learned so much when I'm here and I enjoy talking to the people because there's so many talented engineers here I always leave feeling inspired so thank you all very much uh oh are we okay are we okay I'm just kidding I'm kidding that was me alright so we're going to talk about Mark Compact GC for MRI this is an ongoing development that we're doing at GitHub last year I spoke about garbage collection as well and at the end of my talk I proposed a thing called an idea called heap splitting and we were working on that at GitHub and we ended up abandoning the project so this is unfortunately I am a failure of a software engineer so I'm now a senior software engineer telling you that I am not that great at programming but anyway we abandoned that project for a different project called Compaction and that's what I want to talk to you about today I actually introduced Compaction a tiny bit last year and said that it was something that was impossible to do with MRI but it turns out I was wrong so I am frequently wrong apparently as I was about heap splitting so we're going to talk about Compaction this year now Ruby already has a marking Compacting GC and so we're not really going to focus too much on the mark side of this we're going to talk about the Compaction side but in order to talk about a compacting garbage collector we actually have to have two things already we need to have objects and we need to have free space we have to have something to compact and we have to have some place to compact them into so the roadmap is going to be we're going to look at how allocations work then collections and we'll focus on Compaction and move into the actual implementation details of Compaction and I took a page from Juanito's talk yesterday and reserved this slide for local jokes so if you have any local jokes please tell them to me later so first let's talk about allocation now this is an example of some code that allocates a whole bunch of objects obviously but when we execute this code where do those objects go where do they actually get put and what actually allocates them what I think is interesting is that allocation is actually the garbage collector's responsibility and I thought that was always weird when you hear the name garbage collector it's actually getting rid of any garbage but actually creating garbage is also the garbage collector's job as well so it's not just a garbage collector it is also a garbage producer which is interesting to me and if you look through Ruby's source code the source of all objects comes from a function called rbnewob so if you want to dive into the code go look for this go look for this function you can read through it and learn how objects are allocated this function actually searches the memory space for a new place to put the object and returns that object back to you so the way that Ruby actually divides the computer's memory is we have a heap which is some amount of memory Ruby actually divides that heap into pages so we have many pages then it divides those pages further into objects so each of the pages have objects and whenever we allocate an object we look for a space in one of those pages and if there is space then we put an object there and if there is no space then we'll allocate a new page and then we'll put objects in that new page when we have to allocate a new page that's called a slow path allocation so we say give me an object I'm gonna if there's no room then the process will actually go allocate a page and then put that object there this is called a slow path allocation because we have to go through all the work of creating a new page if there is free space that's called a fast path allocation it's because all we have to do is just return that object back we have space for it so when you're running your application in production you want to reduce the number of slow path allocations that you're doing in production and increase the number of fast path allocations so in order to do that you can tweak some of the garbage collection knobs and the one to tweak is this one called RBGC heap free slots and this basically says the number of slots that we're going to have available so we tune ours up fairly high so that when a request comes through we don't have to do any slow path allocations all of them are fast paths so we keep a large heap these pages are also called arenas if you hear that term it's essentially the same thing just an amount of memory that we allocate in some place so if we look further into a page layout I said pages are where we store new objects but what actually happens is when we allocate a page that page gets pre-filled with a bunch of free slots so we have a bunch of these free areas in a page and when you allocate objects the objects get put into those free slots as we're running our program so it'll go through there and allocate put them into those free slots now when objects get freed they actually go away from those slots and they just get turned into free slots again the objects go away when they're GC'd they get turned into free slots and then reuse those slots later now slots are fixed with which so each object is only 40 bytes wide but that's kind of interesting when you think about it like if a Ruby object is only 40 bytes how do we store things like hashes or rays we know hashes grow larger than 40 bytes so how does that actually work the way this works is if we have memory like this our computer's memory and we have some page in the memory when we allocate objects in there let's say we have some kind of hash object in there what we'll do is we'll allocate what's called an ST hash structure out on a heap it's not actually allocated inside of that page it's allocated somewhere else in the computer's memory and the way that we actually allocate those ST hashes is with the mallet call now when a GC occurs and we want to free that hash it's the hash's responsibility to say okay before I get freed I'm going to call the free function on the ST hash now that ST hash will get freed up and then the garbage collector will free the Ruby hash object itself so that's how we're able to store objects larger than 40 bytes in the computer's memory and this will be important a little bit later so this memory up at the top here that's managed by the system allocator where that slice at the bottom is managed by the garbage collector now interesting facts about these two areas are that we can control the layout and the format of objects that are allocated inside of the GC so we can move those things around and manipulate them but the ones that are outside the system allocations we can't necessarily control where those are with the system allocator we can do some little tricks but we don't have nearly as much control over them as we do with the objects allocated in the GC so anything allocated in the GC is much better for us than if it's a system allocated so in summary we're going to summarize this in Ruby code a heap has many pages a page has many slots a slot can either be empty or have an object those are the conditions of our memory so when we look at collection we're going to talk about collection we talked about this a little bit in the last year but we're going to do a little refresher because it's important to understand how collection works in order to see the tricks we can do with collection so Ruby uses a mark and sweep garbage collector and the algorithm roughly goes something like this if we have Ruby objects those Ruby objects form a tree in memory this is kind of what the tree looks like now if we cut one of those cut one of those slice those arrows from the tree we actually have to free up all of those things and that's where the garbage collector comes in we cut that relationship it freees those objects the actual algorithm has two different parts the mark part and the sweep part which is why it's called a mark and sweep garbage collector so the way this works is if we have some sort of tree we go through what is called a mark phase and we walk through the tree starting from the root so in this particular example we'll go from the root to A to F and then from F to D, B and C and then from the D we go back to A and anything at the end of this mark phase that hasn't been marked we free it so that's what happens those all get freed and that's the overview of mark and sweep mark and sweep implementation now the thing that actually marks those edges is a function called GC mark and that's going to be important later where is that GC mark right there so that function is what marks those arrows so it's important this function name is important and we'll look at it later so if we're going to look at this in Ruby code we would say like hey object give me your references I'm going to walk through each of the references and call GC mark on this now the downside of mark and sweep is that it's kind of a slow algorithm but there are different tricks that we can do to increase the speed of the algorithm such as incremental marking and lazy sweeping and we actually discussed those in depth last year so if you want to know more about that watch the talk now on to compacting we looked at how objects are created and destroyed but how does this actually relate to compacting now let's say we have a bunch of objects like this and we free up some of those objects like that the amount of memory that's actually consumed is still the size of all those pages even though we freed up all those objects we didn't actually decrease the size of the memory right now let's say we're freeing more objects and all of a sudden that top page becomes free in this particular case what will happen is when the page becomes completely free Ruby will actually free that page up so it goes away and now we're able to reduce the amount of memory that the process uses so now what if we have a situation like this where we freed up many objects and the sum of those objects equals the size of one page now if we could move those objects around it would be nice to move them around like this and then we could actually free up that page like that so unfortunately today objects can't move so if we can't do this in rubies that you have installed on your production machines today and this is where compaction comes in and this is the work that we've been doing at github is working on a compacting garbage collector and you can actually find all of these changes here so this is the logici on legit sorry so if you want to find all that work it's up here on github so I'm going to talk about the actual compacting side of this now the compaction algorithm that we're going to talk about is called two finger compaction there's actually many different compaction algorithms but this is the one that we implemented it's called two finger compaction yes I was playing with keynote and it was fun so the downside of this particular algorithm the first disadvantage this algorithm has is that it's not very efficient and you'll see the algorithm in action a little bit later and the second problem with this algorithm is that it actually places objects in a random location in memory you'll see why in a minute here but the the major advantage behind this algorithm is that it's actually very simple to understand and simple to implement and I am very lazy so those things appeal to me greatly so the algorithm has two steps which is moving objects and updating references so it first it moves the objects and then it actually updates all the references for the objects and we're going to look at the algorithm here so the algorithm starts this is a page that we have with some free slots in it and some objects on the slots and up at the top there those are the actual addresses of the objects so it starts out by putting two pointers on either end of the page and the left pointer is called the free pointer the right pointer is called the scan pointer and what happens is the left pointer moves to the right until it finds a free slot and the right pointer moves to the left until it finds an object and when those two things happen it swaps those two so in this case we stop right here and we just swap those two and then it actually writes out a forwarding address so the object that used to be in B now is in one so it leaves the one address there and we repeat this so this the left scans until it finds free the right scans until it finds an object it swaps and then leaves a forwarding address so it keeps repeating this process until those two pointers meet so it'll repeat and then finally when they meet we've moved all the objects next to each other so the actual code for implementing this looks like this and it's really very simple you can see up there we take a scan pointer and a free pointer those are the things that are moving through the heap and then we basically just swap the two we do a mem copy of one to the other and then we're done so unfortunately if we have let's say some Ruby code that looks like this the objects are related to each other they have a relationship with each other so indicated by those arrows there we have maybe some hash it points at a symbol and a string now unfortunately after we compact the heap it'll end up looking like this and now that hash points at two locations that no longer contain the objects that we're looking for so because of that we have to make another pass through the heap and actually update those references so it points at the right thing and this is a very easy algorithm all we do is start at one end and we just move along looking at each object to see what references that object has if it has references we look at the follow those references look for the forwarding addresses then we simply change the references to point at the forwarding addresses so in this particular case object number four used to point at six and seven now it needs to point at five and three so all we do is update those arrows to point at five and three and then continue on through the heap updating references like that so once we reach the end of the heap we're done and we've updated all the references everything is working yay then the final step is to go through and change all these forwarding addresses back to free slots and now our heap is back to a normal Ruby heap and all the objects are moved next to each other so I wanted to show a little bit of the updating update reference code and I'm just going to show it for just for arrays but basically all we do is we have this C code that basically iterates through every single one of these slots in the slots in your heap and then calls an update references function down there at the bottom and updates the references for those objects now if we go look at that that update object references is actually in a very large function and what it is is it's looking at the type of the object and depending on the type it updates the references so for example one of the things is an array this is a case statement inside of that function where we handle arrays and we say okay well this is a shared array we update that pointer if it's not a shared array then we go through and update all the references inside of the array so that's all we have to do we're done it is that easy yay so there's actually the cost of this algorithm is that we have to visit each slot three times we have to visit every object once in order to move all the objects we have to visit every object again in order to update the references and we have to visit every object again in order to remove those free slots in the future we can probably remove that last step and just treat these forwarding pointers as free slots but you can see we have to walk through every single object twice at least so compaction challenges I want to talk about some of the challenges we've had implementing this, implementing this compactor at work one all these problems boil down to essentially object addresses so every object in your system every Ruby object has an address an address in memory now unfortunately a lot of the code out there specifically the C code out there today believes that these addresses are constant that the address for that particular object will never change now unfortunately we will have some C code that points at this object and it says oh this address number four that will never change right it's fine now unfortunately with a compactor that is no longer true so the primary offender of this rule is specifically C code and we're going to look at those look at how to address these C code issues because we have to address this in order for it to actually be a production ready thing the three problem cases will look at our C extensions and how we dealt with that hash tables and also global variables in C so for C extensions the way we dealt with that is a little trick I used now so far the object relationships we've looked at have been hashes that point at strings and symbols and these are just normal Ruby objects right now these are regular references and we can handle these regular references pretty easily because they're implemented inside of MRI we know how to update those references we can fix them but what if we have some C code over here some C data structure that points at our Ruby object how do we how do we actually handle that so if we don't handle that then the garbage collector could actually free the hash and then our C code would explode right if you have C code it has to do something to make sure that that Ruby hash stays around and if that Ruby hash doesn't stay around then that C code must explode so how do we how does the C code actually handle this so the C code has to say well I want that hash to stay around because I'm referencing it and in order to do that in order to do that the C extension author actually has to call a function called RBGC mark so they call RBGC mark on that Ruby object in order to say hey I'm still holding a reference to this don't free this don't free this object now what's interesting is if you remember back to our mark sweep algorithm you'll note that the way we actually marked references was via a function called GC mark not RBGC mark so actually internal data structure normal regular Ruby objects use this function called RBGC mark so or called GC mark so in this way we can differentiate between objects that are held by C extensions versus objects that are held by just normal regular Ruby objects so in this particular case we can say okay well you know don't move that hash it's being held onto by a C extension but the string and the symbol those are okay to move okay we can move both of those because they're just being held by a Ruby object and we know how to fix those now I want to prime you for questions to ask me later what happens if you have a C extension that references a symbol like this but doesn't mark it what is that case what do we do then so please ask me that later so let's look at hash tables this is probably the least interesting problem but also most frustrating or maybe both of those I'm not sure it's an interesting slash fun and not fun problem and you'll see why here so a normal hash table a hash table all it does is it has some sort of a bucketing system where we take some we take some value and we compute a hash code for that value and based on that hash code we stick that data structure into some bucket right so we have some key value pair for example we have some hashing function that computes a hash for the key and say in this case it computed six and it stores the key and value there that way later on what we can do is take that same key compute the hash key for it again and get access to that value we're all familiar with this we know roughly how hashes work we use these day to day I think especially if you're a rack user so the default hashing function unfortunately is this you don't need to understand the C code I'm going to point out exactly to you what the problem is the default hashing function actually uses the memory address that uses the object address so if you take the hash value of some Ruby object it's going to be the object address now the problem is that if that object moves well that address changes and if that address changes well then the hash code changes for that object and if that hash code changes then we can no longer look up the object in the hash this is an issue for hashes yes I said this if the address changes then so does the hash so how do we fix this clearly if you use some key in the hash we want to be able to look up that value again so what do we do well the solution that we have taken unfortunately is to don't allow hash keys to move so we just say these certain groups of objects they cannot move okay now I put a little star by that because I want to feed you another question which is what about strings so ask me this later in the Q&A section or if we don't have time ask me over a beer so the final thing I want to talk about is global variables and I'm going to show you an example of global variables that we had to deal with in MRI and I sent a patch to fix this so in this case we have a global variable here called separator and this separator actually points at a string called slash it's just slash you'll know the slash from when you do file.join okay now that global variable is used again down here when you say file.join it says okay take those two things join them together with that string and we're great now what happens if we assign the separator to nil and then GC so let's take a look at this code we say assign file colon separator to nil and the other one to nil then we do a GC and then we start you'll actually get a SEGV and that is because I'll show you in a minute this SEGV does not occur with JRuby so I shouldn't plug JRuby since I'm on the MRI team but there you go so the reason this SEGV happens is because we have this string in memory and we actually have three references to it we have the global variable in C pointing at that object and then we have these two Ruby constants pointing at the object now this code here that's in the lower right I'm assigning those global objects to those constants to nil and then we do a GC start so we assign those to nil we do a GC start the GC says well there's no references to that slash object anymore so I'm gonna free it so it frees it because the global variable does not know how to mark it so it frees it and then when the final file.join runs everything blows up because we have no reference to that object anymore so in this particular case what we did or what I did was submit a patch to stop using this global variable so look up the value rather than use some global variable as a cache so this issue is similar to compaction because we're not able to update the references of those global variables which is what the compactor needs to do so global variables the hash issue and the global variable issue are very similar because when those objects move we can't update those references so the general problem is essentially that we can only update Ruby objects to point at other Ruby objects and we can update those we can only update core classes to MRI so I want to I showed a few examples of the issues there are many issues but we've fixed them one by one piecemeal and it seems to be working most of the time so I want to show you a result of the compaction we're actually using we use the compactor in production and I want to share some of the results with you that we found this is an example of a Rails heap I'm just going to show a bare Rails project so the compaction code looks like this essentially we open a file we write out the heap to the file we have to specifically compact the heap by calling gc.compact then we write it out again so we run this with the Rails script runner now if we graph all the graph what the slots look like if we graph the heap before we run compaction the heap looks like this now each of these columns are a page and we talked about pages earlier each of the red dots are objects that we cannot move each of the green dots are objects that we can move and each of the white location that is free space and these pages are sorted by the number of objects that cannot be moved now after we compact if we look at this graph again it looks something like this or you can see now we've got all these green objects moved to one side of the heap and we have all the white space on the other side of the heap and we can allocate into there so before compaction we had 552 pages and of those pages 528 of those pages have space on them for allocation now the problem with this is is that every time we write to a page that page may be copied to a child process where we're using unicorn in production which is a forking web server so copy on write optimizations are important to us now after compaction we only 167 of our 552 pages have space for allocation so maybe only 167 of those pages can possibly be copied where previously 528 could now this lower number is better for copy on write optimizations because we would like to have fewer pages be copied to child processes now we can actually improve this we saw all those red dots at the bottom if we can fix those red objects such that there are fewer of them we can improve this number ideally we would have no red objects but you saw earlier we have those issues where there are some that we cannot move there are some that we've some that I didn't show in this presentation that we say oh we can't move them but with a bit of development work we can so I want to show you some of the heap graphs well a heap graph from our production application and the reason I didn't include many of these graphs is because they're very big we have a very large heap in production this is what it looks like after compaction so you can see it is quite wide now the good news is I mean the bad news is there's a lot of red objects on here the good news is that it follows the trend of a basic Rails application so we can say basic Rails app is generally representative of the heap so why compact I've talked about how to build compactor and I gave one reason the one reason is to free up space like this we want to free up if we can we want to free up those pages the other is for copy on write optimizations which I mentioned a little bit earlier we use a forking web server unicorn and a forking setup all the child processes share memory with the parent process but as soon as either of them writes those processes we get that memory copied and that increases the overall memory used on the servers so we want to reduce that so for example in this case before compaction basically all of these pages could be written to all of them have free space on it so potentially all of them could be copied to the child process whereas after compaction you can see here only those pages can be copied to the child process and these pages remain shared among the parent and child processes now I want to talk about reality here for a minute unfortunately let's get back to reality sorry we're going to talk a little bit about agile development so agile development tells us to fail fast now I've created a new development style a new better development style that I call fail continuously so I've actually created a new continuous integration server where we actually continuously fail and today I'm really proud to present a new product from adequate industries it's called Aaron CI where your tests will fail constantly so I guess I am coming around here making a tongue-in-cheek assertion that I am somewhat of a failure and it is true because if we look at the real world results of this compactor we saw in production maybe only a 2% improvement or a 2% reduction in memory usage on our production machines this is nowhere near the amount that we expected because if you look at these graphs here we'd expect okay well you know maybe in this case we have most of this shared and presumably only 30% of it can be copied but you'd expect us to have maybe I don't know 60% reduction in memory but that's not true or a 30% improvement it's not true and we're trying to figure out why like I said this is ongoing development and I want to share with you two possibilities or two reasons why we're not seeing the improvements that we expected there are two possibilities one is essentially poor compaction and the other one is maybe a small percentage of the heap is actually compact I'll explain that in a second I glossed over some stuff in this presentation now if we look at this before compaction graph I've divided up these pages into four parts and the reason I've divided these pages into four parts is because if you write to one page the entire page does not get copied only 25% of the page gets copied okay so the problem is that this free space the 25% of every single page that's open that particular free space that can possibly be copied may be equal to this free space here so if that's the case and we'll see no improvements in copy and write so if the number of quarter pages is equal to the number of full pages after compaction then we won't see any improvements now the other possibility is that maybe the ruby heap is actually much smaller than the amount of memory that we've allocated we talked a little bit earlier about hashes being allocated in one area and ruby's GC allocated in a different area maybe ruby's GC is just a small percentage of the memory that's being used so compaction will have very little effect on it maybe it's large maybe it's small so we have to measure that and I want to talk a little bit about measuring and then I'm going to hurry because I'm totally over time we're measuring this now with a thing called malloc stack logging and what this is is we say every time you make a call to malloc we log the stack so we see who made that call and figure out what the problem is I'm going to show you how to do this on OS 10 but you can also do this with Valgrind what's called the massive allocator so look up that later the way to do this with OS 10 is you simply set a variable called malloc stack logging no compact and what this does is it logs every single malloc call and now you put the process to sleep because you can only analyze live processes because of a thing called address randomization which don't just don't worry about it so you put the process to sleep and say ok in another terminal ask for the malloc history give me all by size for that particular pit I just put in some pit that's obviously not the pit you're going to have so what that does is it actually prints out all of the live allocated things so this malloc stack logging logs every allocation logs every free and the malloc history actually takes that history and calculates what is allocated now and what are the stacks for those particular things so a sample of the stack log is like this I know you can't read it one record looks something like this where we say ok we had one call for some amount of bytes and that's what the stack looked like so we can see who made that call now I wrote some code to process it and it's kind of small but I wanted to point out a few things about this processing code one is that I use the flip flop operator the other is that I use the protected method and the other is I have a method here with no parentheses on it so I am using all the worst practices from this conference so if we use this code to process the output of this malloc stack logging we can see an empty program will have something like this where only 17% of the heap is managed by the garbage collector where 83% of it is managed by the system and that's for an empty program this very basic empty program here now we can actually adjust this we can adjust this let's say we change the number of slots that we initialize with 100,000 ruby objects and get the stack for that this is what it will look like so we started with 17 with an empty program and we can actually increase the amount that's used by the garbage collector so we're testing this in production right now unfortunately logging this data is slow so we're trying to come up with a better way to do it the test results will guide us how to actually fix this and improve the performance of the compactor so depending on what the test results are it will tell us where we need to focus so if the test results show that most of the heap is taken up by the garbage collector then we need to fix all of those red dots and allow those to move if it turns out that the ruby GC is a small percentage of the heap then we need to allow variable width objects and start allocating hashes into ruby's garbage collector now the truth is that both of these things need to happen but the test results will tell us which one to focus on so that we can get the best results for this so I'm going to finish up here because I am way over time I will summarize why should we compact compactness allows us to save memory in two ways which is freeing up pages and also improving copy on write performance we've developed this patch as open source software which you can go check out here the plan is to upstream this sometime this year I'm hoping to get it done by September for ruby kaigi and that is all thank you very much I'm so happy I could be here questions there you go hi I'm going to get a new laptop so I was wondering when will we get a new edition of stickers with Gorbachev and Chuchu no new edition right now I brought stickers with me if I'm invited back next year I will definitely bring new stickers I brought stickers we've had the same stickers for a year we recently moved so my brand manager hasn't had time to design new ones so I'll have some next year awesome and now for a more serious one what will happen with object IDs since they depend on the memory where object was allocated sure that's a great question so the question is what will happen with object IDs? Object IDs are based on the location of the object in memory so when you call object ID that number is directly related to where that object is in memory and the way we're handling that is we actually lazily figure out what the object ID is and cache it on the object and then we have another lookup table for collisions so basically what will happen is your object has no object ID until you call object ID then it will calculate the object ID based on you the address or if that value has already been taken some other value so the moral of the story is please do not call object ID you can do it but it could be slow in the future yeah what would be copied there is this expectation that the stuff in the green the red and green that that would just be shared but can't those same can't the things in those pages also change over time and the copy on right could kick in for those pages? Sure yes that's a great question so the question is can't those the red and green objects that I showed earlier move around or get freed or have something new be written there over time and then have that ruin the copy on right optimizations and the answer is yes it's possible but what we did is all of those objects that I showed you are immediately after Rails boot so it's all just code that we've loaded in and initialization objects so those will likely not change those should be long lived objects but yes it's possible okay yeah I guess I just don't intuitively know like active record is in there just like messing everything up after you've loaded it and like you know messing around with singletons and what not I don't know how much it causes that stuff to... Sure no the code most of that most of the stuff that I showed there is just compiled Ruby code it'll never that stuff will never change if we were doing compaction later on like after each request or something like that yeah it would completely ruin it okay I figured the answer was something like that thank you alright okay hello hi thanks for that it was really good to see the internals of Ruby oh thank you two part question actually in the loop does it go through the entire heap or just a page ask that again you mentioned about the loop for compaction does it go through the whole page or the whole heap it goes through the entire heap okay so is it possible that a pointer after moving the object you leave a pointer there and that pointer itself could move later does that happen a pointer what do you mean so the way you're moving the object can a pointer also move is it even possible no I mean you can move so the object the object will get moved but after we've updated all of the references then none of the none of the Ruby objects should reference pointers that aren't actual Ruby objects okay thanks yeah up there hello everyone hello how does an anonymous class act in this compacting is it going to stick around forever or is it going to be moved like a normal object all objects are moved equally it doesn't and if I assign it to a constant like if I say I don't know a equals yep it doesn't it doesn't care okay great thanks for sharing just now you mentioned you got about 2% optimization right about the memory location yeah like I have no idea how the compassion script is written but one thing that I am quite confused is like you are trying to free the whole page right if there is no I'm quite curious because the location of the new object might be faster than your compassion script so I mean that you are unable to free the whole page sure yeah okay so the question is how does the how can the compactor keep up with allocation yes yes of new object yeah yeah that's a good question right now it basically stops the world you have to actually call an object so we modified our boot process to say okay before we actually listen for requests we compact everything and that pauses the process for maybe our particular heap it would pause the process for about 10 seconds and then continue on so it doesn't actually allow any allocations during the compaction phase alright cool okay there's one in the back there's a mic there you can go to the mic sorry I want to ask you mentioned about the Ruby manage memory as well as system manage memory I think it's a weird question why do we need to separate these two types of memory management why can't Ruby manage their own memory internally and you also mentioned about the 40kab size sure why can't it be dynamic sure that's a good question so the question is why can't Ruby manage all of the memory why do we have this 40 byte limit right so the answer is that GC algorithms are a lot easier when you have a fixed width fixed width slot so with a fixed width slot it's much easier to write a GC and I think that's basically the origin of it we can have I don't think there's any particular limitation or reason why we couldn't store everything in Ruby as garbage collector essentially it's just that someone needs to do the work to implement that so if I had to guess the reason we have fixed width slots is historic reasons maths maybe yes he's not paying attention but there you go I will make that the answer thank you for the talk you say something about the retos let's say the hash you cannot move the location because of the GC do you think that's retos will affect the way we write call let's say like having a lot of objects that has actually affect the performance and therefore we may end up writing few hash or kind of stuff do you think we affect our way of writing code right so the question is will the behavior of the garbage collector or the implementation details of the garbage collector will that impact the way that we write code it shouldn't I don't think that it should impact it I think most of those red objects we can actually eliminate so I think that with a bit more development work we can reduce it such that the percentage of objects that can't be moved is very low and you would I mean it would be so low that there is no reason for you to know that it is even a thing so no I don't think it would impact it okay thank you very much Aaron thank you thank you for having me