 And next up we have Brian Burham, who is going to be talking about the Go Garbage Collector. Ronald Voplas. Hello. Well thank you for coming along so late. Who has tuned a garbage collector? Okay, not too many people. Okay, so I was wondering who would show up to this talk. Anyway, I'm going to talk about tuning the Go Garbage Collector. All of the settings you can use to do that with. That's a joke that nobody got. Okay, we'll get there. So first of all, let me introduce myself. My name is Brian Burham. You can follow me on Twitter and ask any questions you like afterwards that way. I work for a company called Weaveworks. We do DevOps tools, continuous deployment, stuff like that. I like making code go faster. It's like a video game for me. So that's kind of why. That's where I'm coming from when I want to tune the garbage collector. Oh, I need to stay here. I need to not walk. Sorry. I can walk around here? It's too far? Not far. Yeah, so let me tell you about my terrible life. So the first Unix machines I programmed had four megabytes of RAM. And we had a server with 16. I used to sneak on to the server to do my compiles because I didn't have enough RAM. And fast forward at Weaveworks we run the SAS, this cloud system on Amazon. We had these 32 gig machines a couple of years ago and we prepaid a year. And we spent basically a whole year running out of RAM on 32 gig machines and tuning it. And then finally the prepay ran out and we went to 64 gig machines. I'm certainly much more relaxed now. And so we care about the garbage because if you don't take out the garbage it fills up and your life is a mess and so on and so on and so on. These are some slides from the talk I did last year. I might just skip through those in the interests of time. Basically the point I was making is every time you garbage collect it kind of runs all the way through memory in your process. So you have stuff in cache. Can I move the mouse around? Yeah, the cache is over here. So the cache has got stuff in it. And then after the garbage collector runs it's kind of run through your whole memory space looking at stuff and basically emptied your cache. And it slows down your program in ways that you might not expect. Certainly it's garbage collection slows you down more than just the time it spends garbage collecting and it interferes with the cache behavior of the machine. So this is another picture to try and make the same point. Stuff you have on the stack is very easy to clean up. When you finish using something on the stack we just go zip to move the stack pointer. It's all gone. Stuff on the heap. The garbage collector has to go all the way through it looking carefully and saying is this thing still in use? Can we find it from somewhere else? That's my introduction to the theory of garbage collection. So we need to worry about it. It's going on all the time in the go run time. We are impacted by it whether we notice or not. So let's look at the options to tune the garbage collector. These are all the options you get to tune the garbage collector in Java. Okay, they're not all. These are all the ones I could find in like 20 minutes of Googling. Let's look at the list of options you get to tune the garbage collector in go. You ready? There is one option. I signed up to do a 30 minute talk about one option. 25 minutes, okay. Actually the timer is not running. Who knows how long this takes? You can. Good. Okay, so let's look at... I'm going to hopefully do a live demo. If the live demo doesn't work, I have screen captures. So I'm running a client server micro service setup as my demo. The server is a thing called pod info from my colleague Stefan. The client is from Jana Dogen who was here this morning but is not here now. Client is a thing called hey. They just like generates HTTP load. So let me see if I can get my demo going. Okay. So the first setup is a big server process. A big server process that has some kind of large amount of data in memory that basically sits there all the time like a cache of results or some data set that it's processing or something like that. And I have simulated this by... I scrolled it up too far. Yeah, I actually added that option to Stefan's program. I just allocated half a gigabyte of memory just in a byte slice. This I could probably try and make a bit bigger. There you go. So what I want to point out is this uses like garbage collection in action. I'm hitting it with like five requests a second so it's not really doing very much. It's using basically nearly zero CPU. But the memory is going up and up and up and then bang down again, up and up and up, bang down again. Everybody got that? So this is garbage collection in action. You good? Thank you. And this is the default behavior of the Go garbage collector. It will clean everything up as much as it can. And then the next time it garbage collects will be at twice that value. So I hogged half a gigabyte of memory in my program. My program is not doing anything. I'm returning some strings saying it's alive or something like that. But the program as a whole is using an extra half a gigabyte of memory over and above what I allocated because that's the behavior of the Go garbage collector. So that setting, Go GC, is a percentage. And the default percentage is 100%. So it gets 100% bigger and then it collects. So the way I got to learn about this is actually I found this setting in the Prometheus system. Prometheus is a good example of a program that sits there very large in memory. And the Prometheus authors tuned the Go GC value down. So let's just try and do that. I'm going to put the microphone down and type for a second. Okay, so I'm terminating my server. So I'm going to run the same server. I'm typing one handed so that I would probably use control keys and things. Go GC equals 50. Oh, I typed. I typed Go GC with a small letter. So let me just control C that and do it again. Thank you very much, Mr. Microphone stand. So I'm running pod info inside a container that has nothing to do with the demo. I'm monitoring it using Prometheus. That again has nothing to do with the demo. And I'm visualizing it using Grafana, which is drawing these charts. Who knew Grafana could do white as well as black? Yeah, there you go. You learned something else. So what I've done, I left the load generator going and I changed Go GC to 50. Right here. Okay. And so what I, it's going to take a few seconds just to flow through on the chart, but what we can see is it's now garbage collecting closer to the kind of baseline, right? And in fact, it should kind of steady out at the 750 level. So this is a trick you can use. You just set that environment variable Go GC. If the situation you're in is your program naturally has a huge amount of memory. I mean, you know, our Prometheus servers sit at 20 gigabytes. In fact, we've worked runs a hosted Prometheus as a service. And we actually, we have this thing called Cortex, which is like a multi-tenant Prometheus. And those servers, we have like a bunch of them that sit at 20 gigabytes. We do not want to spend an extra 20 gigabytes of RAM just doing that, just kind of letting the garbage collector chill out. So we use this variable in production tuned down for the processes that have a big stable memory baseload. Okay. And my chart, yeah, my chart's kind of working. I did one earlier. Let me just find that back to the slides. So large stable data set. Yeah. So if you tune it to 50, the garbage collector lets it go 50 bigger and everyone's happy. What's the next one? High GC rate, small heap. Okay. Let's try and emulate that. So shut down that server. Yeah, go on. Just shutting down that server. Do not hog the memory. And we're going to leave Go GC on the default. Okay. And then we were running hay like five a second. I'm going to run, so this command is 10 concurrent threads doing 20 seconds. So 200 a second. It's going to run for two minutes. So let's take a look at the charts. Oops. Let's open that up. Right. So just for clarity, I have totally changed my perspective. The first example was even though I'm running the same program, I'm in a different pretend now. The first case I was pretending I had a program that ran with a really big stable data set. So now I've moved to pretending I'm working with a program that uses very little memory, like something like a reverse proxy. It turns out we have one of them at work as well. And so what we observe, let's rather than let it run the full two minutes, let me put the pre-canned stuff up on the screen. The CPU, sorry, the heap was like tiny, like four megabytes or something like that. And I hit it with these 200 requests a second and it went up to like 10 megabytes. The CPU usage went way up. The garbage collection rate went way up. You know, before I was showing you it was garbage collecting like once a minute. Now it's garbage collecting 60 times a second. So we can go in the opposite direction. We can raise Go GC. We can say, dear Mr. Go garbage collector, I would like you to use more memory. Just like chill out, man, and use more memory. So this is Go GC equals 200. I could run the live demo, but you'd probably get bored. Go GC equals 200. So now we're letting the program use twice as much heap, right? We can kind of see that sawtooth effect here. It's kind of leveling out at about 10. It's still not using very much more memory. But if you compare it to the previous slide, it's garbage collecting half as often. So like, you know, I told you my life story, my whole machine had four megabytes to begin with. Now we use 32 gig machines or whatever. We can spend more memory than 10 megabytes. So let's set Go GC equals 1,000. There we go. So now we're using, you know, it's a tiny, tiny demo program, but we're using like 30 megabytes of memory here. The Go, this is the point where I ran it. You can barely see that GC rate went to about five or something like that. And the latency is lower. Let me just skip back. Let me use this thing. So that was the original one. The latency was like 10 milliseconds. And the latency just by setting Go GC just to use like 50 meg instead of four. Latency came down like a third. So certainly in my production life, getting your latency down by a third is a pretty damn good result. So that is the second thing you can do with Go GC. So the first thing you can do is set it down. The second thing you can do is set it up. Is there anything else you can do with it, Brian? You have a question? It's a percentage. Go GC is a percentage. The question was what is the unit of Go GC? And it is a percentage of the minimum heap size. So when it garbage collected in this example, it collected all the way down to about four megabytes. Times a thousand percent is 40 megabytes. So that's where it gets up to about here somewhere is 40 megabytes, right? Yeah, so it's a percentage relative to the minimum size. So I'll give you a link at the end. There's a really, really good presentation that goes through the last five-year history of the Go Garbage Collector and all the reasons why they did things and the background and so on. One of their aims is to make the thing kind of self-tuning and self-adaptive. That's why everything is relative. That's why it doesn't look like the Java page. So this is a relative measure, but there we go. So Go GC set it to 10 times the original and my latency went down by third. My CPU usage went down to about two-thirds. The GC rate went down by a factor of 10. So that's a pretty good result. There's another thing. You can kind of see it in this picture as well. When the load went away, we don't actually get all the way up to 40 megabytes, right? And in this example, you can barely see it. This is when it was like ticking over on the 500 meg case. So it turns out there is another setting for the Go Garbage Collector, but it's not a setting that you're allowed to set. The Go Garbage Collector will run every two minutes if it has otherwise not run. So that's useful. If you did something very intensively in your program and then stopped and your program just waited for input, then by the rules of Garbage Collector when you hit a limit, it would never run the Garbage Collector. And so your program would sit there hugging all the memory in your machine. So they run it every two minutes, two times 10 to the 9 nanoseconds, every two minutes. And they will actually free the memory back to the operating system if you're no longer using it in that scenario. So that's this sort of, which I didn't actually know about until I did these slides. I didn't know there was a two-minute max on the Garbage Collector. It is a variable, but it's an unexported variable in the 4,300 line file that I had to read to find it. Okay, one last trick. There is one more thing you can do with Go GC. You can set it to the word off. So why would you do that? You would do that if you had a truly infinite amount of RAM. No, nobody has an infinite amount of RAM. Okay, but it turns out basically there's a certain set of programs that just will terminate before they've used a huge amount of RAM. So let me see. Do I want to do the demo? No, I'm scared. The pre-canned version. So I built the Pod Info program that I'm using for the demo, and it builds in like seven and a third seconds on my laptop. If I turn off Garbage Collection, it builds in five and a half seconds. And this is a four-year-old MacBook, so it certainly doesn't have infinite RAM. Yeah, I think the compilation of that program, which like vendors Kubernetes libraries and stuff like that, it's not a trivial program. The compilation runs in 50 meg of RAM or something like that. It does not need to garbage collect. And so it's a little trick you can try. You can turn off Garbage Collection in your compilations, or if you have some other kind of batch operation written in Go that just runs for a bit and then terminates, you can just turn the Garbage Collector off and you might get a speed benefit or you might get a crash. Now, this is open source, no guarantees. Okay, fantastic picture from Ashley McNamara there. We might get more Go Garbage Collector settings to tweak in the future. We might approach the Java page over time. So there are two that you can read about, clearly talked about. One is set max heap because, you know, this thing about, I'll just go twice as big as I was last time. You know, at some point it goes, bang, this was my life, right? For most of last year, our processes would crash in production. And so just telling the Garbage Collector like, yeah, I love your sawtooth, but just don't go above 20 gigabytes, or whatever number you pick. So what that does is it trades performance for safety, right? Because if you actually need that much memory, your sawtooth is going to get faster and faster and faster until eventually the program does nothing because it's just garbage collecting all the time. So this is in an experimental branch of the Go runtime that I couldn't actually find, but I can read about it and you can Google it, you can find all that stuff. On the other side, there is also, I think, accepted support for a minimum heap size. So this is basically the same thing as I did with setting Go GC very high, saying, look, I want you to use more memory. But it's like a safer way to do it because the multiplier has that dangerous effect that the sawtooth might get out of hand. So Go GC min, I think that will show up. That's a much simpler change to the runtime. So probably show up faster, sooner. Okay, I am done with my talk. Questions? No? Okay, a couple of links. Oh, a question. Yeah, do I run with the mic? Thank you. So what about GC times? Are GC times strictly linearly dependent with the amount of RAM that needs to be wiped or is there a sweet spot between GC times? Okay, so I think you're asking about the time the garbage collector takes to run. And, well, so number one, I will say the impact of garbage collection on performance is way bigger than the time the garbage collector takes to run. So Go over the last few revisions moved to a largely garbage collecting in the background. So the actual time it stops your program for is under most circumstances, tiny, tiny, tiny. There are edge cases. That whole slide about the cache and so on, garbage collection will slug your program, will really slow it down in ways that are really hard to understand. You know, it's just this fact that it's trolling through everything in your program. So there is no metric. Other than the end result, like my latency went down by a third. So that was the impact that GC was having on my program. There is no measurement of GC time that will add up to a third. And I... So, you know, sorry, but I do this a lot. And that is absolutely my finding. There is no simple relationship between anything and anything. You've got to kind of try the different values and see what comes out. Okay, out of time. Thank you very much.