 Hey, so my name is Chris. I work for Twitter and So this this year I thought I'll do something different And I was a little worried because I'm doing basically Everything's life demos more or less and a Wi-Fi didn't really work So I was a little worried about it, but we got it working. So I work for Twitter Twitter has a small VM team. It consists like the four people and then we have another guy working on machine learning stuff If you're going to tweet about this talk, it would be nice if you add that hashtag so that my other team colleagues know What's going on here at Fostom? So in general Why am I doing these talks, you know, you might have seen a talk of Mine before and and they're always about growl stuff and how we do things I don't know if you've seen my previous talk where I talked about That we use growl at Twitter in production. So it actually works and stuff So you can watch that on YouTube if you want My the main purpose of my talks is always to get people to try it like Christine just said try your GCs I say try your compilers. So At Twitter we save money by using growl, I cannot tell you how much money, but it's substantial It's way more than I get paid. So Yeah, I don't know if that's good enough So you can also save money Yeah, right exactly so That's one good reason Or something you can you can approach your manager and you know say hey We might be able to save some money. So you have to try out The other thing is that I want to fix all the bucks that are there, right? That's also very selfish because if you guys find the bug and you fix that bug we at Twitter We don't hit it and Twitter stays up. So that that would be good And then lastly also to improve growl, right? There's still a lot of stuff we can improve on And it's a little easier to work with then see too. So it's actually easier to do improvements and then people come to me after my talks and say well Is it safe to use or does my data center burn down if I run growl? So our data centers are still fine and Since I don't know tune last year or something We are running our main Twitter services. So basically everything you when you tweet or when you read tweets everything 100% of that stuff is running on crawling production today since last tune or something. So it's fine Then people ask me how do I use it? Where do I get it? Sometimes I get emails later and people have been trying it and They mostly complain about the benchmark numbers and they usually say well, it really sucks That's because they don't Really understand that brawls different than the existing compilers And so this talk is to explain this So what is growl? Just briefly it's a child virtual machine just in time compiler. It takes Java bytecode and Translated translates into a native code actively developed by Oracle apps John Rose was talking about it earlier a little bit So I don't go too much into detail. These are the links where you can, you know, read up on it It uses JV MCI and it's written in Java and That's very important to remember throughout the whole talk. It's written in Java. So it has All the properties that Java programs have right it uses Java heap memory It loads classes all this stuff. So remember this Where do I get it? The answer as always is it depends Unfortunately, but this will change pretty soon with with JDK 10 this will change But where do I get it? So there is chap 295, which is the out of out of time compilation chap and this one When you go to this chap, you see, oh, yeah It's in release nine and it's delivered. So right because nine's out When you scroll all the way down, you see the dependencies and it says it depends on JV MCI and then It uses growl and so on and the project will merge growl core into the and it delivers it Magic And delivered in Linux x64 builds and so that's that depends So if you're running on Linux x64, you can take JDK 9 and you have growl automatically If You're not on Linux x64 then you have to either build it yourself Or you can download and a JDK 10 early access build and then you get it there's this there's this one bug here that adds support for Basically the the reason why it's not in 9 is because we couldn't write the the native shared libraries for the aot feature That's what kind of do it Okay, so what do you need is a JDK with JV MCI chap 243 and with aot because that gives you growl for free Basically, and that's on Linux. That's greater equal than JDK 9 or Mac windows Great equal to 10 and you can download it from here and We're not doing doing that now. So you the way I do this talk usually is I use an Oracle cloud container and I start that container up like on the fly and I'm doing this To show you when people like when I when I go to conferences and I see talks where people do life demos I always wonder how many hours did they spend to prepare? The container or the machine that actually everything worked right the Dems and stuff So and but I decided I fire up like a cloud container With no setup at all and do all the Dems and show you that really all you need is download all that You know the the JDK and so on But since it's only 25 minutes, I can't do this so you have to trust me on this one I'm just showing you so I started one up I'm part of the of the Oracle developers champions program and so they give me free access to to cloud Time it's like a budget and then I can use it. That's really nice. So that's what I'm using So I have a I have a machine in Frankfurt as you can see here and Everything's so small that I don't even know how to get there All right, usually I don't know how to get there now. There's more up here where you can see Yes Instances that's what I want. Thank you. So I have a running foster instance And it has a public IP address this one so We just log into this guy right and the second time and And So we have I've already downloaded all the stuff that I need and we have the chatty kid nine That's that's the open chatty kid nine build you can get from Java dot whatever it's called the link I had earlier So just download that one and we are setting just Java home here to this directory and then we are Also putting it on the path and I've done this before so I can shortcut a little bit and then we have This Java version and over here. We are doing the same thing. We're comparing C2 and brah. That's that's why I have two windows here so we're doing a Java home again this guy and the path and Then in this instance. Oh, yeah, I'm also using this Environment variable called Java tool options and I I use parallel to see because it's a little easier later to Do not look at G1 output. It's easier to look at parallel output and and we do A 512 mega heap and maximum and start and there you go and The same over here Wait, no, let me go back so Usually I start up this container and it takes a while and then I talk about myself So the only thing I'm saying here is because Charlie called me out I'm the lava one organizers. So if you guys want to come to Hawaii next January, you know So the rest I skip so back to the demo in What do you can do in JDK9? Is you list all the modules? It's like, I don't know like 75 modules or something like this and then as we've seen before the AOT chap depends on on on JVM see I and and and growl basically and then you can say, okay, Java What's got describe describe module, right? So we want to describe the JDK AOT module and This no, that's not right Describe Here we go. So this is the JDK AOT module and it requires JDK.internal.vm.ci that's JVM see I and it also requires this guy here and that's growl This module contains the growl source code So if you have these two or yeah, these two you don't need AOT really But if you have these modules in your JDK9, then you're good to go. Okay, that's all you need So get a JDK with JVM see I in the OT. Yes check mark. We just verified that we have that and then The only thing you really have to do is to turn it on That's all so how do I use it? So let's go back to our chaps here and Then if we go to chap 243 Here you can see when you scroll down a little bit. Oh look, this is how you turn it on. So all you have to do is Take this so so since these are experimental options You have to unlock them and then you say oh, I want to enable JVM see I which not Automatically turns on the compiler. It just enables the Java interface and you could then use something like truffle to schedule Compilations, but what we really want is to replace C2 with a JVM see I compiler in in our case that's growl okay, so we take that and We stick it into our Java tools options like I have it somewhere here already We just changed that to 512 here real quick And as you can see unlock enable use JVM say I can tell okay So if we do something like this, then we see oh, yeah, the options are being picked up But there's no real difference right nothing's really happening. Okay Then Let me click you go back. Yes, there is so we have now JVM see I enabled and then you can do Print flaks final Version and we grep for JVM see I okay here we go these all the JVM see I options that we have So for example, yeah, we enable it and that's true that what I'm looking for is the JVM see I print properties So this guy here we go. That's a long list Let me a little wider here That's a really long list. So there are a few JVM see I properties. So since JVM see I is written in Java mostly And grad is written in Java the command line options are being passed as Java properties still dash capital B And then you know, whatever so there are a few Java JVM see I properties and then there are a bunch of growl, right? So the one we are looking for is this one here Which tells us a little bit about when JVM see I Starts up, okay So we do this and we say capital D timer We have to say true because we turn it on so we do a dash version and we see Nothing's really happening. So what the hell is going on? so the way it works is We are running tiered, right? And if we do a print compilation here Let me do it again. You see This column here is the level of compilation the tier level and and tier 4 would be C2 or growl and Everything we compiled so far because the dash version doesn't do a lot. We're not actually doing a level for compilations And so JVM see I will never be initialized because it's lazy lazily initialized So we have to do a little bit more We can do we can run the cup before example and do it at their shell So we can see okay, something's being initial initialized and it's called hotspot JVM see I runtime. That's a class But it seems like that's only it and it's not really finishing it And the reason is the initialization of JVM see I and growl would take too long So the the program actually exits before all the initialization is not so we have to run a little bit more So we do a small run of Aurora and is the oh, yeah, that looks much better. So we initialized JVM see I There's a there's an implementation of JVM see I runtime called hotspot JVM see I runtime right that could be Technically a J9 JVM see I runtime just throwing it out there And then we initialize something called a hotspot growl runtime and that's that's really that compiler You see we initialize all these providers and then we initialize we create the back-end for our CPU and so on and so On and then we run the benchmark. Okay, so now we are really using it good well, let's go back to the slides and I have ten minutes. Okay, usually a demo takes five. So maybe I can do two more bootstrapping is You you have to remember it's written in Java, right? So growl is just another Java application that's running in your JVM. So if you want to run it it Will get compiled as any other Java code you have running in your VM. Okay So it loads Java classes because it's written in Java. It has Java methods obviously and These at some point when your compiler when growl starts to compiling your application These methods get hot as well, and they need to be compiled. That's called the bootstrap. Okay So let me do a quick bootstrap demo you can turn this on Explicitly and you'd say oh bootstrap my JVM see I okay This cloud instance by the way has I think four cores so eight threads something like this And it uses a bootstrap it uses all the threads it can get and bootstraps it So what the boot what a bootstrap really does it takes all the Java Java lang object methods Which are super simple right and schedules them for compilation And then because you're compiling these small methods the whole compiler gets warm and compiles itself That's what the bootstrap does so it takes ten seconds to compile 2,500 methods which is quite a bit right so that's something where people are raising well You know, I don't want to wait ten seconds every time I start something But you don't have to explicitly bootstrap it You can also bootstrap it Basically automatically while you go okay, so When you for example, let's do a print compilation here oops a print compilation of Yeah, let's run that same benchmark again. Why not and we crap for Orc growl VM. Here we go So here we see that we didn't do an explicit bootstrap This is just being implicitly done while we go along right and you see all it's being compiled and the benchmarks done and some Methods will still be in compile But what's important here is you can see the compilation this level is always one and there is a flag In growl, it's called Wait a minute what it's called JDMC I want this guy print properties right and we crap for c1 only Here we go So there's a property called compile growl with c1 only and that's by default true and the reason this is set up that way is C1 Throughput is much much faster than any than let's say C2 or growl itself, right? So we want growl the growl itself to be compiled as quickly as possible and and not take as much CPU away from our From our application compilations or even the application itself. So we just compile with c1 and that's usually fine Yes, I can and if I have 50 minutes, I would show you how to do it, but Apparently we don't so yes, you can next year. I'll the other half of the talk will be next year So this is this is what a bootstrap is and how bootstrap work. We could we could also turn off You know if we go here and say okay dash capital B and This guy and then we say false Then we'll see all there are a bunch of one compilations in there and then three and sell at the end We'll get to level four compilations like here. So if you turn it off You go through the whole tiered system and growl will compile itself Takes much longer obviously, but I have six minutes left. I can yeah. Well, I'll show it to you. Why not? So we do the bootstrap and we say okay. We do false here. Oh Wait, nope. No, no, no, I'm not doing this. I have to show you something else This is important and I'm still debating If I should file a bug for this or not Because when you do a bootstrap an explicit bootstrap, it's actually Ignoring the compile growl with c1 only flag. So it's compiling it with itself every time you do and you do an explicit bootstrap So that's I think that's very important and I I don't know if it should do it actually I think it should honor the flag and if you tell it hey only compile with c1 just compile with c1. I should mention Sometimes an explicit bootstrap might help even if it if it's up front time that you were just wasting and waiting on but C1's not optimizing as well as c2 or growl does right. So c1 doesn't have escape analysis for example So all the allocations That will be done in growl the compiler when then go to the Java heap Nothing will be escape analyzed or scalarized or anything So if you're expecting that later in your application run you were still you're still compiling a lot And you want these compilations to be quick a bootstrap or turning off the c1 only flag might be better Okay So going back to the slides But I asked the bootstrap demo what they will learn. Well, we compile about the bootstrap compiles about 2,500 Methods I didn't do the non tiered one, but it compiles even more Because then more of growl gets hot and it just compiles more of itself So you can do it either up front or on demand during runtime. I Would suggest to you to not touch the bootstrap flags and just let it run If I if I could do the production demo that I have you you would see that it really doesn't matter And then by default we compile the only we see one and bootstrap The explicit one doesn't a slightly different. So Java heap usage growl is Exactly it's written in Java. So if growl compiles a method it uses Java heap memory to do that in Depending on the the c1 only flag that you're using if you if you set that to false all the growl methods itself will use Java heap memory that that's very important because usually We at Twitter we do the same right you tune your heap to exactly like how much you need Maybe a megabyte of free memory But then suddenly a compilation can kick in a big one and that can take up 100 or 200 megabyte of memory so Let's do a quick three-minute Java heap demo So we're actually doing something over here as well what I want to do is Do not yes. No, it's still all right. So we log to see and we run I Think we run that one again. So we do that. Okay, and then we see I'm using this benchmark because it's very Computational intense and it's not really doing a lot of allocation and she sees stuff So it runs like 10 seconds, right? And there is no GC happening during the benchmark run. It's just the this the Dacapo benchmark harness does these GC's explicitly So if we do the same over here We'll see that as a little bit more going on right and that's exactly what I'm talking about So if we are compiling the Aurora benchmark itself and while we are compiling it grow is using Java heap memory for that So you have to be aware of this The two things. Oh, yeah, one thing. I haven't shown you yet. I want to show you that too. I still have time So there's something called See I time it's a really badly named Option because what it what it's what that means is is the compiler interface time That's not actually true. What it really measures is the compiled broker time So how much time does it take to compile methods and how many methods does it compile? So let's do that over here too See I time And as we can see so we compile about a thousand methods with C1 and about 300 with with C2 Don't don't pay too much attention at the at the time and the especially the bytes down here when we look at the JVMCI numbers because these Contain this number contains inline bytes while the JVMCI one doesn't and it's just the way because it's C plus plus code and it's hard To get that value back into the VM. So just believe me. It's it's it's really not that important. So We compiled about a thousand here and Here we compiled six thousand. Okay, and that's all the ground stuff, right? you remember the The implicit bootstrap is compiling just with C1. So we're compiling roughly 5,000 additional methods. They're not all ground methods because there are also core library methods involved Because ground is written in Java. It uses a bunch more of the core of the core library So core library method methods get compiled as well And we compile 400 with JVMCI Which I'm not exactly sure why we do that. It's a hundred more and I also think it's We'd have to look at the compilation output But I think it's also our core library methods because the growl ones are only compiled with C1 But if something because of growl gets hot in the core library, it will still get compiled with tier 4 Does that make sense? I hope it does and I'm out of time Yeah That's that's that's the last thing I'm saying so Java user growl uses Java heap memory That will change at some point But today we don't have heap isolation and keep in mind Something that Christine also said it's important how much memory you JVM uses because C2 also uses that hundred or 200 megabytes to compile a method It's just not as visible because it's not on the Java heap. All right. Thank you