 All right, we'll get going. Welcome. So this talk is the next frontier in open source Java compilers, the just-in-time compilation of the service. So hopefully everyone here is familiar with Java. I hope so. That's me. I approved that picture. Rich Agarty, IVM developer advocate. So we all understand Java, the underlying JVM, and the JIT, how that works. The Java code is sent to the JVM for execution. The JVM uses the JIT to compile code in an optimized way, and as time goes on, your code is optimized and runs better and better. That's the theory. Okay? With that in mind, that's what the talk is about today. So we're going to talk about the problem, which is Java on cloud, specifically distributed architectures like microservices serverless. Then we're going to talk about the reason this is a problem, and that is because of the JVM and the JIT, which has a lot of great things, but also is a reason behind the issues we're going to have. And then we're going to talk about a solution, which is JIT as a service. So again, let's talk about Java on cloud. Is it a good fit? So first we need to talk about where Java came from. Java is all about monoliths. And that's when you had a dedicated server, tons of memory, tons of CPUs. This thing would run forever. It never went down. It didn't matter that it took 15 minutes to start up because you never started it up. It just never went down. You would upgrade maybe every six months, one year. That's the old model. So now we want to move to microservices cloud native applications with our Java application. So now we're running in containers. Now we're managed by a cloud platform like Kubernetes. And we need scaling. We need to scale to meet demand. So the main motivators we have for moving to the cloud, the obvious ones. It's going to be so flexible, so scalable, it's going to be perfect. Once you roll out new releases, independently of others, especially with microservices, right? We can update one without affecting the other. Take advantage. So once you're on the cloud, all the new innovations that are coming out of the cloud, we can take advantage. Less infrastructure we need to keep and manage. Saving money, that's the big one. That's the theory. So here's the problem. It gets down to performance versus cost. And why this is so hard with Java. So there's only really two variables with regarding cost is one is the size of your container and the other is how many of those instances you need to run. It's all related to resources. What are you going to get charged for? So in this case, we're looking at cost and performance. So right here, we've got minimal cost. We probably undersized our containers. We're not running enough instances. We're not spending a lot of money, totally unacceptable performance, right? So we may boost it up or run a couple more instances, but it's still not very efficient. We're just spending more money, but still not performing correctly. And obviously the worst case is we make oversized containers. We have too many running. We're spending lots of money. Performance is very good, but you're not going to like the cost. So this is a sweet spot, very, very hard to do. In fact, there's a lot of companies are putting a lot of effort into trying to figure out a Java solution for that. And that's why I'm here. So the JVM and the JIT compiler, the good and the bad. So we're going to get to the reasons behind all this, why it's so hard to do. So the JVM itself has been around for 25 years at least, device independent, right? That's the whole promise of Java. You have a JVM on architecture. You can run your same Java code without change. The JIT produces optimized machine code. So when your JVM starts up your application, it takes that code, it runs profilers into your code. It looks at the inputs, your methods, it determines, okay, these are the hot methods. Let's compile those, and it can compile in multiple, multiple times. And your code becomes very, very efficient. We have very efficient garbage collection, the JVM. That's why your code can run for years. The longer it runs, the better it gets. So the bad. So that initial execution of your Java application is interpreted. Hopefully everyone knew that, right? Bite by bite, very slow, relatively slow, until those methods can be compiled. The hotspots are identified, in fact, the Oracle JVM is called hotspot. So it identifies the profilers to identify the hotspots and says, let's go build that, let's compile that method with the JIT. The issue is this initial loading of the JIT and compiles can cause CPU spikes, lowers your quality of service. Memory spikes cause out of memory issues, including crashes. The end result, very slow startup, very low ramp up. Startup is the time it takes for your Java application to be able to service a request. That could be during the initial interpreted phase. Startup is the time it takes for your JVM to identify what methods need to be compiled and it compiles all those fully optimized. That's the difference between startup and ramp up time. So here's an example of what happens with your JVM. These are your spikes associated with those initial compiles. Here's your memory footprint. So we need to find that sweet spot, which is getting that container to be just the right size to handle those spikes, but not wasting time, wasting space. And we need efficient autoscalers. So let's revisit those again now that we know more about what's happening in the background. Here's the problem. We will always over provision. We're going to run tests on this container, we're going to find out. We need to let's set up this container to be right about here, to handle these spikes. The problem is 90% of your code is running down here. So as soon as we get past this level of spikes, we're wasting space. So you're always going to over provision just the way it is. Plus JVMs are non-deterministic, meaning you run the same code twice, you're going to get different spikes at different levels. So we have slow startup and ramp up times for the container and oh, I'm sorry, this is for autoscaling. Scaling is not going to be efficient. You have microservices that you want to scale up and down based on demand. It's not going to work very well if your startup times are too slow, your ramp up times are not very slow. It has to be quick. Demand comes on, you need four more instances of your microservice. While it takes 10 seconds for them to start up each, it's going to be too slow. Users are going to know. Also CPU spikes can cause autoscalers to give you false positives. So autoscalers are based on metrics like CPU utilization. So if you say anytime my CPU gets over 50% fire up another instance, if you're telling Kubernetes this or the autoscaler, your JVM may start up and start compiling and go over 50%. So it hasn't actually processed anything, but it's over 50%, so it's going to fire up another instance. And that's not right. And it's going to thrash about, it's going to kill them off, bring them back. So it's going to be very inefficient. So solution, we've got to minimize, eliminate CPU and memory spikes. We've got to improve startup and ramp up times. Very easy, right? That's all we've got to do. That's where the JIT as a service comes in. So here's the theory behind the remote JIT. We want to have our, oops, come on, I apologize, this is our normal JVM. We want to have remote JITs that we can access remotely. So we offload the JIT compilations to the remote services, right? These are going to be managed just like any other microservice by your orchestrator, Kubernetes. It's basically your classic monodomicro solution. In this case, the monolith is a JVM, now we're splitting that into a JIT and whatever is left into two microservices. The other thing is the JIT is still available in local JVM. So if this goes down for any reason, the JVM could still function. So we're basically treating our JIT compilation as a cloud service. So does that exist? It does and it's actually a part of the OpenJ9 JVM and it's called JIT server, also known as SAMRU cloud compiler. OpenJ9, does everyone know what OpenJ9 is? So OpenJ9 is a JVM that's been around, well, I'm going to talk about it here in a second. So initially over 25 years ago, IBM came out with the J9 JVM. So IBM identified early on that Java was going to be very popular. They wanted to make sure that it worked on all of their devices. So from their handheld scanners to their mainframes, so they built their own JVM. It's called J9. About five or six years ago, they said we're going to open source it to Eclipse and now it's called the OpenJ9. And it's distributed with IBM's distribution of the OpenJDK. So everyone here, if you've never heard of OpenJ9, I assume you are using Oracle JVM hotspot if you're using Java. So just to let you know, OpenJ9 is built for small containers that's built for the cloud. Here's some testing we have going against hotspot. So it's basically half the size, half the, it ramps up faster, uses less memory. So OpenJ9 comes with IBM's distribution of Java, which is called SAMRU Runtimes. So I don't know if you're aware, it used to be you get Java from Oracle or Sun if you're really old. But it was open source about 10 years ago. There's been different distributions of now. There's multiple players that this distributed, right? There's Amazon, Coretto, Oracle has theirs, Red Hat has theirs, Microsoft has a version, IBM has a version. It's called SAMRU Runtimes. And if you're wondering what the name is, SAMRU is the tallest mountain on the island of Java. So that's the correlation there. So what advantages do you get with a JIT server? From a client standpoint. So if I'm a JVM accessing this JIT, now it's going to be a lot easier for me to provision the container, right? We don't worry about those spikes anymore. So we used to have to provision up here to handle the spikes. Now we just worry about the actual container size, what the application needs. We're going to get improved ramp up time because the JIT server is doing all that hard work. Reduce CPU consumption. The cost. You're going to reduce your memory cost. We're going to show that here in a minute. We did a couple of experiments. And you're going to get a lot more efficient auto scaling. So the scaling is going to be correct. It's going to be based on demand instead of based, not based on, oh, the files caused you to start erroneous instances. And resilient if the JIT server does go down, the JVM still has access to its own local JIT. So we did a test. This is an example of the size of containers. So we basically took four big job applications. And we created multiple instances of each. And we scaled them, the container size, such that they would perform well. And we put that onto AWS, I think, on OpenShift. Yeah. And this is what we came up with. So this is kind of hard to read. But the top one is our baseline. This is without the JIT server. So we have five applications that all have the same color. So example, AM is the Acme Air monolith application. We determined we need to have 500 megs to run efficiently. And we're going to have eight instances of that. And if we do that, we ask OpenShift, which is our platform we're using, to lay that out. It says, OK, we're going to need three nodes of this size to run what you want to run. Then we introduced the JIT server, which is right here. This bottom row is the JIT server. So the JIT server itself is the biggest application in the node. See how big it is? But even with that, all of our applications almost shrunk in half. So we have the same number of instances running. Even with the JIT server, we're now using two thirds of the allocated memory. If you're wondering how they perform against each other, basically it's a wash. They both perform the same. Here's an example of auto-scaling. Again, with the JIT servers in blue, you're going to see that the dips aren't as severe and it comes back up and ramps up a lot faster. I want to show you a demo real quick. So here's a demo. We're going to take three applications. We're going to take one application and run three instances of it. It's called Acme Air and we're going to run it in containers. The first one's going to have 400 meg, the second one's going to have 200. The third one is also only going to have 200, but it's going to have access to the JIT server. The top two is not. So these are on their own. The bottom guy gets that. So this MongoDB is used by the application. The rest of the stuff, all these containers are just needed for the test. So they're going to gather metrics and they're going to display them. So let me go ahead and show that at work. I got this all running right before this started, hopefully everything's still there. All right. So all the containers are all running. I'm going to go ahead and start up my Acme Air. So I'm now going to fire up those three containers and then I'm going to run Jmeter, which is going to apply load to each one of those. Okay? I don't know if you can read that. Hopefully you can. So this is what that Acme Air container thing is. So right here, I'm running three versions. You can see the three Docker runs. The first one has 400 meg, the second one has 200. The bottom one also has 200, but it has this extra JVM argument that says go ahead and use the JIT server. That's the difference. So now we'll go over to, we have Grafana running some graphs back on what's going on. So this is CPU throughput to tell you how efficiently everything's running. The top graph, these two are running without JIT server. This one's running at 400. This one's running at 200. This guy is running with the JIT server at 200. And notice how fast this guy ramps up even more than this one. It's ramping quickly. This one running at 200 is never going to really get very far, right? In fact, probably nothing's being compiled, because the JVM is pretty smart in saying I'd like to compile these methods, but you don't have enough memory. I'm not going to cause an out of memory issue. So we'll just shut it down. We're just going to run interpreted. You're not going to be very efficient, so it's going to run that way. So these guys are eventually going to get to very optimized state, both of these. But this one's doing it on half the memory. That's the key takeaway, right? We got the JIT server running. We're using half the memory. So this is a graph. In case the demo failed, this was my backup plan. So how to use it? It's basically real simple. I don't know if you picked up on this, but basically the JIT server is just another instance of the JDK, OpenJ9. So you can run the JVM normally, like you run a Java, my app, or you can run it as JIT server. It's just a different persona of the same OpenJ9 JVM. Full list of options are open. Again, it's open-sourced under Eclipse, so there's a whole list of options you can run with it. The key point here is we do have a certified operator. So OpenShift operators, I don't know if you guys are into those Kubernetes. It makes life so much simpler. So we have a certified operator for Open Liberty, which uses OpenJ9. And it's as easy as, there's an enable flag and you just say, I want to use the JIT server and you just get it. There's not much you have to do. There's no calls you need to make. It's just the startup of the JVM that's different. Obviously, there's a lot of parameters you can run with some, we don't recommend you using any, let's see, I'm sorry, I had a thought that just escaped me. It does add to the network latency, so you shouldn't be using too much of that. If you can get away with it, don't use it. It does provide some metrics back to Prometheus if you want to track how it's doing. You typically want to use 10 to 20 clients per JIT server. So that's key. You noticed in that one graph we showed, it was the biggest container, well, you need to connect to a lot of clients to make up for that. And you do need to make it pretty big here, one to two gig of RAM. And you don't want to start everything at the same time. You want to stagger the starts. You don't want to start 20 clients at the same time using the same JIT server. There's another feature. I had another talk where I talked about serverless. And Instant On is another feature of OpenJ9. So one of the benefits of OpenJ9 is that both IBM and Red Hat are fully vested into this open source JVM, and they're coming out with really good technologies like the JIT server. By the way, as anyone, I talked about different vendors providing different solutions. The OpenJ9 JIT server solution is the only open source version solution out there. There are other solutions just like this, but they're proprietary. Also on the Instant On is another feature of OpenJ9, which basically takes your job application, takes a snapshot. And then you could start from that point, and it starts up instantaneously in milliseconds as opposed to seconds. Most job applications take five to 10 seconds. We get that down into the 300 millisecond timeframe. There's also different solutions for that. Different vendors are coming out with, but they're, again, they're proprietary. They're not open source. So Instant On is another feature that's come out that addresses serverless. But Instant On can be combined with JIT server. So those containers that you create that scale up and down can be instant on images, and they could start instantaneous. I was going to show you if we had, I think that's it. So final thoughts, the JIT does provide a great advantage, but the compilation is the problem. That's the problem with Java. So if we disaggregate the JIT from the JVM, we're making a JIT compilation as a service. That's what we're proposing as a good solution. The Eclipse Open J9 JIT server is also known as a SAMRU Cloud Compiler. It's available now on Linux Java 8.11.17, and it's distributed with the IBM SAMRU Runtimes. So it's great for constrained environments. You're only going to want to use this with containers and microservice architectures. It works seamlessly with Kubernetes, and it's going to improve your ramp up time auto-scaling performance, and it's going to reduce your memory footprint. So you're going to, it's all about, we talked earlier about the motivations for moving to cloud, and one of the things a lot of people find out is they don't save a lot of money because of resources. It's using more resources than they thought they'd need. And one of the big reasons is because you don't know how to provision right. You need to provision for worst case scenarios, and all we're doing with solutions like that is eliminating those, and you can work to get those costs down. That QR code points to a series of articles and blogs around this technology and using it in different environments like Kubernetes. Anyway, that's the end of my talk, but thank you. Thank you for coming. I know it's late. Did I get done in time? Yeah, I think I did. Any questions? Yes. I can repeat the question. I think he said something about if it works with Quarkus, or is it an alternative to Quarkus? Did you try to use it with Quarkus, and how does it perform comparing with Graal VM? So there, okay. I alluded to that earlier. So Graal VM has a couple of solutions like that, right? So Graal VM has a crack service they have. So the underlying restore feature for the Instant On instantaneous comeback is the underlying, it's a Linux feature called CryU, checkpoint restore in user space. That technology has been around a while. You can use that with Linux processes. You can say run, freeze, or stop, take a snapshot, come back later, start right from that point. Same thing is applied to images that are put in containers. So a couple of different vendors have used that. So Quarkus has another feature, it's called static compilation. Is that what you're talking about, Graal VM? So the point about that is they've decided, okay, Java isn't fast enough, we're never going to get it fast enough, let's compile statically beforehand. So they take a static compile like your C program, they get an executable, very small, runs fast, and then they run that. The problem is it's not built, you lose the JVM, you don't have a JVM anymore. All the things that are nice about JVMs are gone, garbage collection, all that. So it doesn't work very well with short lived, with long running applications, you wouldn't want to do that. So it has a place, but you have to take a whole new framework, right? So I got my Java, I got my process set up, and okay, for this thing we're going to build Quarkus. It's a new framework, we compile differently, how does that fit into our CICD pipeline, how do we test that? Java developers aren't going to compile the code, they're going to run the JVM. And now we have to release this thing in, and we have to have a whole set of tests around it, so it makes your life more complicated, so that's my pitch. But I understand, I mentioned that, that green spot is very hard to get that right, and a lot of vendors are coming up with solutions, and that's one solution, saying Java just doesn't work. We're going to compile ahead of time. All the cool features of Java, dynamic, loading, and all that stuff, you can't do, by the way. You have to have a very small subset of Java, you have to compile ahead of time, and then run it that way. But that's it. Yes? Do all the compilation improvements happen per container? Well, the JIT server can service multiple clients. That's what I was going to get, so does it share the customization, is that where the power is coming from? Yes, part of it. Part of it is the initial compilation that's going to give you, it's basically giving you CP cycles, because it's doing the compilation. But if I have microservice A, and I have 10 instances, and they're running, they're getting scaled up, the first version of that compiled a bunch of its methods, and the JIT server has those. Now the second instance comes along and says, hey, I need this, I already got it, it sends it right back. So that any new container will not have the cold start problem, right? It shouldn't have, well, no. I do want to be careful. Not all instances are created equal, so the profile has to match exactly. So the JIT server just knows that it's getting a request for a method, and if it's already got that compiled and the profile matches, it will return that. Otherwise it's going to compile it again. But even if it didn't have it ready to go, the fact that it's compiling it and your container is not, is the savings, because you have a smaller container. Any other questions? This is not brand new technology, either. Now the instant on is coming out, OpenJ9 is releasing that soon. I'm not allowed to say when, very, very soon. The JIT server or Samaroo Cloud Compiler has actually been around about a year and a half, almost two years, right? And we're, it's slowly getting better and better, different use cases, different platforms, cloud platforms, you know, testing stuff from the cloud is very difficult. There's the matrix, test matrix is huge. All the different Java versions, all the different cloud platforms with different orchestrators. So it's getting better and better. We're finding issues and we're fixing them. All right, well, that's it. Thank you very much for coming.