 Hey, well, good morning, everyone, or good day, wherever you happen to be. Rob Sidor, chief architect at Red Hat, here with Jeremy again. So Jeremy. Jeremy Davis. Today we got Jay Howell with us, and he's gonna talk about Flight Recorder and Cryostat. So without further ado, because I think he's the best person to explain it. Jay, take it away. All right, Flight Recorder. So we have a lot to cover. Here's what I'm gonna cover today. Just an easy, quarkis, rest easy, reactive application. We're gonna cover command line Flight Recorder. We're gonna cover the JCMD, custom events, how to parse custom events, Java mission control, a little bit of visual VM, and then we're gonna move into Cryostat and moving it into Kubernetes. So what this session is not, this session is not how to profile your application. This is how to use the tools on profiling. If you want that, you're gonna have to tune in to part two, when I show you how to use Java Flight Recorder to profile your applications. So without further ado, I'm going to introduce my test app. So this test application is an application that I downloaded from Quarkis, I.O. And it's really easy. It's the first one up there, rest easy, reactive. Downloaded it as a zip and let me fire it up. And right now I'm firing this up in the, there, let me get to this. Firing this up in the developer, and it's up and running. Okay, so this is just a really easy application that I fired up. All it really does is it has a rest endpoint that just says hello from Rust easy. I have another one in here that I pass an ID into and I've passed my name into it, hello J from Rust easy reactive. So this is just a really, really small, easy application that we can play with. Flip back to my slides. All right, so what is Java Flight Recorder? Java Flight Recorder should always be on, it's exactly what it says it is. It's a Flight Recorder. Now the question I get is, I have a lot of profile applications that I'm already using. Can I use this with those? Yes, Flight Recorder should always be turned on and it is going to suck up less than 1% of your overhead. It is made to be lean. It's integrated with the JDK. And really what this is, this is if everything falls apart, your JVM runs out of heat, you're having problems, you drop back to the Flight Recorder, that will tell you what happened. We have a couple of different profiles in the JDK. We have two profiles. So what if I told you, if it's not documented, it didn't happen. You need to turn Flight Recorder on. JFR is event driven, which means that this is how it can be lean and mean. Samples of vets coming out of the JDK, your custom events if you want it. And it also samples JDK events. Why use Java Flight Recorder? It's based on JVM internals. It's a low overhead, continuous monitoring. It does detailed profiling, low configuration, event based. You have a lot of visualization and JDK tools for it. And one of the best things is, it's free. It comes out of the box with everything from JDK 11 on. JDK 8, we'll go over that. If you're running on JDK 8, you probably need to upgrade your JDK, but it does work in JDK 8 with a few more switches. There's several ways to interact with Java Flight Recorder. One of these is locally at startup. If you wanna pass command line parameters in when you start your Java app up, you can pass that in. JCMD is a local application that you can use, and it interacts via the PID versus the process ID on your system. Java Mission Control interacts through JMX or through your local PID if you're running Java Mission Control on your local box and then Cryostat. Hey, Jeremy, am I moving too fast? Am I doing okay? No, I think you're doing okay. One thing I wanna drill into too is this is really meant to be run all the time, right? Because profiling tools are kind of notorious for killing performance, right? That's exactly right. And this is where we, it's a perfect intro to this. The default configuration, if you start your JFR continuous recording, like Jeremy said, it should always be on. It's a less than a 1% performance set, and it's meant to always be on in the background. Now, if you're doing profiling and say you're looking for a problem, you can run with the profile configuration, and that's less than a 2% performance hit. And custom configuration can be a bit dicey, but the aim is that if you have everything turned on in your custom configuration, it's gonna be less than a 5% performance hit. I really have never had to use the custom other than when configuring some of my custom events. And we'll talk about that in a few minutes. This right here is the default configuration, and you will find this in under your JDK, there's a lib directory, and in the lib directory, you will find this JFR file, and this is for JDK 11. And really what it defines is each one of the events that it's going to be monitoring. So take the one in the middle, Java threat statistics, it's going to be looking at the threat statistics and admitting an event every 1,000 milliseconds. So every second, it's gonna be emitting an event for Java thread stats. So all of these are events that are in the JDK. You can add your own events, we'll look at that a bit later. One of the things that I wanna talk about down at the very bottom, where we're looking at JDK threat sleep, you'll notice the synchronization threshold is 20 milliseconds. If you go to, and this is the default configuration, if you go to the profile configuration, you'll notice it's changed to 10 milliseconds. So really what the profile does is the profile configuration is more overhead, but it's meant to take in more events and sooner. Now, okay, with the vets and sampling, I always get this question. Well, I'm gonna miss events. You are. You can't grab every single event. If you did, that would be more like some of the profile tools that you're using, and it would be a lot of overhead. So yes, you will miss some events, but the idea is that you're gonna grab enough events that you'll be able to see what's going on. So this is the normal continuous monitoring for JDK8. Some weird things in here that you don't need after JDK8 are unlocked diagnostic VM options, debug non-safe points, unlock commercial features, initially Sun, I'm sorry, Oracle was going to charge for this feature. And this was a commercial feature that was gonna be put in the Oracle JDK, and it eventually became just part of the JDK, and it was contributed by Oracle back to the open JDK. So... It does go back to Sun though, doesn't it? It does go back to Sun. It sure does. Yes, it's been around. And so this was, this is what you need in JDK8, and then you have your flight recorder options down there. As we move through JDK11 and 17 and 21, what you've found is this has become more baked into the JDK. This is not a commercial feature. You don't have to pay for it anymore. It's just part of the JDK. So these are, this is what you would pass in on your command line when you're gonna start your Java application. You might wanna give it a delay. You don't wanna sample your startup. Your startup is gonna populate caches and it's gonna throw off what you're looking at. So you might wanna delay until your app starts up. This will run for 10 minutes and then it will shut down. And the name of this one is gonna be default. And the settings for this, I told you there's a default and a profile. That settings, you can set whether, which one it is, default or profile. I could have left it off. It really is, default is the default. And then I have the following. So Jay, I can edit those templates? Yes, yes you can. Matter of fact, you can not only edit those templates, but you can copy them and make them your own. And you can change, you know, you can add things, you can remove things. So yes, you can run with your own template. And that goes back to the files we saw earlier. You just have to edit that XML and you should be good to go. All right, so for JDK and 11, really this is all you need for continuous monitoring. You pass this in on your command line. You pass in a file name, start flight recording. And this is what's important, dump on exit equals true. If you don't put dump on exit equals true, if you're, say you end up with an out of memory issue, you will not have a flight recording. So dump on exit equals true is important. But this is really all you need for JDK 11 and above. So you can start multiple flight recorders at the same time. And I'm gonna flip over and I am going to, let me go ahead in this and let me make sure I got everything package. And all right, and now I'm gonna start two as soon as this finishes. I'm gonna go ahead and start two flight recorders on the same JVM. You know what, that actually may be, you guys might need to see that a little easier. Is that a little better, Jeremy? Rob? I can see that fine. So basically what I have here is I'm starting two flight recorders and I'm starting the Quarkus application. Okay, and you notice it says starting recording number one using max size of the default. And it tells me use JCMD and this is the PID that I have for this Java application just started up. And just so you know, it started up. Local host 8080, yep, and yep, there we go. So we are starting. And now what I can do is I can flip over, but let me go ahead and advertise JCMD for a second. If you've never used JCMD, JCMD has a lot of uses. It is, it should be your Swiss Army knife in Java. It allows you to print threads. It allows you to get heat dump, heat information. It allows you to run your GC. If you wanna do a heat dump, it allows you to do a heat dump. This tool only works locally. And one of the neat things that I like about it, and let me flip back over, is when I run JCMD, it gives me, so here's one, here's my Quarkus application. It gives me all of the, there we go. That gives me all the PIDs. I will tell you, one of the things that I have a hard time with is if I'm running in a container inside Kubernetes and I open up a terminal, like my PS is gone, I can look at slash proc. There's a couple of things that you can do that are really kind of weird, but if you have a JDK installed, you can run JCMD and it will give you all of the PIDs for each one of the Java applications. And this is the one that we're gonna be looking at right here, which is my Quarkus application. I will go ahead and start to do some JCMD. So I can do JCMD, a Java flight recorder, and I can do, oh, forgot my PID. This is my PID. Okay, and what the JCMD.JFR check does, it tells me that I have two recordings that are running. And if I want it so I can tell you that I have both the continuous and the profile are running at the same time. And if I want to, what I can do is I can do a JFR dump and I can, so now I have, I'm doing the JFR dump command and I'm dumping the profiler and I'm gonna dump it into my recordings directory. So now I have it in my recordings directory and there is my custom dump right here. So what you can do is you can actually, you can start, you can start it as well. If I wanted to start a new one, what I could say is I could come back up here and say JFR start settings equal profile delay equals 10 milliseconds duration equals 10. I'll call it another profiler and I'll say reporting, new reporting. So now if I come back up here and I check, you see that I have another profiler and it is delayed right now because it's still in its 10 second delay should be running now. So that is, that's how you use JCMD. And flip back over to, so I've got all that stuff here in my presentation. You can see that I just copied this out, pasted this in and JCMD works. Any questions so far? Right on. Doesn't, yeah, we're good to go, no questions so far. All right, so the next thing I'm gonna go over is, this is really useful is creating an event, creating a stock events in your Java application that you're gonna be able to see in any of these tools. Now, has anybody ever done the start millies, finish millies trying to profile a function in Java? I'm sure everybody's done it somewhere and you're spitting out all this stuff in your logs. It's a real pain. What this does is this will take the place of all of that. There is method profiling and it does method sampling inside of the Java flight report, but maybe you have a rest call that's taking a long time, you wanna pass things in, you want to be able to see what's going on, you wanna be able to do some debugging with it, it's typically been a problem, you can create an event. And that's really what this looks like is, you just extend JFR events and you put, I've got these three variables in here, path key and result that we are going to fill out. So I'm gonna flip back over and show you what that looks like if I can grab it, there we go. Sorry about that. So this is my greeting resource right here. This is what was created, this is the hello method. And if you notice that I have instantiated a rest call event and then I've said begin and commit, I haven't really done a lot with it. And like I said, this is what the rest call event looks like. So you added a actual, can you go back to that other code for a minute? So you added a memory leak thread. Oh, further down, yes. Because I saw it at the variable at the top. So I mean, I can do that without having to add anything special. So that's interesting. So you do that so you could catch it? Yeah, so what I wanted to do was, I do have a memory leak thread that will create a memory leak and the reason I put it as a static was so that I could get a hold of it and stop it on another method, right? But pretty much I wanted to do this so you can see the memory leaking inside of the Java flight recorder. So yeah, I didn't realize how alarming that might be. It's like, oh, memory leak. No, I mean, I can create memory leaks without having to add anything special. So it's just interesting. It's cool that you added that so you could observe what it actually looks like. Yeah, what I wanted to do was I wanted to add this so I could start and stop the memory leak because I eventually when I end up with an out of memory error, everything becomes flight recorder and everything. Flight recorder will print out, but anything that I'm looking at from a JMX concept kind of goes belly up at that point, right? But flight recorder will continue after you had an out of memory error. It will finish up and it may not log things exactly correctly after you hit the out of memory error, but it will finish up the file. And as it exits, it will finish writing the file. You're not gonna end up with an open file. So yeah, so I did a begin and in the commit, let me go ahead and look at some of this. I put in, this is a hello delay. I basically waited five seconds and I can't really wait much longer than this because I am using rest easy reactive. Reactive will kill any blocking threads that it sees as blocking. So I have to be careful with this or else rest easy reactive will kill it. So I'll go ahead and wait five seconds here. I just wanna be able to make sure that I can see it. And then this is the one that takes an argument. This is where I add the key, the path and the result so we can see it locked. So I've got three different methods that we're gonna be looking at. Let me go ahead since I still already have, since I wanna make sure I have it. Yep, so that's still running. Let me go ahead and hit a couple of those methods that are up there. So hit the hello. There's me, let me say Rob. And let me hit Jeremy. And then let me do the hello delay. So you'll see it takes a good five seconds to roll through and it'll eventually come through. So all right, so what we can do now is I can now look at those events. What I'm gonna do is I'm gonna go ahead and I'm gonna launch. Let me catch up with my slide presentation. All right, so adding custom events. So one more thing with custom events that I have not gotten down here. You can add a periodic event. And the way you add a periodic event is you call flight recorder. It is a static method for add periodic event. And you pass in a runnable with a class. This is actually, this function is being called inside of a cache class. So I just wanted you to know, I'm not gonna demo this today, but I wanted you to know that you can also control periodic events. If you do have a cache and you wanna see what the cache size is. Or if you wanna see take periodic DB connections, like how much of my DB connections. So if you want to do any of that, you can add periodic events. It's a little more detailed, but it's just basically passing in a runnable to collect events. What are some specific use cases like where would you, how would you determine which one you would wanna do when? So I would want to, if I have anything, like everybody has those pieces of code that are written by that guy, right? You don't wanna be that guy, right? And so if you have, it's a nasty bunch of code. It's complex and nobody wants to really mess with it, right? I would probably make sure that I had JFR beginning to commit on like either side of that to measure the performance, especially if it's just a nasty, nasty, nasty piece of code. For your periodic events, I would go through and I would look at anything that's holding resources. And I would do this for pools. I would do this for any resources that you think are gonna become constrained, even if you don't think they're gonna become constrained. It's really nice when everything falls apart and you're searching for that reason why. Everything is gonna be a good clue for what fell apart. I will say we had a... So what do you type in like say connection pools? Is that what you meant? Connection pools, object pools. So any kind of pooling mechanism you're gonna do, I mean, I would even recommend if you're looking at this, you can even add it to EJBs if you're in JEE, right? And you want a certain amount of pools. You can add it to just about anything that you can get access to. And so I do think that it's something that you wanna go through and add. I wanna give a shout out. There should be somebody on your team who is at least doing performance monitoring and looking at performance. Those are the people in your code review that are making sure that everything's running properly and that person should be adding these events in there. And I'm gonna say that you probably don't need to add it to every single method, but as many methods as you can add it to is great because like I said, when things fall apart, everything's gonna be a breadcrumb trail back to why things fall apart. So what I'm gonna do now is I am going to flip back over here and I'm gonna hit my first tool. I'll make sure I'm up on my... Okay, so one last thing before we go into tooling. JFR is a utility that can help you pull. So what I'm gonna do is I'm gonna execute this rest easy. Hopefully that I do a dump with my custom dump. Let me look to make sure my custom dump. Yes, I did. So what I'm gonna do is I'm gonna go ahead and execute and what this is gonna spit out is, this is gonna spit out. So right here I have a rest call event and really what I did was if you look on the command line, JFR print, I got the event. You can do this for any event that you wanna look at for any dump and what I've done is I've pushed this out right here. This is the rest call event. You can see I don't, this is the hello. You can tell because it's executing the hello method. It doesn't have a path, doesn't have a key, doesn't have a result. You can tell what thread it is, when it started and what the duration is. So this is important. If you wanna dump things to JSON and then parse them out later, this is a really important method to do that. I find that very few folks know about JFR, the JFR print stuff, and it's really useful. So when you do it, you can spit things out and you can parse them with some other parser. I'll make sure I'm not getting ahead of myself here. What are some other parsers like that you might look at? So some of the other parts, a visual VM, will parse that out and so will Java mission control. And we'll go through and we'll look at those. I do get a lot of folks that poke me about logging and they say, well, why not use logging? So logging is gonna take up a lot of space. This is low overhead. You can set thresholds. If you wanna set a threshold for a value, say you're reading a value into your event and you say, well, look, I only want this value to be five or more, right? I only wanna log my cache when I get down to three instances in my cache, right? And I'm gonna have to do something. So you can set thresholds. This also has concurrent session management. You can have multiple flight reporters running at the same time. And this is a question that I think I got the last time I did this session was if I have 1% overhead for the default and I have 2% overhead for the profiling and let's say I've set up a custom one and it's at 5%. Are they all additive? And they aren't. And the reason I say that is you're gonna have a little bit of overhead but it's gonna be whatever your top one is say if you're doing the profiling and the default together it's gonna be a max of 2% because it really is, it's emitting the events. And most of the same events are gonna be emitted for the default that are already being emitted for the profile. The profile is just adding a few more events, right? So you are not gonna incur the more JFRs that you add. You will incur a little bit for the file IO but for the most part you're not gonna incur a large overhead for adding extra flight reporters. Does that make sense? So I'm under the impression that there's a JMX agent and a cryostat agent? Yes. Okay, is that matter when, cause you brought up using J console and some of the other stuff. Yes, so let me go ahead and hit, let me go ahead and hit Java mission control. And yeah, let me hit that first and that'll answer some of those questions. So Java mission control, this is free Sun did contribute it. And you'll see it come up. This is the latest version 8.3. I still have applications that are running and it would help if I put this in the right window. So this comes up and if I wanna go into get rid of this. So right now this is actually looking on my system and it tells me that I have a Quarkis running and it basically does what JCMD does is it looks for the PIDs on the local machine. Now, right now I do have three two flight reporters running and I do not have an M beam server running. The cool part is if you're locally on the console, you can start the JMX console without having to add command line parameters. And you can see here's the JMX console, current JVM heat usage. And you can also get down into your M beam browser and you can start to look at all the M beams that you have in there. You can also, so let me kill that. Let me go ahead and I'm gonna dump this whole recording might take a second. All right, so this I dumped the whole recording and it makes a temporary file and this is what comes up. It's a really great dashboard that comes up and you'll see things right now I have everything that is in green, uncheck this and I have 76 thrown errors. Let me, okay, cannot increase the font on that. Can you guys see that okay? It's kind of hard to see. It's kind of hard to see. All right, that's all right. Okay, so basically what this will tell you is it says there's 61 errors per minute and the largest amount of errors is no such method error. This also gives me a warning that I have competing CPU ratio usage and basically it tells me that it was caused by other processes running. So this gives you really great feedback. Like I said, everything else on here, if you have GC issues, you're gonna see red on the GC and you can look down through, you can look down through, here's all the threads that are running. Here is the event browser. This is every single event that has been emitted and recorded during this flight recorder. Let's see. So what I can do is I can search for my rest of it and I can see that there are, these are the ones that have been executed. You can tell my labels key past in URL path output and you can see that Jay, Rob and Jeremy got passed in for a key. Here were the URL and here were the output. This is the custom fields from my event. So that's, so if you wanna monitor your events, you do, you have to come down to the event browser and you have to look at your own events or you have to parse them out like we did earlier. So if I wanted a custom event for like around an algorithm or something that I'm working on, I'd have to make a custom event and then wrap that, right? Yes, you would have to make a, you would have to make a custom event like this. And then you would have to wrap whatever function that you want in a begin and a commit. Okay, okay. So here is the, here's the one for, here's the one for the, which is the asynchronous one. So really what this is is I have a static method that enable stats recording that takes a cache name and basically the cache, which is a map of maps maybe. And then what you do is you come down here and you add a periodic event to flight recorder and pass the run in. So it's a little more complicated, but this is the cache name. You can pass all kinds of other stats in. So, and you notice they do a begin and a commit. What you're gonna find is that the timings on this don't matter as much because it's more of a, this is what the cache size is versus this is how long the method took to execute. Visual VM, I will, visual VM does- Jay, one question. So if you don't have flight recorder turned on, what happens with those events? What does it do? Nothing. It does nothing? No, if you don't have flight recorder turned on, it's essentially a no-up. Obviously you're going to, you're gonna have the method call, but since flight recorder's not turned on, it doesn't emit the event and there's no subscribers to those events. Okay, perfect. So, anybody hasn't seen visual VM? This is visual VM. And of course it's gonna start up in the wrong. Okay, this is visual VM and you can do the same thing. You can come down here and look at, you can hit remote. I can also load. If I wanna go ahead, I won't do it, but let me see if there's a, I'll have a recent one on here. So, this is what it looks like. Overview, here was the monitor, file IO, socket IO, exceptions. Here is the browser for all events. And this is uncategorized. These are the events that we looked at earlier with a couple other invocations. You can add an annotation, which is at category and you can add category. So, if you wanted to, right now, here's the category's flight reporter, Java virtual machine. These are uncategorized because I didn't categorize them, but if you add a category, like I added a label before, if you add a category, you'll see that underneath here. So, if you get to the point where there's a lot of events and you're omitting them, you definitely wanna look at categorization and making sure your events are nameable. Okay, so I'm gonna go ahead and switch over and deploy these on OpenShift. I didn't wanna do anything that underneath the covers. So, really all I've done is I've added the Quarkus, the Quarkus OpenShift plugin, which is really a wrapper for the Kubernetes plugin. And what I've done is I added this to my project file, which I've said the build strategy is gonna be Docker. Trust search equals true. And I know I wanna a route. So, I wanna make sure that I have a route and that I can hit it via HTTPS because my browser always wants to do HTTPS now. So, that's what these do, it exposes a route and does this. I'll go ahead and execute this. That's an interesting line of code there with the trust search. So, is there security around Flight Recorder? So, there is security around it if you're hitting Flight Recorder through JMX. So, you can, if you're gonna open JMX, you can use a cert or different credentials to hit JMX. So, when we, I basically run with, I run with knives all the time and this is running with knives because I basically trust everything in this environment. But you shouldn't do that. Whenever you open JMX, you should use certificates and make sure that nobody else is, you can do a lot with JMX on your JVM. So, make sure you secure it. So, great point out. And so, why, I had a question. So, why did you choose OpenShift for this demo? Is that, exactly, it's the best Kubernetes distro? So, I used OpenShift because it's a Red Hat product. You can use this for, you can do any Kubernetes for this. This operator works underneath any Kubernetes. Cryostat is amazing. Okay, so Cryostat is changing a lot. I noticed I gave a demo last week and between Monday and Wednesday, I gave two different demos and the UI had changed like significantly, like in two days. So, Cryostat is really being developed quickly. So, let me go ahead and... It's a little curve ball for your demo there. No, no, that's fine. But basically what I've said is, I've said, while this is doing this, let me look. So, what this is doing is, this is using a Dockerfile and all of this comes stock with Quarkis, right? Quarkis already does this Dockerfile for you. What I've done with this is, normally you have this parameter, which specifies the log manager and the host. What I've done is, I've also added management, JMX remote on port 9091. And I've also said JMX remote SSL equals false and authenticate equals false. This is me running with knives again, right? And you should not ever use JMX without locking it down. So, this should be true and this should be true, but for the demo, I went ahead and just opened all the doors. Hopefully nobody hacks me while I'm doing the demo. All right, so, let me flip back over here. We have a quick question. Jay, a question from Chris. Is there a way to use Flight Recorder with COTS apps that you do not have the source code for? And similarly, is it possible to use Flight Recorder with Red Hat AMQ? Yes and yes. So, any Java application that you're running, I do have an application that I've been playing around with and let me see if I can file open recent and yeah. So, I've got this application called Buggy app and really I downloaded Buggy because I was looking for something that was gonna kind of kill my browser. I'm sorry, that was gonna kill the thing and I can show you Buggy. Buggy's actually a really neat app but I didn't have the source code. All I had was a war and a jar and so I went ahead and put together a Docker file and all I did was I am using the OpenJDK UBI8 from Red Hat, but in the command, all you have to do when you're launching the Java application is add your ports in here and authenticate equals false and then here's the Java jar and here's the web app runner and I'm on port 9010 for the buggyapp.war file, right? So yes, you can do this with anything that's generating a jar file. All you have to do is add JMX to it. What you can also do and I don't have it in here because I was gonna roll along on my demo is there is a cryostat agent and the cryostat agent is a secondary jar and the neat thing about the cryostat agent is it allows you to inject events into your code live. Look for that in part two but so right now what I'm doing is I'm using the, I'm using JMX and I'll show you why here in a few minutes but yes, you absolutely can use it for AMQ as well and all you have to do is when you launch your broker add this to it and you'll be able to get all of the stock events that are in the JDK. If you wanted to add, if you wanted to insert stuff you could do it with the cryostat agent if you're inside Kubernetes. Does that answer your question? Yeah, that's gonna be good. And guys, if you have follow-up questions or need some more clarification feel free to throw those in the chat. All right, so we should. Yep, so here's my deployment that's hopefully finished. So we do have another question. If you're not exposing the service say if you were using the Red Hat API management can you get to the JMX via port forwarding and what settings might need changing? Ah, you can get to it through port forwarding. I'll be honest with you with API management I'm not really sure. I will post that back. I'll do some research and post that back but yes, you can do port forwarding on this but you do run into problems once you secure it because you have to forward the certificate along, right? If you're gonna do certificates. So it security becomes a problem with the port forwarding but yes, you can do it with port forwarding as long as you're running with nuts. So here is my application. Let me go ahead and take a look at it. Flipping over to the developer topology. Here is my application, same application but and now I'm running it inside of OpenShift. What I'm gonna do, I'm gonna flip back over and I'm gonna go ahead. I have already installed the cryostat operator because it's really uninteresting. You go into developer hub, you basically click on this and then you click on install and it just takes a little while to do it. So when you go into... Really quickly and just in case you're not familiar like with developer hub and operators, just really quickly, so developer hub is a list of or I'm sorry, operator hub. Operator hub is a list of operators we've certified that would work on top of OpenShift, right? We can come through there and search for all sorts of stuff, right? Yep, there's things like apicurio, app dynamics, cloud operator, aqua security, you can see there's a bunch of AWS ones in here. So if there's an operator out there, we've combed the world for it and we've done some testing on it and we've worked with our partners to make a really robust operator architecture. So cryostat's just one of them and here's my build of cryostat. It does require me to create a cryostat server and here's the cryostat server. I'm just gonna call it cryostat greeting and I'm gonna take all the defaults down here. And yes, there are trusted TLS certificates you can add down here so that you can actually hit your JMX. You can hit your JMX, you can hit your workloads running JMX. Right now, my pod, I'll look at that right now. Right now my pod, which is code with Quarkus is running on port 80. Interesting. So what I really need to do is I really need to open up port 9091 as well so that cryostat can see it. Cryostat will see anything with a named service called JFR-JMX or running a port 9091. You can also add them manually, but cryostat will automatically pick them up. Let's see where we're at with our install. Yeah, no status yet. Taking a while here to install, huh? It is. Let me kill it. So what you'll also notice here, and I won't change anything, what you'll also notice is that I created a cryostat inside of the namespace creating. I could create a cluster cryostat, and that works throughout the entire, that works throughout the entire cluster. The reason I do this is because it does search through all the services, and it's easier to have people at this namespace, like the greeting namespace, like maybe you're a developer and you are working in this namespace. And then you'll think it's not. Seriously, huh? It's not, you probably don't look like a dedicated cluster either, it's a shared cluster, I'm sure. Yeah, it's a shared cluster. I will say, let me, I'll tell you what, let me... If anything goes wrong in the demo, Jay, it's okay because it's just Jeremy and I rubbing off on you, so. So. Ha, ha, ha, ha. All right, so you guys do get to see me, you guys do get to see me create, let me uninstall the operator, and you guys do get to see me create the operator again. I've not had it take that long before. So, but, so what I'll do is, I'll go, I'll come down here and I'm gonna go ahead and make this while we're waiting for that to install. I'm gonna edit the service and I'm gonna come down here and with some YAML magic, I'm going to add a service called JFRJMXTCP. I'm gonna add 90, 91. I think I have it on 90, 91. I'm gonna go ahead and save it, reload it. And so now I've exposed port 90, 91. Let's go back over here. There we go. All right, now this is where we were tripped up at. All right. Greeting namespace, yep. Should at least start to show a status. There is one other thing that you have to install and I completely forgot about it. The Red Hat Cert Manager for OpenShift. So, sorry about that. Yeah, no worries, that makes perfect sense. So let me come over here and let me delete that instance. That does make sense why it didn't install. Okay, so now I see we've installed Cert Manager for OpenShift and we'll go back here and create cryostat and hopefully, okay. At least it is. Got it, all right, so still pending. It is still pending. And leading up to this, I can't believe I'm not gonna be able to, there we go, okay. Now it's progressing, finally. And then it'll come through and when it comes through, I'm gonna get a, not yet, main deployment processing. I knew I should have set this up before we started but I wanted everybody to be able to see this. See, I mean, yeah. And at least I have my service on 9091 that's already there. And you see cryostat also has its service, is also set for 90. It monitors itself, it's a Java application as well. So, all right, so all of that stuff looks great. And we're not there yet. I would, you know, I close my window. Okay, just to let you know, I am doing this on Arrow and so it's trailing up in the cloud. The Arrow is Azure Red Hat OpenShift in case you're not familiar with our internal slang shorthand. All right, finally, we get to cryostat. And it's gonna make me log in and copy my super secret password on there. Is it password with the add symbol to make it safe? It is, it has to allow permissions and then cryostat comes up. We're gonna skip the tour. Okay, you'll see I have my code with Quarkis here that this is my application that I've deployed and you can start to see that it's looking at CPU load, heat dump. I don't really have any recordings for it yet. If I wanna go ahead and create a recording, I can give it a template, say, let's say, tests, duration 30, I'm gonna choose the continuous and I'm gonna go ahead and create it. Archive on stop, yep, and I'll just make it continuous. So now what I have is I have this right here. It's doing live analysis. You can see anything that shows up, this is what changed on me in the middle of the week. This will actually, you'll see things highlight in red. This is live. If I wanna take a snapshot, I can download the recording or what I can do is I can come through and I can say I wanna archive this, take a snapshot of it and here is the archive. So same information, it's just not live. This was a snapshot that was taken. And now do you remember, do you recommend leading this running full-time too? Do you mention it's okay? No? Or yes? Yeah, yeah, so I do recommend leaving it running full-time. Matter of fact, what I recommend doing is coming in here into automated rules and what I like to do is these automated rules will automatically start a recording when you have certain pods that crank up. So I'm gonna say all 90, 90, all 90, 91 pods. So I could watch on Kubernetes labels and annotations in there too, right? That's exactly right. That's exactly right. So all right, so and then I put my description down here and I'm gonna go ahead and copy this from in here. So basically there's an annotation for cryostat port that's in there. I'm gonna go ahead and paste it and what you'll see is that the pods that are in here will highlight if they're enabled. So really what I did was I said, okay, fine, let me go ahead and highlight anything that has an annotation that has cryostat port of 90, 91 and I could say enabled, select the template, continuous. Maximum size, maximum age, I can do archival period so you can archive every 24 hours if you want to and then I'm gonna go ahead and create it and what you're gonna see is 90, 91 pods was created and you're gonna start to see that if I go into recordings you're gonna see this is my original test pod that was running, now I have an auto pod running. If I go in and really hope this works, if I go into my developer on this and I say I want to increment the number of pods so now I have two Java processes running and I'll say three Java processes. All right, so scaling the three, if I come back over to finish scaling up so now I have three Java processes running and you'll see that it actually discovered those other targets and it started a continuous running for the auto 90, 91 pods. This is really important because if you have an auto scaler you need to make sure that your pods are tagged in a way that cryostat can automatically start a continuous recording on. There is one more thing that you can do, you can view this in Grafana and Grafana, I do have a, I have to actually get the password in order to view stuff in Grafana because Grafana comes with cryostat and what I will do is come down here and there's my admin password for Grafana. Okay, admin on my password managers. So what you can say is this recording duration was 41 seconds, you can see the CPU spike. Honestly, this works so much better if you come to like your dashboard and you're looking at these, let me go ahead and pick one of these. What I can do is if I come down here to recording, if I had a longer one, I could view any of those recordings in Grafana, this one's running for two minutes, it's a little longer, but this is a live recording since it's a continuous recording and it will give you live stats. So if this were running for a while, I'd see CPU load and physical memory and stuff running in Grafana for those Java applications. All right, I see I'm three minutes over. No, that's perfect, any questions? Any more questions? I think that's great. What about this at scale, Jay? So what happens when I'm now, I have hundreds of containers running and on a single node, not let alone in my cluster. Yes, so the first thing I would recommend is that so when you do this, so inside of OpenShift and Kubernetes, you can do a persistent volume claim and when you specify a recording, when you create a recording, you can come down here and say, you can come down here and you can add a persistent volume claim when you snapshot your recordings. And so what I would make sure you do is, right now every pod comes with temp space. So all of these that I'm, if I do a download recording and it downloads it here, that's local space. If I say view in Grafana or if I archive something, all of these are going to be in temp space. When I down this cryostat pod, they're all gonna be gone. But you can add into when you do the operator you can add a persistent volume claim. So all of your snapshots get saved in the same place. I would definitely organize your cloud storage that way. So if you have say 47 different pods and they've each got 10 instances that they are each moving in a different place. And this is another reason why you would install cryostat when you look at your operators. This is another reason why I would make sure that I install a cryostat per each namespace because this cryostat sample, this is gonna have your persistent volume claim in it. So it is important that if you're say, everything from greeting goes to one place, everything from say your Moz app goes to a different place. So it's important that you set these up separately so you have a different place to store these. If you wanna store them in directories, you can do that as well. So we talked a bunch about JMX and you talked a bunch about things that actually require introspection. So what about when I'm using Grail VM or Mandrel to native compile? Ah, that is, that's amazing. Yeah, so if you're using, if you're using Mandrel is not as compatible as Grail is, but you can do it and the way that you would do it is. So, okay. You said I was amazing. So I just tried to record that and call my wife in here, but no, I'm sure. Right? So right here, I'm actually gonna go ahead and I'm gonna go ahead. This is, I'm packaging this and I'm telling it, it's native and you have to pass this in here. Minus D, a quarkus native monitoring equals JFR. If you don't pass that in, you don't get native monitoring, but this additional build argument, the signal handler based execution sampler is another switch that you should pass in. It changes the samplers from the simple sampler to the execution sampler, but let me go ahead and kick this off. And when I go ahead and do this, what the way I would run it is, it might take a little while, but you run it the same way when you run your native executable. You do have to pass in the JFC. That's, remember we talked about the default or the profile configuration and it's an XML file. You have to pass that in by itself because there is no default that you end up with for Grawl. So you have to actually pass in the configuration. Hopefully this will finish here in a second. And so I'm actually gonna show you my cheat sheet so we can talk about it while we're doing it. So really this is my target right here. I wish I could actually, so you can't see that very well. We're still building, but basically what I'm doing is I'm actually running the runner and I'm passing in the XX start flight recording. And the only thing that's really different is that I had to pass under settings, I had to pass in my configuration, which is my JFC. And I'm using the JDK21J configuration for that. So close, not there yet. So still running my native compile, but I will. While we're waiting for the compile, can you dump any, where would I go to find, if I wanted to pull this down and play with it right now, where would I go? If you wanted to pull cryostat and start playing with it locally. Or Java flight recorder, sorry. Cryostat IO is the place. And can you guys see that or can Jeremy just see that? Got it, you got it. Got it. So that's cryostat.io. You can pull down. So flight recorder is built into the JDK. And it's in the JDK documentation. I do have, here are a couple good, there's a couple good links that I have out in my slide deck that are down here at the bottom, somewhere, I threw them down here somewhere. But yeah, Java flight recorder built into the JDK. Here's the real documentation on how to use that. And it's access.redhats.com. And so if you look at any of the JDK, Oracle's JDK documentation, look at, this just happens to be Red Hats, but it has all the information about Java flight recorder. And there are a ton. It's been around for a while. So there are a ton of samples that are out there. It's beautiful. We mentioned earlier that it's been around since sun. Was it around from prior to sun? Was it earlier than sun? Or it was just sun and created? I mean, I mean, I mean, sun, I don't, I think JDK-18 was the first time it was around. So sun, yes. Yeah, before Oracle, but I definitely think it was, it was around in eight. It, and they've really done a lot with it since eight. So if I want to execute this natively, you guys can still see. Basically, I'm passing in my target, which is my native executable. And I'm just passing in exec start flight reporting. Here's my settings, which is my default JFC. And so it started. The problem that you run into with a native compile, you don't have JCMD. You don't have access to, you have to, you can either shut it down, or you can, this is a duration of 60 seconds, or you can do this. It is not easy to get to yet. Mandrel and Grawl are both putting features in to allow Java flight recorder to work in there. So most things do work. I will say some of the things are limited. I would like a utility like JCMD to be able to dump these without having to wait for the timeout or without having to kill my application. Oh, and by the way, so J-Ri goes back to the J-Rocket JVM from BEA. Oh, it does. It goes back to J-Rocket. Okay, that's why, you're right. So this came out of J-Rocket. Nice. Cool. Anything you want to close out with, Jay, that you want to parting words of wisdom? I mean, he says the fact that we both want Jeremy's hat, because... You're right. Yeah. This bell is seriously annoying. Yeah. But at least birds know that I'm coming. I should have worn that last night because I broke into the Christmas cookies already, so. My last parting words would say, I would say, look, add Java flight recorder to all of your, whether you're running in Kubernetes, whether you're running in just bare metal, whether you're running in vert, whatever you're running in, add this to your Java application. You never know when it's going to crash, when you're going to have problems. It's a very minimal overhead, and you can still use other tools with it. Just because you're using Java flight recorder doesn't mean you're not going to use other profiling tools, but this should be your stop gap for anything bad that happens. You should be able to go back to this and figure out at least some breadcrumb trail about why it happened. Well, thank you very much, Jay. Great presentation. Thank you very much. Great. I learned a lot there, and based on how we're arranged on the screen there, you're a rose between two thorns. So we appreciate you coming this month, and we'd love to have you back. All right, great. Thank you guys so much. I appreciate it. All right, cool. All right, thanks, Jeremy.