 What I'm going to tell you since time is very short is an anecdote. It's not my usual talk about this subject I thought I would do it a bit differently this time. I'm at a guy on IRC I don't actually know his name as you know IRC is mostly anonymous, so I'll call him Martin and Martin is a system administrator who has like what a dozen maybe 20 Tomcat servers in production and And we got to talk as you do and because he had a problem that Every once in a while he would get an SMS message that one of his systems had died And it's funny how these things never happened during the day, but you're always at 4 a.m. In the morning If you're any system administrators here, I'm sure you're well aware of this Season nodding here So being a system administrator for Martin The the Java project applications. He was running weren't very visible. He was in the process No, he was in the process of rolling out Zabix Zabix is a monitoring tool Actually, we have the product manager of Zabix sitting right there. I like see if you could raise your hand, please Zabix is a monitoring tool much like the ones you use like HP open view mu-nin cricket So it's one of those I like Zabix because it's very accessible even for developers such as me not just to system administrators Martin was already rolling out Zabix for all of his servers But he had not yet found a way to look into the Java processes that he was managing As you can see in this, this is the normal stack that you that you meet when you have a Java process You have the hardware in the operating system with the JVM on top with Tomcat in this case on top and the actual Application on top of that and he could look into the hardware and the operating systems, but not into the rest So to him Java is essentially a black box and frankly, I think that's the good way to do it I mean system administrators shouldn't have to learn all of Java But you do want to have A few specific holes in it that you can look into the JVM and see things that as a system administrator you need You don't want to learn all of Java just about threads and memory and things that help you do your job The system is very slow as I spoke to Martin. I noticed that the world of system administration Sorry wrong slide system administration and Java development was very very much a part They're really different worlds and the worlds with different tool sets as a Java developer I'm used to using J console JMX log for J those tools. That's what I do my work with But when I spoke to Martin, he knew almost none of them Even though he was a system administrator running several Tomcats. He was used to using in his case Sabix he was used to you a monitoring through as an MP and Writing scripts to read through the log files And again, I still I'm not saying this is a bad thing this separation In fact, I think if your company is over five people, you should have this separation You should have these two worlds visible and people living each on in a separate world But what you need is that there is a bridge between the two worlds both at a technical level It's me. Sorry both at a technical level and at a personal level and that's my open source project Subcat does just that it links these two worlds Technically and I hope also on a personal level Technically it ties into JMX on the side on the Java side and it's a little web application It comes as a war file that you just drop into J boss or OSA for J or maybe Tomcat and On the other side it speaks to Zabic protocol So to system administrators. It's nothing more than just another Zabic agent And on top of that sits JMX or the syntax of JMX allowing system administrators and Developers to speak about objects that are present in JMX speak about memory speak about garbage collection So this is the solution that Martin used for his system There you go Now he can look into the rest of his systems into the Tomcats and into the JVM's Let's make this concrete. I'll make it so concrete. This is what Martin actually did probably I mean, I've never met him, but What you do this is a configuration screen in Zabic's for those of you who use Zabic's you know this This is where you make a new host Java programs will show up as hosts in Zabic's as though they were real machines Not just virtual machines. You give it a name Tomcat this is the name of my web server. This is actually the configuration of my web server at home You give it the IP address and the port number that the agent is listening at and you apply a number of templates to number three in the slide and These templates help system administrators filter out of all the JMX enabled information To get just through information they need or at least Some of the information they need to do their jobs. For example, there's the Java template, which tells you about garbage collection and memory There's the hibernate template that tells you about session leakage in hibernate So you can graph that in Zabic's and there's the Tomcat template which does things like HTTP sessions and shows you about worker threads on the various ports Let's have a look how that looks For example, this is one month of information of my web server again as you can see it's pretty much idling The green line is the amount of memory used The red line at the top is the maximum memory and as you can see you see the familiar sawtooth At least familiar to Java developers All right. This is an application that's stressed for memory And you can see the memory dragging up and down as the garbage collector are struggling for breath So if you see this as a system administrator, you know that some things are wrong Notice how this is no different from monitoring memory on a regular system on an operating system Suddenly it's accessible to both system administrators and Java developers And it allows you to talk about this. For example, if I were to see this You'd have to do two things as a system administrator. First give the application more memory At least for the time being second go speak to your developers and ask them if that memory should be used And maybe you can optimize a bit This is an image showing the garbage collectors garbage collectors always a bit magical in the Java world But if you study them, it's really very simple When the JVM starts it picks a strategy depending on a few parameters and it always picks two garbage collectors a Sloppy one that's cheap just runs quickly, but doesn't find all the garbage to collect and an expensive one That does find all the garbage that it can collect, but it takes a lot more CPU power to do so I've graphed the two here You see the green line being the cheap garbage collector and the red line at the bottom is the expensive garbage collection And again, this is my website. It's a month of data that you see here The gray bars are the weekends and you can see that Since this application does have enough memory It uses just the cheap garbage collector and doesn't spend as much time garbage collecting at it would otherwise The red garbage collector the expensive one is completely ignored on my system. It just everything does everything with the cheap one If you look at the same thing Foreign system that's stressed for memory. This is a little test application that I wrote just to produce this slide You can see that at some point Java just gives up on the green cheap garbage collector and start using the red one start using the expensive one And now you know that your system is firefighting You'll probably see in the memory. You see the the jagged line and You know that you need to do some work and you get an advanced warning that maybe this system is about to go down As Abix helps you with this what I've done in the template is put a trigger on the line where the green line and the red lines cross You get an email or an SMS message when these two lines cross and you know that you have work to do And hopefully it won't be in the middle of the night threads These is it. Hmm. I seem to move two slides every time This is a view of threads in Java threads are like processes So this is something again that system administrators and Java developers can talk about The red line in this case isn't the maximum number of threads because threads are essentially unbound in the JVM But it's the peak number of threads So gives you an idea how many you had at a maximum and the green is the number of threads currently running This is from the Java templates now if we look at a Tomcat template it gives you another view of the threads But it's specific to some cat There you go This is just the pool of Worker threads for port number 80 for the HTTP port In this case the red line is the maximum because it's a pool and it's a fixed-size pool and There's the green line at the bottom showing how many threads are actually working And as you can see not many people visit my site one of the reasons I'm here To get more people visit my site see this line go up So this allows you as a system administrator to look at the number of threads for an application, but also study How these threads are used and if the pools that are being assigned to the HTTP port are large enough and being used efficiently Let's get back to Martin Because that's why I'm here that that's what I was speaking about really This is an actual image that he put in a paste bin for me for those of you who can't see it's a bit dark, but At some point the number of threads. This is a number of threads on his application just jumps from what 50 up to 130 140 and It doesn't go down and this is over the weekend the gray bar is not very visible But you can see this is like Friday and here Sunday ends if we zoom in If we zoom in You can see that it coincides with the SMS message. He got that weekend And this is also where we need to see the need for a bridge between the two worlds Because the problem that he had was not solvable either by me or by him Only because we too got to talk about it and used Zabix and Zabcat to make the problem visible Where we're able to solve this problem The actual problem was that at this point in time The syslog demons were being restarted Now I don't know I didn't know whether or not they were using syslog But as you know log for Jake and also log to syslog, but he didn't know that so I asked him look Maybe does your system use syslog for logging and he went over to his developers asked a question and came back With yes, we were using syslog So then he could reproduce this problem on the test system But only after I suggested that maybe he could do that So here you see that only by bringing these two worlds together Were we able to track this problem? So if there's anything I wanted to take home from this talk I would like to to ask you to build bridges like the one I've done Preferably using Zabix and Zabcat of course. Thank you very much