 So our next speaker is Joe Kutner. He lives in one of those flyover states. And he's the author of Deploying with JRuby and a Committer to the TorqueBox app server. OK, app container, something like that. OK, so thank you, Joe. All right, so how many of you played with LEGOs as a kid? And how many of you play with them as an adult, too? All right. Well, I have an excuse. I have a son, and he's just reached the age where he's started to enjoy playing with LEGOs. So this is a very exciting time in my life, but it's also a very expensive time in my life because of stuff like this. This is a little LEGO motor. You hook it up to a battery pack. It spins, and then you build cool stuff out of it. So I bought my son a kit of these. And the first thing we built, of course, was a car. And like any red-blooded young boy, he wanted to build a second car so that we could race them. Is that warped? Is that stretched? Yeah, all right. Well, anyways, having two cars actually turned out to be kind of fun because we experimented with different gear ratios so that my car would always win. I'm very competitive. But eventually, my son wanted to build a third car. And we couldn't because we didn't have any more motors. Effectively, these cars were very expensive. Each new car was a complete copy of the other car, and eventually we ran out of parts. But my son, undeterred, had the idea that instead of cars, we would build a train in which a single engine acted as sort of a locomotive and pulled a bunch of lighter weight cars behind it. And it's allowed us to build more cars, carry more passengers, including the Statue of Liberty, and have more fun. But it occurred to me as we were doing this that the difference between the train and the individual cars was very similar to the difference between deployment architectures of JRuby applications and MRI-based web applications. And that difference is what allows both the train and the JRuby systems to scale better, perform better, and ultimately be more reliable. And that reliability is what helps you sleep at night. So yesterday, Charlie talked about JRuby. I'm not actually going to talk about JRuby today. Instead, I'm going to talk about Ruby on the JVM. Small difference, right? But I find it's a big one. Because when most folks see that letter J, their initial concern, maybe terror, is that they'll have to write Java code, or even worse, XML, to get their apps running on the JVM. But it's simply not true. In fact, you can write pure Ruby web applications and run them on the JVM. And all these features that JRuby provides, like the Java import, the Java namespace, they're cool stuff. And if you want to integrate with Java libraries, that's great. But they're completely optional. You don't have to use them. This is so true, in fact, that I wrote an entire book on the subject of deploying JRuby applications. And there's not a single line of Java code or XML anywhere in the 200-plus pages. And the reason this is possible is that the technologies we use to run and deploy our applications with JRuby do a very good job of shielding us from these uncomfortable parts of the Java ecosystem. So as Charlie likes to say, I write Java code so that you don't have to. And what that means to you as a developer is that using JRuby at development time doesn't have to be all that different from using MRI. All that stuff that Adi just talked about, you still get all that. You still get to write Ruby code the way you love to. There is that startup time thing, but there's some workarounds for it. There is somebody who will notice this shift from MRI to JRuby. And they're important in our jobs and in our careers, but we very often neglect them. And that person is your CIS admin. Now, CIS admin is arguably the worst job in this industry. They are on call on weekends. They get phone calls and text messages in the middle of the night, which means they don't spend a lot of time with their family. And then they start drinking. And it's a downward spiral from there. And the worst part is, when I say we get to sleep at night, that's the title of my talk, I'm not talking about you, I'm talking about your CIS admin. Stop being so selfish. But the worst part about the CIS admin's job is that they're cleaning up after RMS, the mistakes that we made very often. We're the ones that wrote the code that had the memory leak or the error condition that brought the system down, but they're the first line of defense. A good example of this is when it's time to deploy new versions of our applications into production. So in most organizations, you'll tell your operations team, hey, go run CAP deploy. And even though you know there's going to be all this infrastructure shifting around, processes restarting, and things reconfiguring, it's just admin trusts you. I don't know why, because we've let them down so many times, but they trust us. And when they run CAP deploy, many times, the first thing that happens is CPU pegs. They start running out of memory. Website becomes unavailable for whatever reason. And then they get emails from their boss. And of course, all this is happening on a Friday afternoon when you're at home spending time with your family, and they just want to go home. They don't have families anymore, but they still want to go home. I love CIS admin. I really do. But all these problems that they have are the result of this. What this shows is the general architecture of any MRI-based web application. Regardless of the server you're using the platform you're deploying to, they're all doing basically the same thing, and that is handling multiple HTTP requests by putting a proxy in front of a pool of application instances. And each of these application instances has to run in a separate process. And furthermore, any background jobs, schedule jobs, cron jobs, rescue workers, they have to run in their own processes. So the first problem that this causes is memory growth. In order to scale this system up, we have to create more processes. And each of these processes contains a complete copy of our application in memory, all of it's supporting gems, rails if you're using it, the Ruby standard libraries. So each new process is very heavy, kind of like my son's cars. And ultimately, we end up running out of memory before we really maximize the potential of our hardware and of our system. The next problem is that we have to balance these processes. So our proxy has to be aware of which ones are handling long-running requests, or which ones are idle so that it can distribute the new requests evenly across the pool. And sometimes it turns out that those long-running requests aren't supposed to be long-running, and they're actually stuck processes or zombies. So we have to introduce more infrastructure, like God and Monet, to watch these things and recycle them, maybe restart them if they hit certain memory thresholds or if they become stuck or something. And it's good that we can restart them one at a time, because if we try to restart them all at once, we very quickly overtax the CPU, bring the system to a crawl. So we have to do these rolling restart things. Now, passenger and unicorn have made that problem immensely easier. But let's be clear about passenger and unicorn. I use passenger when I deploy MRI applications. I love it. But let's be clear about what it's doing. It's not solving the problems that this architecture creates. It's just making them easier to deal with. It's an important difference. So beyond those process management issues, there's other issues. Like we have to replicate sessions data across these processes. And each of these processes has its own database connection pool. And having an unbounded number of database connection pools in many ways defeats the purpose of having a cap on the number of connections in your pool. So we introduce more infrastructure, like if you're using Postgres, probably a PG pool to put a layer between the application processes and the database so you don't blow out your connection limit. If we were to analyze each of these problems that I just described, and dig down and find the root cause, we'd see that it's the global interpreter lock. This is a mechanism in the MRI runtime that allows it to map multiple Ruby threads to a single kernel thread, such that only one is running at a time, effectively making the runtime single threaded. There's a lot of folks within the MRI community that would like to see the Jill go. In fact, at RubyConf last year in New Orleans, someone asked Matz, will we ever get rid of the Jill? Very good direct question. And Matz's answer was something like, no. He went on, though. He said, I don't want it to be that kind of platform. And for me, when he said that, I finally understood why the Jill is there. It's not that they're lazy and they don't want to take it out. It's that Matz's philosophy, as Abdi just said, programming should be fun. It should be fun and easy. But concurrency is hard. And so Matz's solution is to eliminate concurrency. And for some applications, that may be a fine compromise. Web servers are not one of those applications. In a web server, throughput is of the utmost importance. And the way we achieve higher levels of throughput is by paralyzing requests. And in order to do that without incurring all the problems I just described, we need a runtime that does not have a Jill. And that's the JVM. The JVM maps Ruby threads directly to kernel threads, which allows the operating system to schedule them on multiple core processes at the same time. You probably know all this. What it allows us to do, though, is change our architecture to something like this, where we have a single process with a single application instance. And then we handle multiple HTTP requests with threads that run in parallel against that same instance. And this is more like the train that my son and I built, where the threads are like the lighter weight train cars. This also solves every one of the problems I just described in the previous architecture. Memory growth is not an issue, because we only have one application instance. We don't have to balance these threads. When they get stuck, it's not a big deal. We don't have to replicate session state, because they're already sharing memory. And there's only one database connection pool. So my argument is that this architecture is more scalable and more reliable. But is it more performant? Well, that's why I brought data. So the TorqueBox team, and I'll talk about TorqueBox in a moment, and they've done extensive testing of not only TorqueBox benchmarking it with real applications, but they've done that in comparison to Passenger and Unicorn, to MRI servers, and Trinidad, another JRuby server. And they have a whole slew of data that'll show you how great Trinidad is, and why it performs so much better. Most of that stuff is the result of the things that Charlie talked about yesterday. But I picked two graphs, in particular, that have a different reason. In these two graphs, we see that the four servers, two MRI and two JRuby, follow almost exactly two distinct curves. In the top graph throughput, requests per second over time, all four servers have about the same level of throughput up until a certain point. And at that point, the two MRI servers flatten out. That point is defined by the size of their process pool. That's the point at which they can no longer service more requests in parallel. But the two JRuby servers can continue to create more threads and service more requests. The bottom graph, CPU usage, we see something similar, is just the JRuby servers have consistently lower CPU usage. And the reason for that is basically there's just less going on. Instead of one garbage collector per process, there's one garbage collector. So given this, and the things Charlie talked about yesterday and the wonderful things about JRuby, there's still been this nagging question. And Charlie calls it the gap in the JRuby story. Yeah, JRuby is great. It's powerful. It's fast. But how do I get my applications into the real world? There's a few reasons that this has been a point of confusion for people. The first is, what do you need to do to your application to get it ready for JRuby? As turns out, that's a pretty simple problem to solve. The JRuby Lint gem, you install it. You run JRLint against your application. And it tells you exactly what you need to change. You won't have to write any Java code, but you might have to change some of your dependencies. If you take nothing else away from my talk today, run JRLint on your projects, I think you'll find it's not such a big deal. But even beyond getting your code ready, there's still this question of, how do I handle deployment? And there's two reasons that that's been confusing for people. The first is the change in architecture, which I already described. But the second is that deploying JRuby applications, or your options when you're deploying JRuby applications, are more multifaceted than with MRI. So in the MRI world, regardless of how you're deploying, you're essentially doing the same thing. And that's taking code that's in a repository or directory and transferring it onto a production server as loose files and then restarting each of your application instances. In the JRuby world, you have some options. You can actually package your entire application into a single file that acts as a unit of deployment. You can continue to use Capistrano if you want and push those loose files across onto the production server. But the way you restart your applications differs. Beyond that, the way we compose our applications in production starts to change. So we can integrate background jobs and scheduled jobs into our applications so that they don't require external infrastructure to run. Even beyond that, there are capabilities of JRuby that don't really have a good analogy in the MRI world, like clustering. So given these different options, there's really three different strategies that you can use to deploy your JRuby applications. And each of those strategies is facilitated by a different technology. Warbler, Trinidad, and TorqueBox, which I'll talk about today, each of these technologies has its own set of characteristics, which means they each solve a different problem. We'll start with Warbler, which is different from the other two that I mentioned, because Warbler is not actually a part of your runtime infrastructure. It's a build tool that you install as a gem, provides you essentially with a set of rake tasks, the most important of which is the war task. You run war against your rails or rack project, and it produces a war file. People in the Java world, I don't have to explain what that is. If you're not familiar with a war file, just think of it as a zip file that follows a bunch of conventions that allows you to put everything you need to run an application into one package. Once you have that war file, you can deploy it in production by simply giving it to your sysadmin. And if they're familiar with Java deployment, they'll know exactly what to do with it. Most likely they'll install either Tomcat or Jetty from their systems package manager, which provides them with all the init scripts they need to run. Once the server is up and running, they drop that war file into a particular directory that Tomcat is listening to. So there's a bit of a difference there. We're sort of have this inversion of control between the application and the container. In the MRI world, you run your application and it starts up a container. In the Java world or in the JRuby world, you run your container, your server, and deploy your application to it. And actually, in most cases, the server deploys your application for you. So this has a number of advantages. To begin with, we can actually hot deploy our applications. So that is, deploy new versions of new war files without ever bringing down the server. It's not quite zero downtime deployment, but it's significantly faster. We can also deploy multiple applications to a single server, which further reduces the overhead or the additional infrastructure we need to run multiple applications. And we might find that this encourages us to break our applications down into smaller, more cohesive packages or modules. So there's really four main advantages of Warbler. The first is portability. A war file is a war file, no matter where you deploy it, how you deploy it, what operating system you're using, what Java server you're using. It's always the same. The second advantage is security. A war file can actually be signed. And if it's been corrupted or tampered with along its journey, these Java servers won't even deploy it. You can also compile your Ruby source code into Java byte code, such to obfuscate it if you're deploying maybe to an untrusted environment. The third advantage is speed. And I mentioned how you can hot deploy these applications. But also, the deployment process itself can be moved along more quickly, because once you get your war file onto your production server, you don't have to do things like run bundle install, because all of your dependencies are already there. The fourth advantage, and certainly the most important, I think, for warbler is consistency. You have a high level of assurance that what you're deploying to test, stage, production, all your production servers is exactly the same thing. And for a lot of folks, that's very important. Now the disadvantages of warbler, there are two. The first is that it's very difficult to have a development environment that mirrors your production environment. And that's because you don't want to generate a new war file every time you change a line of code. So you end up using web brick or something in development, and then find out there's some difference when you deploy. The second disadvantage is that we're really towing the line on the Java world. I promise there would be no Java code or XML. And there's not. But we're using this Java standard file format and these Java servers. And if your operations team is not Java savvy, they're going to hate you for this. If that's the case, if they're already deploying MRI applications, for example, then Trinidad may be a better solution. Trinidad is a lightweight JRuby web server. And one of its goals is to feel friendly and familiar to folks who are already deploying MRI applications. So you install the Trinidad gem, and it provides you with everything you need to run your server. You run it with Rails S or rack up, just like thin or something of that ilk. In production, you deploy to a Trinidad server with Capistrano. It still takes advantage of this inversion of control that I talked about, where the container is deploying the application. But the way in which you configure the Capistrano deployment file, it becomes sort of invisible to you. And actually, your Capistrano deployment scripts tend to look pretty much like a passenger script. One of the other advantages of Trinidad is its rich set of extensions. It provides a number of sort of plug-ins that you can add into your application, like the job scheduling extension, in which you can eliminate dependencies on things like whenever or cron, which run their own processes, and instead, run those scheduled jobs as threads within your runtime. Now, the disadvantage of Trinidad is that by sort of hiding some of these more uncomfortable parts and making it feel like MRI deployment, we're not really maximizing the potential of the JVM. In order to do that, we need a full stack solution, like TorqueBox. TorqueBox is a JRuby application server. And I'll explain more what that means in a moment. But the TorqueBox guys have done a really great job of trying to make this feel as sort of comfortable and familiar as possible to RubyUs. You can install TorqueBox as a gem, but then it provides you with a TorqueBox command, which you use to run the server independently and deploy to it. So it's very explicit about that change. In production, though, you can deploy to a TorqueBox server with Capistrano, or you can deploy as a knob file, which is very similar to a war file and has most of the same advantages. But the real power of TorqueBox comes from its various subsystems. And these subsystems are what makes it an application server. TorqueBox is designed to run any kind of application, not just web applications. Some of these various subsystems are for job scheduling, like the Trinidad extension, messaging maybe to replace rescue, long-running services like daemons and stomplets, which allow you to push messages to a browser. But the most powerful of the subsystems are the clustering subsystems. So when I say cluster, I don't mean a set of servers running in parallel. I mean a set of servers running in parallel and communicating and coordinating with each other. So I'll give you an example. Let's say your application has a scheduled job that runs every night at midnight. When you deploy to multiple servers, you don't want that job to run on every node in the cluster because you might be hitting an external service and duplicating something, things go wrong. So you only want one instance of that job to run. In order to make your cluster homogeneous, you don't want to configure one differently. So now you have this conundrum. You want it to run in one place, but you still want to keep things sane. What TorqueBox does with all of its job subservices is it configures them, by default, as high availability singletons. So when you create a scheduled job in TorqueBox and you deploy it to the cluster, when the cluster starts up, one node will identify itself as the master and it'll run that job. If that node goes down, for whatever reason, the other nodes will be aware of that. One of them will identify itself as the master and continue to run that job. That's really just the tip of the iceberg of what you're capable with TorqueBox. On a larger scale, from my experience using TorqueBox, I was working for a company that was building an analytics engine, essentially. We had millions of records in our database and we needed to crunch the data and discover trends and make predictions. That was the business, making predictions. And so we built a system using TorqueBox and RubyCode that took this data, decomposed it across multiple nodes in the cluster, did some computation, and fanned back in to make predictions. We started with just a couple of nodes, but as the business grew to hundreds of millions of records, all we had to do was stand up more TorqueBox instances. That's how we scaled. It was a very effective approach. There are a few other servers that I haven't talked about. Mizuno and Puma are viable options. And TorqueBox Lite has actually been just announced, so it's not ready to go. But the idea is a TorqueBox server that is more like Trinidad, actually. The most common question I get about JRuby deployment is, how do I deploy in the cloud? And three or four years ago, that was an issue. But today, there are a number of options. The JRuby team has a community-supported build pack for Heroku that works very well, has a lot of great features. Engine Yard, for at least a year now, has had Trinidad as a deployment option. Red Hat's OpenShift Cloud Platform as a Service, very interesting product on its own, has support for TorqueBox. If you're looking to deploy a war file to the cloud, Cloud Bees, who supports the, you may know them for their support of the Jenkins CI server. They also have a war file in the cloud sort of platform. There are things like Google App Engine, which don't use any of the technologies I talked about today. They sort of rule their own deployment tools that use XML, so I don't use them. But ultimately, deciding which of these technologies and which of these strategies is right for you is really a question of the people, processes, and technologies within your organization. So who are the people that you work with? Are they Java people or are they MRI people? What processes do you use? What level of security do you need, or what level assurance do you need that what you deploy to test, stage, and production is exactly the same? Because Warbler can provide some features that can help you there. On the other hand, if you're doing very rapid deployment, maybe from development straight to production, people do it, you want your development environment to mirror your production environment. And that's where Trinidad is good. Finally, what technologies do you already use? If you're using Capistrano with success, it probably makes sense to continue using Capistrano. However, if your organization already runs an Enterprise-class Java web server, you can probably deploy a war file to it without anybody even knowing there's Ruby code running in it. If after thinking about these three things, you're still unsure, you can read my book. And that's a discount code, the JRuby thing. It's somewhere between 1% and 99% off, I don't know. But in the book, I talk more about this why and when to use each of these strategies. But I spend a lot more time on the how, and actually how to write puppet scripts that provision a dedicated server to run each of these different technologies. Oh, I forgot it. I have a copy of the book. I'll find a table to put it out on. Feel free to flip through it. Please don't steal it. But in the end, no matter which of these JRuby technologies you use, I believe that it will ultimately bring you and your sysadmin closer together. And maybe one day, the two of you will have little Rubyists running around, and you can play Legos with them. Thank you.