 Hello everyone. Everyone have a good lunch. Good, bad, Mexican? No, okay. So yeah, as Jim said, my name is Jared Dillon. I am the chief architect with Quick Left and I'm a Rubyist. I've been a Rubyist for a long time and I do way too much work with AMQP. If you know what that is great, if not, we'll talk a little bit about it here in a second. So welcome to the Ruby Enterprise. We're going to talk a little bit about message systems and integrating applications in a service environment and we're going to talk a little bit about patterns that we can do in this environment, horizontal scaling, among other things. So if you didn't get it before, we have an enterprise system here. And you know, when we're talking about these systems, we're talking about really, really large systems. And these systems are not only large, but they're homogenous. We have a whole bunch of different things that are wrapped up inside here that are all individual parts. You know, we have app servers, we have web servers, we have all kinds of different things that, frankly, we really don't care about. You know, and imagine ourselves in this scenario, like many of us have been in, where we're building a Ruby app, be it Rails, Sinatra, whatever, to work with this enterprise system. And we work within this system, but we also work apart from this system. In that we can't integrate, so we do have to stand apart. Whether that's a language barrier, a political barrier, a cultural barrier, we really don't care. But we don't exist apart from that. And the application has to live outside with the boundaries of this enterprise. However, whatever that means for you. So the real reason this talk, and a huge purpose of it, is talking about how do we connect these two systems together? How do we, how do we move Rails and Ruby into an environment larger than itself, larger than these standalone applications that have been typical in the past? So a quick way to do that, in ways that have been talked about in the past, are APIs. You build out APIs, you know, you need to consume some information, and you make a request out, and you get it back. You know, that's not so bad. So we, we add another request. And some more requests. And we have a whole bunch of requests now. And so we're collecting and consuming information from this larger provider that we need to work with by contractor, whatever else we need to do. But now we also need to produce information. They need stuff from us. So we talk about going the other direction. We, we connect back. We build an endpoint. We build some more endpoints. And more endpoints. And you know, we have lots and lots of endpoints. All of a sudden we have a mess of 14 integration points here. You know, that are tightly coupled. If something breaks, then you know, a lot of your system can break. And these APIs are unreliable. And part of that comes to the fact that HTTP is a fairly unreliable protocol. And you may also have a public API. So this is, you know, a lot to maintain. It's difficult to scale. So you got to ask yourself, when you're starting to integrate, is there a better way? You know, we're looking at some tight coupling problems. You're building to a huge implementation. It's synchronous in nature. So all these requests that we make, let's step back, all these requests that we make block both sides of the application. Something in this homogenous mass or our, our application. These controller actions are going to be synchronous. So let's go back to the drawing board here. You know, how can we rethink this? What, what do we need to do in order to consider, okay, what's a better way? Or at least, what trade-offs can we make to come to a better decision about how to integrate these two applications? And so we start thinking about, okay, well, some of our problems were the tight coupling. Well, if we could put an interface in between to reduce the coupling, that would help that. We, uh, we have many interfaces to different places. And so we can reduce that to a single interface. We're synchronous. So we're blocking our threads in our application. So what if we could make that asynchronous and also get rid of that problem and allow our applications to consume things as they're needed instead of being force-fed every time someone needs to consume or produce something. So instead of this, this huge, huge mess we have, we can get down to something more like this. And for the purpose of today's talk, we're going to talk a little bit about a rabbid MQ. And rabbid MQ is a message queue that follows the MQP standard that was developed by JP Morgan Chase in 2005. And it was in response to problems like this where they had to integrate huge parts of their application together and do it without expensive proprietary solutions. So they developed an open source standard that really allows and makes us available to all of open source and it's battle-tested. It's in use at JP Morgan every day. So now we don't care about the enterprise. All of a sudden with this solution, we can focus on integrating with a middleman and not really care about what we're originally integrating with. So we don't really don't care about the enterprise setting work. We only need to talk and interact with the message queue both ways. So, you know, for those who aren't familiar with how a message queuing system works, we have a, we transmit messages through an exchange and think of that as some form of channel that's eventually delivered into the correct queue that is ready for the app to request and consume items as it needs to. And this doesn't tell the full side of the story because, you know, our app server isn't really just a single homogenous math itself. It's made up of a web server. It's made up of workers. And so in this asynchronous environment, we certainly need those workers and some of you may have done some of these worker systems before. So, we're happily consuming. We're requesting things out of the queue and we're processing them and failure is handled. You know, the queue, things are only popped out of the queue if the worker acknowledges it until you know you have some sort of failure handling built in. So I'm going to show you a very simple example of a worker. There we go. Very simple example of a worker. All we're really doing here, it runs in an event machine loop. I'm also a huge event machine fan, but it runs in a simple event machine loop and subscribes to our queue. And as soon as you get something, it processes it. I didn't want to do anything too complex because it's really not the point, but this is done with the Ruby A&Q PGM. And there's some synchronous examples if you're doing it in another environment like Carat or Bunny. They're all out there. And all these resources are available at the end of the talk as well. So, all this is really doing is we're subscribing to the transactions queue and then we're waiting for messages and we're doing things with them as soon as we get them. And we can scale this out horizontally. If you have more messages entering the queue, process more messages. You know, we have three workers. Maybe you need five. Maybe you need ten. You know, you're doing some huge processing at this point. But all of a sudden, you have changed the way you scale your application and you've moved out horizontally across processes instead of trying to do it in a more traditional fashion. And the reason we part of what's aided with this is, RabbitMQ automatically load balances. And so, you know, worker one will request and worker two is requesting, worker three is all subscribed. And all of the load balancing is handled for you. There's no locking. There's no, there's nothing to think about with that. It'll automatically send message one to worker one, message two, three, four, and it'll round, round through them in order to evenly distribute the work between all of them. So what does all this mean? You know, we've now talked about this system and how to set it up and, you know, kind of talked about some of the benefits here. Well, we can go from this. You know, this ugly, ugly mess of integration to this. But there are some tradeoffs with it, of course. You know, you now are talking about a single point of failure. If that goes down, you're not communicating between each other at all. But you can guard against that. You know, with RabbitMQ, there's the notion of persistent queues. And so, you can have your queues right to disk if necessary for critical things like transactions we're talking about. We can also do some sort of clustering and sharding, other things to protect against it. And we aren't in an asynchronous environment. So if the workers can't consume, it's not like there's active failures that are happening. You know, so, okay, so let's assume that we've gone with RabbitMQ. Problems are solved. Our application is running. It's growing. It's integrating to the system. It's coupled very loosely, but it's coupled. And for all intents and purposes, we've solved our challenge. So, we're beginning to see performance problems. You know, we don't really care about the enterprise. You know, we care about our application. Our application is doing a lot of growth as well. And we can say it's a railroad app by itself, but this doesn't necessarily tell the full story. We've talked about the web servers and the application servers. And we've talked a little bit about the workers that are inside there. But there's another piece. There we go. And we have a database server. And that's another constraint that we have upon our system that we have to think about. And as this system grows, you know, the constraints on that database server also begin to grow. And we have to think, okay, most of our rails out by itself is synchronous as well. So our controllers look something like this, trivial example, but we're creating a purchase and we save it, we redirect it. And so what we're saying here is we have to make a request out to our database server, save this object, and we can't continue until this is returned. Now, it may not matter in most environments, but if you're in an environment where milliseconds matter, it can. And so every transaction to the database can adversely affect the performance of your requests. So we have a direct interaction with the DB here. But we can pose a question. What if we applied the same model that we did in the enterprise to our intra-application communications? It would be interesting. All of a sudden moving from synchronous to asynchronous. And some of you may have done this in a certain way, but I propose a slightly more aggressive way. So we can apply this model here to facilitate intra-app communication. And we can solve performance problems in our own application and think about all of our controllers, everything we do to produce information or application asynchronously. So, instead, we talk about, you know, instead of doing this where everything is synchronous, what if we did this? What if we created our object in memory, made sure it was valid, never touching the disk, and if so, throw it off somewhere? Let the web server do what it's great at and let something we designed more for this purpose to do that. And we eliminate most of the disk IO in doing this. And there's no database interaction at this point, allowing requests to happen very, very fast. So, what we're talking about again requires some sort of worker queue system. And we've already talked about a bit about RabbitMQ. That's not the only option in the sphere. So let's talk about a couple of others. And the first, we'll talk about delayed job. Who here has used delayed job? I'm really sorry. I have to. I know there's a lot of pain points with it. Who's experienced some pain with it in general? Okay, a couple. The more you do with it, the more painful it tends to become. You know, on the plus side, you don't have to set up anything new on your service. It's just, you know, that piece of infrastructure and it works. It's also very, very simple. You know, at the end of the method, you put handle asynchronously. You say, you know, my class or my object.delay.method, and it figures it out. That's great. But if you're talking about you're in an environment where we're already performance constrained, this is also going to affect the performance of your database. So maybe we don't want to do that. Another option we have, who's heard of queue classic? Cool. I'll be, this is awesome. So queue classic is an awesome library written by Ryan Smith of Heroku, and it takes advantage of the fact that Postgres has both a key value store as well as a public subscribe system. And if you're not using Postgres, I mean, it has many, many advanced features you should use, but we're only going to talk about these ones. So of course, because it takes advantage of these, it has to be run on Postgres, which is awesome. It also keeps your structure simple, and the queues themselves are very, very simple. The resource constraints of places on your database server are very minimal, but for the case of this, we're just going to say that it is constraining on the database. So maybe we don't want to use it, but it's definitely something to keep in mind over delayed job if you have to do this in the future. And just a quick look of how the public subscribe works in Postgres. This is not part of the SQL standard, but we can start to listen on channels. And then as soon as notifications go out, all of a sudden we're starting to receive broadcasts as you see in the third line. Very, very powerful stuff and allows a lot of things you can do just with Postgres itself. And it's extremely lightweight as you can see. So, all right, we go back to the drawing board now, and we have another option. Can anyone guess what that third option is? It's a fairly common library. Bam, rescue. So rescue sits on top of Redis. And for those who don't know what Redis is, rescue or Redis is a very fast, lightweight key value store with some set algebra function. Pretty cool stuff. It's fast. It's simple. You do have to add another piece to your application or to your infrastructure. So, you know, you may look at that and you may not want to set that up, but it's very easy to set up. And so rescue is a queue system that sits on top of this and interacts separately from your database server. So we actually don't have to touch our database unless we really have to. We don't have to do it for the process of checking the queue and interacting with the queue. So, you know, here's the quick example, given our prior controller action, you know, how we could possibly set this up. So we have an asynchronous create, which enqueues our object with what we wanted to do and some parameters. And that gets enqueued. And so now this transaction class will actually work it as soon as we start doing a rescue work on that queue, on the purchases queue. And so we can actually do this later instead of at the time, which I don't know about you guys, but when I'm creating, when I'm producing things, I entirely need that object out from the database right then. I already have it in memory. So, now our async controller makes sense. You know, we pass out, we can run through this entirely in memory by just checking if it's valid and let this piece handle a lot later. It's kind of awesome. It gives a lot of added power, a lot of added functionality and making our applications very fast and scaling in a very cheap way. And again, like in the prior model, we can go from, you know, something like this to a much larger model like this. And the big thing is, is we've added a little bit of complexity to our application, right, because we're now no longer thinking in a very synchronous way. Nothing's direct anymore. Everything's indirect. And so that can have implications on your code in that you have to have a little bit more of it. You have to handle consuming this information because your web server is not going to do it. Your web server is responding to requests. And that's important to think about. Now we have, in this case, six queues running or six workers running, all handling different queues across different systems. And so we've gone from just having the blue enterprise and the red Rails app to something much, much, much more complicated. You know, whether you take that trade-off, engineering, of course, is always trade-offs guided by, like, very conscious decisions. And in this case, it may be much, much more appropriate. And in other cases, it may not be. So it's really up to you to decide. But when milliseconds matter, you can go asynchronous and stick with the worker pattern and move up horizontally. So thanks, everyone. I tossed out these slides a couple of days ago and redid them. So any feedback, please give me feedback. But here's all the resources available for this talk. If you're curious about any of the topics here, there's a RabbitMQ, we have some, the RubyMQP library is awesome. The most important link, I think, is check out the queue classic and how HStore and the PubSub works. So thank you very much. Brighter Planet is a cloud-based computation platform. So we provide an API for developers to build complex scientific calculations into their applications pretty easily. One big client we have is MasterCard International, and they have a partnership with us where users of corporate cards who charge things like flights and hotel rooms onto their corporate card, all those pieces of data go into a big database and we go in and, using all those details, calculate the environmental impact of all those different activities and put that information back in the database. So the users of these corporate cards can use that information for making energy efficiency adjustments and corporate reporting.