 Fun fact, there's two other talks here that end with four fun and profit. So the lesson learned is no matter how witty you think you are, someone else is probably going to tell the same joke. What's that? Oh, see, I'm not. Yeah, I'm all about the fun and profit. Hey everybody, I'm Mike. So a little about me. My day job is I'm director of engineering at the Climb.com. We are based in Portland, Oregon. We're an e-commerce vendor. We have a refreshingly old fashioned business model. We put stuff in a warehouse. We put the inventory online. People see it, buy it with a credit card, and we ship it to them. It works out pretty well. And we're hiring if that interests you. So during the evenings, though, I do a lot of Ruby open source. And so I've done a lot of gyms over the years. My latest project is called Sidekick. I hope a lot of you have heard of it before. If not, we're going to be going into detail. So this talk is broken into three parts. I'm going to talk about the basics of async processing, why you want to do it, how we do it. Maybe some design. We'll talk about a little bit of design. Then I'll give you some pro tips that you may not think of when you're writing your Rails applications, Notch applications, something like that, that can influence how well the async processing works for you. And then the last third, I'm going to go into Sidekick and dive into its design a little bit and talk about Ruby in general. So why do we want to asynchronously process? The answer is simple. In a word, it's the user. You respect the user's time by making things work as quickly as possible. You don't want to make the user wait. The user gets bored. He goes away. You lose money. So the other reason is that modern I.O. is very slow. It's orders of magnitude slower than RAM. So anywhere where we do I.O., we can throw it, we can spin it off into the background and save a lot of time and save the user a lot of impatience. And also, I.O. is unreliable. Discs die all the time. The network dies all the time. So it's a good thing if we can do this in the background in a way that's repeatable where we can retry if things fail. So what do we want to asynchronously process? Basically anything optional. For us Rails guys, that basically means anything that's not required to generate the HTTP response throw it in the background. It's safer that way so that the user has less potential for seeing 500 errors. And it's also going to be faster. So this is the general idea behind asynchronous processing. Your code is the client. You put some sort of message on a queue that you want to be processed. On the other side, the other end of that queue, there's the server which pulls the message off and then hands it to some sort of worker to be executed. Really simple, right? So how do we implement this? This is the simplest, most straightforward Ruby I could come up with, which is basically here's a method. It just asynchronously executes that block. And in fact, the go language basically does something identical to this. It has a keyword go. And you just pass it a function. And that function happens in the background. And that's exactly how go routines work. However, there's a problem here with how we want to do things. We want to execute this block in another process. We want to spin off that work to some other machine possibly. And how many of you were at Pat's talk yesterday? You learned all about what a block is. A block is a closure, right? So basically, that closure can access any variables outside of that block that are in scope. So if we want to execute that block in another process, we have to serialize the entire closure over to that process. And Ruby can't do that. It's just not possible. So we can't do that in Ruby. So that's effectively impossible. And even if we could, that could potentially be hundreds of k or megabytes of data. Just not very efficient. So there needs to be a simpler way of saying execute this block in another process. Well, what if we could reference a block by name? Instead of an anonymous block, if we just had a name for a block and that server process knew what that block was, we could just call it by name. And in fact, Ruby has something like this. You have a named block, which accepts arguments. Does anyone know what that is? It's a method. You've probably used them before. So we can do this. Instead of async, processing, and anonymous block, we give it an instance, the name of the method we want to call, and a set of arguments to pass to that instance. Now we've got a much simpler problem to solve. We need to marshal that instance over the name of the method and marshal the arguments. Ruby can do that. And in effect, that's how the Rails for queuing API is going to work. You give it an instance. It marshals the instance over to a separate process and then calls a method on it. However, the problem with this is you still have a lot of data that you're going to marshal. That instance could have a lot of data in it. You just don't know. Also, the arguments could be large. They could be full objects themselves. So there could be a lot of data associated with this. Is there some way we can simplify this a bit more? Well, yeah, there is. We could do class methods instead. That takes away the instance. So now we're just calling method on a class and passing in some simple arguments. That solves half of the problem. Now we only need to worry about marshaling the arguments. I just said that. So congratulations. We've just derived how rescue and sidekick work. You are essentially calling class methods in a separate process and passing it arguments. And so this is an example of sidekick, how you can spin off a method into the background. You just have a class method, which takes an argument. And then you call dot delay on it. And that actually sends the method invocation to the sidekick process to be executed in the background. Pretty simple. Sidekick has another way of doing it, which is through a worker. So if you want to encapsulate a particular job within a class, you just provide a perform method, which performs that job. And then you call a class method that is the sidekick API, which is called perform async. Pass it the arguments. And it effectively does the same thing as the previous slide. That actually turns out it becomes very efficient now. Instead of talking about serializing that possibly megabytes of closure for that anonymous block, now we've simplified things so that we can efficiently just pass in the class name and a simple set of arguments. And that message is very efficient and very small. So that brings me to my first pro tip when doing asynchronous processing. Small stateless messages. Stateless means that the queue is not where your data belongs. Your data belongs in your database. That's why it's called a database. It holds a base that holds data. Your queue contains actions that you want to perform on that data. And so your message really should be perform action x on object y. That's it. So your messages should contain object identifiers that you then look up using active record and so that you can then perform that action on that object. You don't want to serialize that entire object onto the queue because then your object is sitting in two places. Your user object is sitting in the database, but it's also serialized into the queue. Well, what's the problem there? The problem is you get stale data. This is being executed asynchronously. You don't know if it's going to execute five seconds from now or five hours from now. In the meantime, the user could go and update their data and change things such that when the job is actually executed off of the queue, that data could be out of sync. If the message just contains the identifier, you know that you're always going to be using the latest version of the data. Simple types. Simple types are easy to read, they're cross platform, and that's why rescue and sidekick use JSON because that forces you to use those simple types. JSON does not allow you to serialize a full Ruby object. By doing that, tooling becomes much simpler. You can now inspect the contents of your queue visually and see exactly what arguments you're passing into that method. Instead of passing in a kilobyte or a few kilobytes of binary data, you're just passing in a few simple integers. Now I can pop into Rails console and execute that job myself manually. You can see at the top there, I'm having to use YAML because I'm actually delaying that class method and technically, Sidekick serializes the entire class over unfortunately because you might have class state. But it still is reasonably readable as is. Tip number two. We've talked about the client and passing messages. Now on the server side, when you're doing your work, you want your work to be what's known as idempotent and transactional. Idempotent is a fancy computer science term. It basically means performing the action many times will not harm your system. Examples of things that are idempotent, canceling an order. Once you've canceled that order, you can cancel it as many more times as you want. Nothing bad's gonna happen. You update your email address, you change it to bobbatexample.com. You can continue to update it to bobbatexample.com, not gonna harm anything. Things that are not idempotent. Charging a credit card. You ever gone to the place order screen and it says don't double click? They're not idempotent. That's exactly why they put that warning there is because that developer is lazy and does not know what the word idempotent means. Sending an email is not idempotent. You send an email once, Gmail will happily deliver a second email to that same user with the exact same contents. So why does this matter? This matters because your code has bugs and Sidekick will retry your code if it fails. If it raises an error, Sidekick will retry it with an exponential back off. The reason for that is simple. Like I said at the very beginning, IO is unreliable, networks die, third party APIs go down all the time. So it's nice to have an asynchronous processor that recognizes that and just will retry it for you. And so if an error is thrown, you see the email and you don't have to get up and do anything because you know five minutes from now Sidekick's gonna retry it and it's gonna succeed. So that means that you need to design your jobs with the idea in mind that this job could be executed many times. So if you're doing a non idempotent action, you need to check to make sure you need to actually perform that action first. Idempotent and transactional are sort of similar ideas. This is an example of a asynchronous worker. If I'm returning an order, I issue credit for that order and then I send them an email saying that their credit was issued. If I issue the credit successfully but then the email blows up because Gmail is down, is Sidekick's gonna keep retrying that? Are they gonna continue to get more and more credit until you realize that? That's up to you as a developer to make sure that that doesn't happen. So tip number three, embrace concurrency. The standard for Ruby until recently I think has been the single threaded model. I've been trying for the last couple of years to get people to embrace threads. I know the JRuby guys have been trying to get people to embrace threads, the Rubiniest people. It seems like everyone but MRI is trying to get people to embrace threads. And so people who have been using rescue and delayed job which are pretty much the standards for asynchronous processing these days. It's common to have a handful, maybe a dozen workers and that's not terribly concurrent. So concurrency is not something you really need to think about too much. You're not gonna crush your database by having 10 extra processes, 10 extra connections hitting it. But Sidekick is heavily multi-threaded. Every process has 25 threads in it by default and you can up that to a lot more. So if you're running four processes, you've got 100 workers and you've got potentially 100 threads hitting your database all at once. Same thing goes with any third party APIs you're hitting. So Sidekick in fact goes from being a nice polite character to being cone in the barbarian. And Sidekick will crush servers. It'll take down services that you haven't prepared for. Prepared for. We rolled out Sidekick in production at the climb about six months ago and within 48 hours we had taken down a third party API that we were calling because we pushed 500 jobs onto the queue and it processed them all at once and that server was not prepared for 500 connections. So concurrency is something you need to think about when you have that volume of workers and that volume of jobs. There's a couple things you can do. You can use a connection pool so that your processes are limited in the number of connections they can make to that service. You can also mix parallel and serial execution and blend them together. So instead of, if you've got 100 items to process, instead of creating 100 jobs, you can create 10 jobs which each process 10 items serially and that way you've only got 10 workers hitting that server. In my experience over the last year dealing with Sidekick is that thread safety has not been much of an issue. I've found three gyms, I think, that had thread safety issues and we use about 150 gyms at the climb. So it's a very small percentage. Cocaine and typhus were the last ones that had thread safety fixes put in. The base camp gym is also not thread safe but I was able to fix that myself. But for the most part, gym maintainers are responsive and if they're not responsive, that's a sign you probably shouldn't be using that gym at all anyways. So that's a little bit of the theory and some pro tips around asynchronous processing. Let's talk about Sidekick and Ruby and the innards. So Sidekick's mantra is simple efficient message processing. It's multi-threaded rather than single-threaded so it is 10, 20 times faster than something like delayed job or rescue. And that's the reality is that modern Ruby isn't terribly slow. You can do a lot of work if you have the right design. It's single-threading that's killing us as a community. To scale single-threaded model, you need to create lots of processes. That works fine when you're gluing together little command line Unix utilities as was Ruby's history 10 years ago. It does not work fine today in the era of 300 megabyte Rails processes. This is an example from the Climes production server. We've got five unicorns running at about 250 to 300 megabytes each. Each one of those can handle one request at a time. That's pathetic. At the same time, right next to it, I've got a Sidekick running with 10 workers, 10 worker threads taking 250 megs of memory. So Sidekick's doing, effectively, can do twice as much work while being the same size as one of those unicorns. To put some numbers in front of you, 400 megabytes, you want 25 request process at a time. That's 10 gigs of RAM. That's an EC2 extra large instance. That's $480 a month. If on the other hand you use threads, one gigabyte of RAM, let's say, that fits onto an EC2 small for $60 a month. One of the earliest Sidekick customers was on, had 160 dinos of rescue. And he switched to Sidekick, went down to 10 dinos. And a dino cost $35 a month. He's saving $5,000 a month and hosting costs alone just by switching. So that's the inefficiency I'm talking about. That's why I say Sidekick is efficient because it saves real money. At scale. So I was a little bit disappointed yesterday at Ruby 2.0's feature set. When I think of 2.0, I think of a major version increment that breaks things for the better. And I was really hoping to see some concurrency changes take place. Specifically, we need to get rid of the gil and we've got terrible GC. And all this, I think, in my opinion, and my understanding is, a lot of this is just simply hampered by the C extension API that continues to linger on. And so the GC and the gil are there to deal with C extensions. Which is kind of funny because the C extensions are there to speed up performance. Right? With a C extension, I can take my well-tuned Ruby and maybe make it 50% faster. But now I can't use seven other cores. So what's bigger? 50% faster or 800% faster? That's the sad state of where we are today, I think. So if Sidekick internals, the client, we've got your Rails process with your app code. You call the Sidekick client API. Sidekick has a concept of middleware. There's a pipeline that messages flow through from your code before they're pushed onto Redis. This allows us to add, in the same way that rack middleware allows us to add features to Rails, allows you to add features to Sidekick. So there's a client middleware pipeline, which is executed. And then once it flows through the middleware, it's serialized into JSON and placed into Redis. On the server side, the Sidekick server uses celluloid. I know the celluloid's gotten a lot of love so far. And I'm the first person to admit to be another fanboy of it. But celluloid really makes it easy to deal with the multi-threading here. I've got three actors in Sidekick. There's a fetcher, which just simply listens on queues and fetches messages off of those queues. He hands the messages to the manager, and the manager is who keeps track of all the processors. So the processors are the guys that do all the heavy lifting and execute all your work. So you're gonna have 25, 50 processors within your Sidekick process. They each execute the server middleware with your message. Again, that server middleware can modify the message or simply stop the message from getting processed. But once it flows through the middleware pipeline, it's passed to a worker whereupon it gets executed. And that's where your code is called and the work is actually done. Conceptually, it's pretty simple. So there's two versions of Sidekick. There is the base Sidekick, which is free open source, LGPL licensed. There's also a Sidekick Pro. It has more features and it costs money. I charge for it. And Matt yesterday talked about motivation and he talked about love and altruism. Love and altruism only gets you so far. I think money is a great motivator too. So I like to have customers. So as long as the money keeps flowing in, I'm going to be happy to support Sidekick. In terms of feature sets, again, in terms of concurrency model, Sidekick uses threads, the other two use processes. I like rescue's use of Redis. I love Redis as a data store. One of the most frequent questions I get is, can I use Mongo or Cassandra or React or MySQL or Postgres to store my data? And my answer is always the same. No. No. I don't want to make a lowest common denominator storage API and abstract all that just so a few people can run on Mongo or whatever. Redis is awesome. It's got all sorts of cool data structures and I leverage pretty much all of them for the features in Sidekick. So to port to something like a database would be very difficult. I like the middleware design rather than callbacks so that's what Sidekick uses. And as I said before, Sidekick aims to be simple and efficient. Part of that simple phrasing is gathering together all the features I think 90% of us use all the time. One of the things that disappoints me about rescue is I think the feature set is a little bit bare bones out of the box. A lot of people, how many people here use rescue? Okay, so about half the audience. How many of you don't use add-on gems that add on additional functionality to rescue? Yeah, nobody. There is a lot missing from rescue that I really missed. A delayed job has a lot of functionality built into it. Scheduler, the retry mechanism, web UI, the delaying of class methods and that sort of thing. I use that stuff every day and that's why in terms of simplicity, I build it all in so that it's well-designed and all works together well. So the future of Sidekick. I just checked in, I actually just released 2.5 which has a nicer web UI. I still think it can be cleaned up a little bit but I wanna add more functionality to it. I just, actually 2.5 also released APIs for managing queues and retries so that's there. But the big one on the horizon is the Rails 4 queue API. The day that that API is solidified and supported, there'll be a Sidekick release which supports that API also. So the rest assured, Sidekick will move forward with Rails 4. Sidekick Pro, I'm just gonna add enterprise features. Sidekick is where functionality for 90% of you goes. Sidekick Pro is where functionality for 30% of you might go. That's the optional stuff, that's the enterprise type stuff, stuff that maybe hobbyists don't necessarily care about. So to wrap up, I gave you those three tips, small stateless messages, item potent transactional work and embrace that concurrency. And then Sidekick is a different design from I think what Ruby has been offering in the past. And I hope more people learn it, understand it and embrace it. And that is it for me. Thank you. Any questions? Why? So Sidekick's waitable, you mentioned your release 2.5. Do you have any plans to like, be able to remove a job from the schedule queue? The API option, I'll do that now. Can you repeat it? Can you repeat it? Yeah, programmatically. Or you're not talking about the web you like. No programmatically. Yeah, it's in to my pocket. Thank you. Yeah. It looks like it used to be. So, the asynchronous processes you recommend being an active record or data sort of get to get the associated data. Yes. What have you seen or what are your knowledge of like any sort of performance impacts that have on the things that are on the good end of the system? Well, typically you're passing identifiers over. And so that's gonna be a lookup by primary key. And as long as you've got a buffer pool, everything cached in your buffer pool, that's gonna be super fast. So, I wouldn't think that there's gonna be much of a performance impact. I would be much more worried about stale state than I would be about the performance impact. I have a real time problem process. And those jobs are made in a synchronous, but I'd like to know if using fibers with side kick, they have a good integration. There's nothing that I do that is fiber adverse. You can use fibers. Side kick jobs are executed within a thread and a thread can run as many fibers as it wants. So, there's no reason to think why you couldn't use fibers with whatever functionality you want. Shouldn't be a compatibility issue or anything like that. So, I'm gonna go back there. Yep. Yes, just like you, we really like currency and we're getting ready to productize our Rails app. We use a lot of gems. Is there a way to know that it's called a gem to thread safe, or is there some website or how do you determine that? Is there a way to determine if all my gems are thread safe? I'd say that's pretty much equivalent to the halting problem. That is to say it's not really computable. No, I mean, my experience has been if you're using popular, well-supported gems, you're not gonna have much of a problem. Side kick actually did find a Rails threading problem in 3.328 that is fixed in 329. But for the most part, it's very rare. Kind of like unit testing. You can unit test your code all you want but until you actually click through stuff, you don't know it's actually gonna work, right? So you've got to still load test your subsystems. And only that way are you gonna be able to determine if this thing is reasonably stable. Yeah. I've often found when I'm using both rescue and side kick that I'm breaking a lot of big jobs down into small, more common ones. I was just wondering, so you can practically, do you have any advice for how to track those? The guidance started with a report back. I think it initiated the job over here. So in other words, you've got one process which is creating 1,000 jobs and then you want to report back when all 1,000 are done. That's exactly what Sidekick Pro does. It has the concept of a batch where you create those 1,000 jobs within a batch and then when the batch is done, you can be notified on it. But again, that's one of those pro features that I charge for. Anybody else? Yeah. Yeah, not saying it's been a big problem, but you mentioned sending email is, yeah, you could send, send the same email several times. What's your pro tip to avoid doing that? That's a great question. So one of my slides, I'll show you real quick, had a comment that sort of showed a safer way to do this. So when you're issuing credit and then you actually try to deliver that email, that network could be down, right? And so the email will raise an exception and you might issue them credit over and over and over. What is safer is to spin off that email delivery as its own separate job. And so you're doing that email delivery as an atomic job where that's all that job does. So if it fails, there was no email sent. But if it succeeds, you know the email is sent and that's all you, and so by doing that, you take away a lot of the risk, yeah. Yeah, and I agree, it takes away a lot of the risk and I would do it that way, but it doesn't eliminate the risk because you can send an email through an SMTP connection and then you botch up the hang up and you decide, oh no, I didn't manage to do that. Whereas the receiving signs, oh yeah, I've received that. Yeah. You know, at that point, all you can do is really mark, you can store some states saying, yes, I sent this email. That's really all that I can think of you can do offhand. It's just mark a flag that said this was done. Any other questions? Yeah, back there. Can you go with your batches, have batches? I'm sorry? In approach, now that your batches have batches of self-designed numbers. Right. All those children have batches. Right. Someone actually asked for that feature and I started implementing it, but there's a implementation detail that I have not worked out how to deal with yet. So currently, the answer is no. Yeah. You don't need, it's not correct credit, but you still might be able to crank some of the stuff. Reduce process size anyway. Do you mean sidekick or what? Sidekick. Okay. I did do some basic performance testing on JRuby on JRuby to test the number of client messages I could push onto a queue in JRuby versus 193. JRuby was something like twice as fast. It could push about twice as many messages. So the performance is definitely there. But I haven't done, unfortunately, on the multi-threaded side. No, I haven't. It's so hard to get a real world benchmark of, you know, there's no such thing as a micro benchmark for a background worker system, unfortunately. Way in the back. Yes, or the few... Well, Rails does, is not threaded-versed, I should say. It's not Rails' job to create the threads and execute things. That's your application server. So things like Unicorn versus Rainbows. Unicorn is single-threaded. Rainbows, you can choose to use a thread pool to execute Rails. If you're using JRuby, you can use Trendadad and it'll spin up a thread pool to handle requests. And again, Rails doesn't care about that. All Rails knows is it's handling a request on this thread. So yeah, I mean, the issue is, is that you do have some trade-offs there. There is some developer happiness trade-offs. JRuby is super fast in production. And I think it's a great project. But there still remain some developer happiness problems. You know, they got startup problems, startup time issues, as always. And that's just due to the JVM. There's not much they can do about that. I'm sure they've done as much as they can to minimize that. But at the end of the day, thread safety also requires you to turn off auto-loading so you don't get the benefit of reloading your, or hitting control R and reloading your browser on every request. So, you know, it's a trade-off. People just need to be aware of those trade-offs and make their own decision. Yeah. It's like, pro, it's about $500 per month. I think that makes sense for a business, but it's not for a month. Oh, it's for a life. Yeah, yeah. But is there also a trial for that? Yeah. Okay. And no refunds. No trials, no refunds. It's, you know, I'm selling a product and I'm giving people the opportunities to support me in my efforts and my open source. And if people choose to avail themselves of that, great, I'm gonna do my best to support you. Sidekick's still there. It's free, it's open source, it's LGPL. If you're happy with that and you just wanna use that, that's perfectly fine. So the retry is almost identical to delayed job. Delayed job has the same sort of idea. And in fact, I stole the exact formula that they used to calculate how often you retry. It basically retries with an exponential back-off. So if it blows up, it'll retry 15 seconds from now. And then with every additional iteration, it lengthens the time that it retries. And it retries for about three weeks or 25 total retries. And what's great about that system is that if you, if it blows up, you know, I assume everyone's got air brake or exceptional or something integrated into their Rails app. I hope you do at least. If you get an email saying that there was a bug and your worker blew up, you just go in, you fix the bug, you push a new release. Sidekick will retry it eventually. You don't care when, right? You don't have any state in that message, right? Because you followed my pro tips. So the problem just sort of goes away in having to rerun that job. Any other questions? All right, thank you.