 Keep pie a cat, pie a cat Keep going what you do, going what you do Night, hey night I'm just trying to say Only be, be, be, be I'm just trying to say Only be, be, be, be I'm just trying to say Only be like I was your pie cat Is y'all not doing nothing for the black community, right? You know? I'm just trying to say Niggas over here, we ain't got no R-Spec We ain't got no R-Spec R-Spec We ain't got no R-Spec R-Spec We ain't got no R-Spec We ain't got no R-Spec We ain't got no R-Spec We ain't got Tess Unic Tess, Tess, Tess Unic We ain't got Tess Unic Tess, Tess, Tess Unic Banshee stuff, rails Specs, Banshee stuff, Tess Unic All we got is a rail combo That's gate 4.0 Try to do some for you Try to do some for you Try to do some for you It's a great try to do some for you Not as fortunate Maybe it's a great bet It's a great bet Yeah, anyways I figured, you know, it's nice when speakers have an entrance song or something What would be my entrance song? We ain't got no R-Spec Anyways, my Twitter name is Greg Pollack in case you guys want to say anything about or have any comments about this talk and the hashtag you can use for this talk is On the Edge of Rails Performance RubyConf 09, oh my god, WTF by bbq, ha, ha, ha, ha. You get extra points if you actually get, but anyways. Before I get into Rails performance, I wanted to share a little, a couple discoveries that I had last night. How many guys go to the startup crawl? Yeah, well, that was awesome. So we had a couple discoveries along the way, which I thought I would share with you. They'll be in a bigger video, which we're putting together for summarizing the conference. But there was a few discoveries. Starting with the Pivotal Labs. The first really cool thing we saw over there was they have a build board for all the different projects so you can kind of see the status of the builds and are they all in the green? Are they red? Are they building? That was kind of cool. The second cool thing we saw at Pivotal Labs was their awesome whiskey collection. You wonder, it's like, okay, maybe like if all the projects turn green, everyone has a shot or something, I don't know. You sip, everyone has a sip. You don't shoot whiskey. I learned that at Pivotal Labs. Okay, so next up, we checked out Dropbox. And one of the cool things we saw at Dropbox was that the CEO, this guy, has like four screens vertical, which was kind of cool, until you realize he's using windows and he's going like all this stuff around here along with his deodorant and everything else. I'm gonna try to turn down the lights a little bit because I'm gonna be showing you guys a lot of screencast. So it'll be easier to see. Okay, that'll leave you a little better. And the other thing at Dropbox, totally unexpected, is the CEO turns to us and he says, is this desk over here? That's Zed Shaw's desk, Zed Shaw. And we kind of looked at the desk and we kind of looked at each other. Take a moment to think, what would you do? Given the opportunity, it was tempting. But yeah, you know, Zed Shaw, that guy up there. So I thought it was a little disappointing he wasn't there at his desk because I would have got him to autograph his book. In case you can't see it up at the top, this is the subtitle up at the top. Yeah, you know, I figured, well, another thing I thought about was, this was Dropbox. So if I really wanted I could drop something on his box. But no, we didn't do anything bad. Luckily though, an Nathaniel Bibler who does the Ruby 5 with me got a great idea as we were walking out because we love this application GoWalla. Anybody here GoWalla know what that is? Cool, it's an iPhone application basically allows you to sort of like a virtual geocaching where you walk around and you can check in at places, right? And so you basically, you know, you might call up your iPhone app and like these are all the close places around you and you could press one of those and then you could see who else is checked in. You could check in here and like leave items and that sort of stuff. And so Nathaniel got this bright idea. Hey, let's create a place for Zed Shaw's desk. Like little, you know, it's a monument there, you know, in San Francisco. So if you're walking down the streets of San Francisco you might see like, oh look, Zed Shaw's desk, I'll check in, right? And what's really cool about this is when you create something on GoWalla you have to categorize it. And Nathan found the perfect category for it. It's an antique, like Zed Shaw. Yeah, anyway, so that was sort of some discovery from the startup crawl. But this talk is about scaling rails. But earlier this year I released a bunch of screencasts on scaling rails. And have you guys watched any of these? Bunch of you guys, very cool. And so for this talk, as I said in the description I wanted to really briefly go over what we've covered already. Because a lot of the stuff I'm going over today is advanced stuff. You guys already have Rails apps, you're looking for ways to optimize them. But in case you're just getting started with that, here's the scaling rails in 60 seconds. First of all, the first thing you wanna do to scale your Rails application is work on page responsiveness. This is front-end responsiveness. Install YSLOW, get it running on your page, and optimize using what it tells you to do, like compress your CSS and JavaScript files together. Next up, page cache as much as you can. You can't do that, then you do action caching. You can't do that, then you do fragment caching. And you can't do that, you do object caching. And if you don't know what any of these means, well, go watch the screencasts. Cache expiration sucks, just in general. Avoid it as much as you can. Luckily, Rails comes with all this stuff built in so it's very friendly with memcached. And with memcached, if you program in a certain way, you never have to worry about cache expiration. So that's sort of something you might look into. Also, scaling, use background processes, delay as much work as you can. Some of this might be obvious to you guys, so anything that takes, say, longer than 500 milliseconds, put it on a background processor. Client-side caching, this is where you do to deal with ETags and Last Modified. And with Rails 2.2, we got a lot of little keywords and methods that help us do this. Next up, don't pre-optimize. So what I'm talking about here is that 99% of the time, you're gonna need zero of everything that I just talked about, right? So when you start out a project, some people make the mistake of trying to cache too early, optimize too early. And really, if you're worried about caching, set aside some time at the end of your project to do load testing and make sure it scales. And that's when you need to figure out, at that point, you can figure out what you need to do for caching. And then lastly, and this theme will come up many times during the talk, which is don't abuse your database. Because that's sort of like the number one thing you can do wrong with your Rails app. So let's get to the cool stuff. That was scaling Rails in 60 seconds. So a couple months ago, I started up this podcast called Ruby 5, which covers the latest news in the Ruby community twice weekly. And while doing this, I came upon a lot of Ruby libraries and gems, specifically for helping you scale your Ruby or your Rails application. And so what I'm gonna go over, so the talk really, at this point forward, could definitely be called Ruby and Rails libraries that just might save your ass. So it looks a little bit something like this. We're gonna be going over nine different libraries and plugins. Showing you guys a bunch of screencasts. First, we're gonna look over active record plugins for database optimization. Rails plugins to help prevent memory bloat. Then lastly, Ruby libraries to help scale. And this last bit, this is like the most interesting bit at the end, so I highly encourage you guys to stick around. So first up, the bullet library. The bullet library helps you reduce a number of queries with alerts. And don't worry about scribbling this all down. I'll show you guys the URL that I showed at the beginning which will give you links to all of this sort of stuff. Cool. So let's take a look at the bullet library. Do you guys know, guys in the back, do you know if there's any way we can turn off some of these? So basically I have a, here's my view. I'm basically iterating through all the different posts. And I'm doing post.user.name. So I'm getting the author of each of these posts. So let's go ahead and run the server. And then you see when I refresh this. Oh hey, check that out. Bullet gave me a growl notification that's saying n plus one query detected. So it's basically saying, hey, you might want to add include. Awesome, thank you. And if you look at our logs, we can even see it tells us which line of code this actually was triggered on. So we can see now we've got post.user.name. Now, so what I want to do is I'm going to add include and now it's a refresh. Oh hey, now we added include as it recommended we do and now we don't get that growl notification anymore. The cool part about growls it works both ways. If you're using an include that you don't necessarily need now that I took it out. Oh, it says, oh hey, you've got unused eager loading. So you might want to remove include. Okay, so that's cool. Shows me when I don't need to use include. The next thing I want to show you guys with bullet is I'm going to go ahead and add comments. A comment count after each post. So I want to show all of the comments for each post. So I'm going to do post.comments.size for each one, which is going to trigger a query for each post, which isn't good, but it's going to recognize that and it says, hey, you might want to add a counter cache here. Cool. So that's bullet. That's the first plugin we're going to look at. Just allows you to help you reduce the number of queries with alerts. And what's neat is if you're not on windows, you might not have growl. By default, it's going to use a JavaScript alerts if you don't have growl built in with Mac. So next up, oh hey, now it's time for a pop quiz. Here's, oh, I just gave you the answer to pop quiz. What's missing from this migration? An index, right, as Rails developers we're not always that good at remembering to add all the indexes we need. And I'm not just talking about like foreign keys like this. I'm talking about indexes based on what we're querying upon, right? So if somewhere in my code I search on title, then I'm going to want an index on title. Well, how do we then look over code bases and figure out where we might be missing indexes? Well, here's one solution. But you led MITAR called Rails indexes, which basically gives you two different rake tasks, which we're going to run now on our application. So first I'm going to do is I'm going to install the plugin. Now I'm going to do rake show me immigration and what it's going to do is it went and looked at all of my models, all my relationships, looked into my database to see which foreign keys were actually in my database and recommended that I add this foreign key. So it basically gives me a migration. But like I said before, that's not all the types of indexes we need. What about when we have like find by title? Well, that's what the second rake task is for. The second rake task goes through all of my controllers in my views and looks at my find queries and based upon that, give some recommendations for some foreign keys, for some indexes. And I'm going to go ahead and show you guys, if I go ahead and change the code, so you can actually see that this works, I'm going to say find by title and published. And if I rerun it, I can see, sure enough, it gave me a composite index. Cool, so this is just two rake tasks you can run on your Rails app if you think you might be missing indexes. The next library I want to show you guys is Scrooge. Any of you guys use Scrooge before? One, cool. So let me show you why we might need Scrooge in our application. So in the examples that I showed you guys, we've got a post table. And a post has all of these different columns. And in my controller, I'm calling post.all, which is basically doing a query which looks like this. But this obviously isn't the most efficient way to run a query for the page that we were rendering before. Because in the page we were rendering, we weren't even showing the body of the post. So really the most optimal query that we might want to run is something that looks more like this, which only pulls the data that we might display on the page. So obviously if you wanted to follow this methodology, you could go into all your code and you could add the select option to all your find methods, but that's not totally optimal. That's where Scrooge comes in. So let's give this Scrooge a try. So obviously there in my post controller, I've got post.all, cool. And as you can see, without the Scrooge plugin, it's gonna do select star from post, get all the posts when in reality we might not need all that data. So now I'm gonna install the Scrooge plugin. And now this time, the first time I refresh the page, Scrooge is in the discovery phase. It's basically doing something down here. We see it say post Scrooge, doing all this stuff. It's learning. It's learning which items I need in that page. And the second time I refresh, I apologize for you guys in the back. I know there's a dead zone down here. It's only gonna query for the columns that I need, only the ones that I need. So the cool thing about this is it also works with relationships. Let's take a look. So we include the user item table, post.user.name. So we're referencing the relationship. Now the first time I refresh this, it again is doing a discovery. So it's figuring out which columns are being used in this page and getting all the user information. But we don't need all of that. We don't wanna pull the password. All we really care about is the name and ID, which is what it pulls the second time that page gets loaded. So as you can imagine with the Scrooge plugin, if you're, is it really, with this sort of application, it's not gonna make that bigger performance difference. But if you're pulling like, you know, hundreds of objects, tons of data, you know, you've got tables that have tons of columns with lots of data in them, this could really shave off some milliseconds off of each request. So it's something we're taking a look at. So that's Scrooge. Up next, we've got a couple plugins to prevent bloat. And oops, I'm gonna go back. So a couple weeks ago, Sudara Williams wrote this blog post on the Engine Yard blog where he started talking about that it's not memory leak, it's a bloat. Because this guy works for Engine Yard. He troubleshoots Rails apps on a daily basis and sees that people sometimes have a misconception that the reason why these mongrels on the server are bloated while they're 150 megs of RAM is that Ruby's leaking all over the place, right? Which is a big misconception. What he finds most of the time, the reason why these Ruby mongrels are bloating is because they're instantiating thousands of active record objects. That's usually where people abuse active record, which is something that you see over and over again. So he recommended three different tools in his blog post to help you not run into the situation, which I'm gonna show you now. The first thing he recommended was Rackbug by Brian Helmkamp. Any of you guys using Rackbug? Few more people, cool. Well, I'm gonna give you guys a real quick demo. So first we're going to install the plugin. And by the way, all these screencasts since Thursday are now available on the ScalingRail screencast. So you can get access to all these if you wanna go back and look at them later online. So Rackbug is a piece of middleware, which means you're going to have to be using Rails 2.3 or higher to take advantage of it. We're gonna add it as a config middleware, so insert into our middleware stack. I'm gonna give it two more items, two more options. The first one is so that I can secure the Rackbug. It's gonna be my password. And the second one is so that I can actually run database queries, which you'll see me do a little bit later. So let's go ahead and start up my server. Cool. Now the first thing we need to do before we start using Rackbug is install the bookmarklet. I'm gonna go to the secret URL, which it tells you about in the readme. And from there, I can just drag this link up to my browser and turn it into a bookmarklet. Once I have it there, I can go back into my app and I can hit the bookmarklet, type in my password. And it gives me a bunch of information about the request, which just happened. First up, it gives me the milliseconds that it took to render the request. The CPU time, system time, total CPU time. Next up, under Rack ENV, it shows me a bunch of environment variables. This is, you know, if we do a flash, if we have cookies, any sort of environment variables that get thrown back and forth between the server and the browser. You can see right here. We can also see that it did 10 queries, which took 22 milliseconds up at the top of the page. We can see that query breakdown. If we scroll over to the right, we can see exactly where in our code that query was triggered from. So in this case, it was line six post controller. I can actually execute that query against my database to figure out exactly what it's returning so I can quickly debug. I can even explain the query to make sure it's using the right indexes. So it's all right there in the browser. And this is something that we took, I think, from a Django slightly, right? Because they have a Django toolbar. This is our toolbar. We can see that how many active record objects were instantiated for this request. So obviously, if this was in the couple hundreds, maybe even thousands, that would not be good. Might want to do a factor. I get cache usage. So if I'm using any cache on this page, it'll tell me which cache was used, which templates got rendered and how long they took to render. I can look straight into the log file, which is awesome. Especially if I'm running this on my staging server. So I don't have to go to the staging server and look in the log for the request. I can just hit the log here and see exactly what I printed out the log. Very convenient. And the last bit of information up in the top right is showing me something really interesting. First it's showing me the Ruby process which rendered this page is currently occupying 35 megabytes. And from the time that it started processing their request to the time that it's ended this request and sent it back, it increased in memory usage by four kilobytes. It's great information to know. If you see this gets jumped up by 20, 30, 50 megabytes every time you hit this page, might be time to refactor. Which leads us to memory logic, which is one of these libraries which will help you figure out which actions are bloating. Where in your code you're getting memory bloat? So let's jump into memory logic and see what it does. So I just installed plugin. I have to do a little bit of configuration to get memory logic working. Just add a little bit to the top of our application controller and this is all the stuff that I'm doing is in the read me. And if I run my server, I'm gonna click around a little bit. I can look back at my log now and see that it has some additional information listed after each item. Take a look at that. So basically after each line, it's telling me the process which is rendering this action, here is the current memory footprint. So like if I'm getting some bloat on an action, I can go step by step to see where it started getting bloated, at what point in the code. Okay, right after it ran this query, what happened? I even have a process ID which is really convenient. So if I'm looking at say like New Relic RPM and I can see that oh yeah, this one process went out of control, I can dive into my server. Okay, what's the process ID on this one that's taking 150 meg of RAM right now? Grab that process ID, jump into the production log file and then quickly search on that process ID and figure out what exactly happened step by step. Cool. And that leads us to Oink. So Oink, we gotta figure out where our Ruby process is hogging memory, Oink. So yeah, so this takes it a step further and builds upon memory logic. Let me show you what that means. Gotta do a little bit of configuration to get this one going too. So now if I take a look at my logs, it's giving me a little bit of the same information from memory logic, but a little more organized. So it tells me memory usage for this current request is 34 meg process ID and the instantiation breakdown. So the number of active record objects which are getting instantiated. We also saw that on Rackbug and our Rackbug toolbar but this time it's printing out on our log. Now where Oink starts to get interesting is when you start, it lets you do some metrics on your actions to figure out which ones are the biggest hogs. So to do this, first we have to, I had realized you had to add some, change your logging syntax to use the HODL 3000 compliant log or so I'm gonna add that in there. And now I'm going to run my server and obviously from this point we wanna start collecting data because we want some metrics so I'm going to click around. I mean in reality of course you'd run this on your production server and start collecting data on your production server. And now we can start running some metrics. We're gonna run script Oink and we're gonna give it a threshold. We're gonna say look at all the actions that increased memory by more than 75 megabytes. And obviously this is just development, this is a demo so there was none. So I'm gonna change the threshold to something really small just to zero. So it went through, looked at everything in our log and figured out which ones are the worst requests that increased memory the most. So at one point in time the post controller increased memory by 486 kilobytes, 286 kilobytes. So I can figure out really quickly where I might need to refactor which actions are taking the most memory increasing memory the most. And then down here at the bottom it aggregates so you can see the worst actions. Now in reality if you ran this on a real Rails app you might get results that look a little bit more like this. So I can see here that I've got a request, the worst request increased memory by 157 megabytes. That's bad, right? And the next worst and the next worst and down here it's aggregating. So I can tell okay I might want to refactor the dashboards controller because that doesn't look so good. It's increasing memory by a lot. Cool. So we've reached the last section of this talk where I've got three more libraries to talk to you about Ruby libraries that can help you scale. And this is sort of a smorgasbord, a couple of different libraries. The first one is called Rubber by Matthew Conway. And this you can think of as a replacement for Chef with Chef Deploy. It basically allows you to deploy a cluster into EC2 really, really easily. Let me show you what it looks like. So the first thing I do when I'm running, I'm running this is I vulcanize when I'm running Rubber. So this is going to create some scaffolding. So you can think of it as scaffolding for me and vulcanize if in case you don't figure out the connection. Vulcanize is what you do to Rubber to harden it I guess. So we're gonna harden it. And we can specify here which components do we want and our EC2 cluster. Cool. And one sort of aggregate cluster that we can create that's set up for us by default. We can say complete passenger MySQL which gives us all these components. We wanted to add say Sphinx. We could say script generate vulcanize Sphinx. We wanted to add our own custom generator. We could do that as well. So once you do that, what it does is it goes into your Rails app and it's gonna add a new directory inside your config directory called Rubber. Now it's got a couple things inside of it. First it has your deploy information. These are all your Capistrano tasks for all of your different components. So you can see Apache, HEProxy, Monit. So these are all the Capistrano tasks to deploy those items. And down below you've got your YAML files. This is just your configuration. Yeah, so if we opened up one of these directories over here, if we opened up the roll directory inside Rubber, we would start seeing some files we recognize. Like you see we've got this My.CNF and it looks like a Monit script as well. If we looked inside of that, here's what we would see. We can see that it's populating from our YAML files some configuration items and then it's actually going to render this as an ERB file into our MySQL configuration file before it gets deployed on your server. Cool. So let me show you the secret sauce here. If I go ahead and run these Capistrano tasks on my server, once I have Rubber installed on my Rails app, you can see this is basically creating a bunch of different items for me, different rolls. Out of the box it's going to give me a complete cluster in EC2 after a couple lines. What's really cool is the guy that created this was really smart. System Administrative Nutrition Guy has ton of stuff built in. So you've got like by default, you've got database backups going on. You've got Monit monitoring your services. You've got Moonin, all these stuff like sort of optimal cluster of servers working for you. So yeah, it also has dynamic DNS updates, which is great. So if you want to do a new deploy, you could just deploy to an entirely new EC2 cluster, right, change the DNS automatically. You could do that automatically in your Capistrano task, kill the old cluster. That's built in backups like I said, so sort of like best practices for deployment. Easy staging. One of the really cool things about this is that when I talked to the guy that created this gem, he said that in their project where they have this, if any of the developers ever need to show off a feature, well, they deploy to EC2. They start up a staging instance. The awesome thing about that is that staging instance has the exact same configuration of their production instance. It's just on one box instead of many, right? And it just makes it so easy to spin up new instances. Oh, I'm gonna show off a feature to a client. Okay, we'll just spin up a new staging server. I'll run one Capistrano command, go. I'll throw it away afterwards. It's kind of neat. You've got Monit built in, which does all your process monitoring. And like I said, you have Moonen to do all of your stats across all of your boxes. So that's rubber. What's great about that, what's kind of neat is it keeps deployment a first-class citizen. So because you're putting all of your configuration files for your cluster right there in your Rails app in these directories, well, all of your configuration management is in source control, which is awesome. It's right there in your Rails app. Kind of feels good to just have it right there in your Rails app. There's something cool about that. It's also very scalable by default. Obviously, you wanna add another server, you just change a small configuration line and redeploy your cluster. Cool, so that's that one. Now I'm gonna be talking about Cloud Crowd a little bit, which is a background processor, which is really neat. A simple background processor. So these guys from Document Cloud had this problem. And really, they have this problem in their new application that they're developing for journalists. Basically, journalists are gonna be sending them PDFs, big PDFs, and they need a background process, which will hopefully use sort of like a map-reduced-like function to take that PDF, split it up, get images of those PDFs, strip the text off of them, and then combine them all together. So kind of like Scribd, where they'll take a PDF and sort of just put that onto the web. And that's obviously they have a background process, and these guys need a background process too. So out of this, they create a document cloud to do this for them. So here's the anatomy of their application, of their, of Document Cloud. So we've got an application or website. It needs to start a job, maybe to parse a PDF. That's gonna send a request to a central server. That's gonna have some sort of database. Could be MySQL, could be SQLite. That's gonna dispatch some work to what we're gonna be calling nodes. Those nodes might have workers that do work for them. Once those are complete, it's gonna save those files to S3, notify the central server that the results are done and that can send that back to the user. So this is basically the anatomy of a cloud crowd cluster and also probably many other database, background processors. Let's go ahead and run it though because this is where it starts to get really cool. So I'm gonna create a cloud crowd application. So I just do this install. And basically this gives me a couple files. First it gives me a rack file. So this is going to be my server, my central server that's gonna be monitoring the processes. And I can do some customizations in here. I can specify the number of workers that I want. I can specify the storage in here. Since I'm in development, I'm just gonna use the file system. Now I can say what sort of database I wanna use. I can use MySQL by default or in this case, since I'm just playing with it, I'm gonna use SQLite, cool. And now I need to create my database which is gonna hold my background processing. My jobs. Cool, and now I'm gonna go ahead and start up the central server. And here's where it really starts to get cool. Check out this UI. So this is running on thin, start it up, and here's sort of the UI that you get for cloud crowd. On the left, it can tell us all of the worker nodes. We don't have any yet. On the right, we can see all the work units in the queue. Notice that it's updating, it's doing polling so we can see what's in the queue at any point in time. I can see the jobs in the queue and what they're doing. And I can see active nodes and workers. It comes with this really nice interface which I'll show you, we'll check it out in action. So let's make this background processor do some work. First thing we need to do is start up a node. We need a node so they can do some work. So we say crowd node. And if we go back to our website, we can say, hey, yeah, look, okay, we've got a node here that's ready for work. So let's give it some work. So we can see that it comes with by default, a couple actions that we can do. We can do some graphics magic processing. We can process PDFs or we can do a word count. Let's do a word count. So here you can see we have to have a process action and a merge action process. It's just gonna do WC and count the number of words and whatever you are all I send in. And then merge is going to sum up all the counts of all of the documents that I send in. And I'm adding a sleep five to delay it just so you guys can see it in process. Otherwise it would be too quick. So cool, so how do I start that action? Well, I simply do a rest post request to jobs. I say, do the word count action and I send it a bunch of inputs and all these different inputs. These are just basically Shakespeare plays. So I wanted to count all the words in all these Shakespeare plays. So it's just gonna do a quick request. So let's go ahead and start that. As soon as I start that, I can go over and look at the, oh hey look, so now we've got five nodes all doing work. I can click on one and see exactly what it's doing. You can see, oh we're 23% complete. We can see exactly how an hour 41% complete it's running. We look over at the graph, we can see it started out with 23 work units in the queue and it's slowly processing using the five nodes that it has. Down here we can see the number of jobs in the queue that just won and so hey we're complete. Pretty cool. So now to figure out the status of how did it complete successfully. I happen to know the job that we ran would have got the idea of one. If I go to slash one, I get in JSON that it succeeded. It took 21 seconds and there's 558,804 words in all of those Shakespeare documents that I had it count. Actually we're like, ta-da. So that's Cloud Crowd. Why is Cloud Crowd interesting? Well it's all in Ruby and they made it so it was gonna be very simple and hackable. The disadvantage of using something, using a different background processor like Gearman or what's the new hotness? What do we? Like Rescue, what's the? What's that? Or like, what's that? Oh like, no, not like Nanite because Nanite's like sort of like Ruby too. But the disadvantage of using these really sort of robust background processors that you see people like Facebook using is that these are all in C. So you're gonna be, if you ever need to hack them and make them behave a little bit differently you're going to have to learn C. These guys from Scratch wanted to keep it under 2,000 lines and make it very hackable. So I have the confidence in knowing that if I add this as a background processor and I ever wanna dig down on the code I'm not gonna be intimidated. I'm gonna be able to make it do exactly what I need it to do. And there's some sort of good feeling that comes from knowing that the code that is part of my system I can go in and I can hack if I need to. So that's sort of the benefit of Cloud Crowd. Cool, now we're down to the last bit which is good because I get to rant. I get to do a little bit of rant on email because there's two problems with doing email and any Rails app that really suck. And I'm sure there's a few of you guys that really feel my pain. The first thing that sucks about doing email in Rails is mailing lists. Every client that comes to you inevitably needs a white list or some sort of subscription list so that when people sign up they can say yes send me the newsletter, send me occasional discounts on lab tests and you've gotta maintain who's subscribed, who's unsubscribed, maybe even built in the ability to send out mass emails or at least export to CSV or something, right? And that sucks because you end up having to do that over and over again sometimes. The other thing that really sucks about email is client changes. Because in Rails all of our HTML, all of our emails here are inside of our Rails app. So every time the client wants a change they have to go to you, the developer, and say, oh hey, could you change this grammar for me or could you change this one line of text? It really sucks that me as a developer, I have to deal with that, that I have to do that work, that that sucks. It's too bad there isn't a way that I could maybe not do that. While there is a way, there's hope for dealing with these two different things. There's several different ways that you deal with this, but one of them that I'm going to mention tonight is by using a web service called Mad Mimi. And these guys aren't, they're not paying me anything to say this, but I want to give you guys a demo because they're our web service, they're our paid web service, but they're fricking awesome and I'm going to show you why. So in Mad Mimi, when I want to add a user to a mailing list, they have a very simple API, so they solve that problem for me. So I can say, add tom at example.com to this particular list, so I just need to do a call, restful URL call. I can also just use the simple URLs, I can add a user, I can remove a user, I can even remove several users to a mailing list and add some CSV parameters. So I can just push them all there. What that means is now the client, whenever they want to send out an email, they can go straight to Mad Mimi and Mad Mimi's an email marketing service that does all this sort of tracking stuff, so they can go straight to Mad Mimi, create these bulk emails that they want to send out to all the users, create it there, send it out, create the campaign there. Mad Mimi does all the tracking and everything, so I can have them interface with Mad Mimi whenever they want to spam their users and we don't have to worry about exporting any CSV files because we're keeping the whitelist pretty much on Mad Mimi. So that's the first cool thing about Mad Mimi. The second thing, which is where it gets really neat, is when it comes to client changes. So what we did with this project of mine, we had this situation here where we had a lot of emails inside of the Rails app. We took them, all these emails, and we put them inside of Mad Mimi. Mad Mimi calls emails promotions. So we recreated all of them, even like the welcome emails. All the emails that the system sends out recreated them using the email builder inside Mad Mimi. Cool, so now in our Ruby application, our Rails application, we just replace ActionMailer with the Mad Mimi gem. Mad Mimi's a Rails app, by the way, just kinda neat. So we add the Mad Mimi gem and now to send an email, this is all we do. So for our new account, we say from this guy to that guy, here's the promotion name, which is the name of the email we created in Mad Mimi and here's the subject line. And to make that email send, we simply say deliver new account notification. It goes out to Mad Mimi, sends that email. I mean, that's pretty sweet. And now I don't have to even deal with like, SMTP servers and crap like that. So the one thing you might be thinking at this point is, well, what about for dynamic emails? What if I have like, I wanna put the user's email inside of my emails. If my emails are gonna be on somebody else's server, how do I deal with that? Well, Mad Mimi solves that as well, using these sort of brackets like you see here. So if I've got name, email, account, all I have to do is this. It's gonna wrap that up, send it over to Mad Mimi, populate the variables. So now my client is much happier because every time they need to make a change, they go straight into Mad Mimi. Every time they need to make a grammar change or a spelling change, the client go into Mad Mimi user, they're very easy user interface, make those small tweaks, don't have to bug me at all when they need to rearrange the emails. Oh, awesome, right? And so yeah, so these are the promotions inside Mad Mimi. Yeah, they can go right in here and they can edit the new user email, the forgot password email, anything that goes out. They can edit it right there. My client can edit it. So I did this with an application. It was going really well. And then all of a sudden one day I got this really awesome call from my client. I just couldn't believe this really happened. My client gave me a call. They said, okay, when somebody buys this item in the Rails app, I want this email to get sent out. And the email that I need to get sent out, I already created it for you in Mad Mimi. All you gotta do is hook it up. So I went, I was like, damn, right? Seriously? So all I had to do is go into the Rails app, hook it up to Mad Mimi and it was done. I mean, that was like, okay, that the client did like half the work I would normally have to do for me because they created the email. So that's Mad Mimi. And a lot of the stuff that I talk about with the scalability has to do with creating web services. So you can kind of think of this as another web service which can help you scale, right? Because you want to have, obviously when you scale stuff out into big pieces, all the big applications, whether it's Scribd, they've ended up splitting things out into components that are really focused on doing one thing. And really this is one thing. This is doing, taking care of your emails so that you don't have to deal with it. If your system gets really big, you'll never have to deal with emails and it's probably cheaper to pay them a subscription fee than it is to maintain that hardware, figure out how to do it and hire someone to maintain it. So that's all that I said. So all these screencasts, as I mentioned, if you go to this URL right here, you can get really quick links to all of these items which I talked about. Also, if you want to see the screencast again, they recently got posted up to the ScalingRail screencast series, so they're on there, so you can show other people if you want to show somebody the screencast again or you want to watch it again. Yeah, last bit, I did want to mention before we do questions, if there's any speakers in this room and I haven't gotten your 30 minutes, because you know I go around to the conferences and get all the speakers to give me their talks in 30 minutes, come find me, if you're 30 seconds, sorry. I aggregate them all and it's about 30 minutes. And then also, I talked about Gowalla at the beginning. If any of you happen to be sticking around tomorrow, Nathaniel and I are going to be doing two trips. Basically, they have these cool things like where you check in at a bunch of locations and you get a badge for doing it, right? And so there's, in San Francisco, there's two, there's North Beach Inn going to some of the coolest places in North Beach, and then I think we're gonna rent bikes and do the Golden Gate Gallop and go all the coolest places along the water and across the Golden Gate Bridge, renting bikes. So if any of you in here, feel free to come up if you wanna join us, come talk to me. And I guess I have maybe two minutes for questions. Any questions? What's not solved? Well, I think the most interesting bit going on right now is web services, right? I still think people aren't taking advantage of web services as much as we can, and I think Mad Mimi is a great example of that, right? If you have a web service for, like I know I've also heard people doing like MongoDB, like I think you can get MongoDB in the cloud as a web service, right? So I think it'd be interesting to see what other components of our web applications people can create sort of cheap paid web services so that we can easily, we can scale easier by taking those sort of like background jobs that maybe we don't wanna deal with, or email in some shape or form, and pushing that off to a separate web service that we might not have to deal with. So I think that's really interesting. Oh, and yeah, and if somebody creates that stuff, we'll be bound to talk about it on Ruby 5, so listen to Ruby 5. Oh, also, if any of you guys sitting in the audience ever have any open source libraries or tools or gems you wanna get publicity for, get the word out, send us an email, because we love people emailing us stories and we typically cover what you send us unless it really sucks. Yeah? So just a comment. So another solution for memory profiling is to use Ruby prof and patch Ruby, which lets you do full line-level profiling and have like cash grant interface on top of it. Yeah, I guess you guys couldn't hear him in the back. He was saying, another way to do memory profiling is to use Ruby prof with the Ruby patches on top of that. Yeah, that's definitely one way to do it. Yeah. And also New Relic, I mean New Relic RPM is really good for keeping an eye on all of your servers. None of the stuff that I mentioned here sort of is sort of like a replacement for that. I'm a big New Relic RPM fan for monitoring your servers. It's also really good, especially if you start optimizing with memory stuff, because you can then go back and see, okay, a week ago, what do we look like? And today, what do we look like after the optimizations? They make it really easy to do that and compare results. Any other questions? Yeah? Does anybody know where Scrooge stores the optimized queries? Does anybody know where Scrooge stores the optimized queries? Not offhand. Sorry. I don't know what works with multiple databases. How it will work with multiple databases? Well, I suspect what it's doing, and this is just me sort of just, I suspect what it's doing is some form of a little bit of metaprogramming so that it's actually maybe writing some method definitions which get called. So I suspect it might not even matter what kind of database you're using, because it's actually writing out method calls. At least that makes sense to me. Any other questions? Are we good? All right, thank you guys for coming to the talk. I really appreciate it. And don't forget to check in at Zed Shaw's desk.