 So I crammed as many acronyms and words and project names as I possibly could in the title. I could have put some more, I guess, but I had to make the font smaller. So we're going to talk about Rails 3.1 and everything we could cram into it. All right, so we're going to start a quick little introduction on me, OpenLogic, background of where we needed Rails 3.1 and all that stuff to come in, what kind of problem we solved, how we went about it, and then a little bit about each of those technologies, how they work and don't work. And we'll wrap up a little bit with some final thoughts and save some time for questions. Okay, so I'm CTO and founder of OpenLogic. Been in software a long time, some large companies, small companies in between startups. And right now I'm actually writing a book, Cloud Computing and Action Innovating with Open Source for Manning, and there's a little code there if you want 40% off the early access. OpenLogic, quick background. We help primarily large companies with Open Source getting up to speed, policies, training, SLA support, 24-7 production, governance scanning solutions, kind of a one-stop shop. And we have a free site, olex.openlogic.com with over 350,000 Open Source packages on it. So you can search and slice and dice and learn about things that you might want. And then we certify over 650 so you can download them, get support from us. And that's all free. And we have a couple hundred enterprise customers today. Okay, so the background, why we need all this stuff? Well, we decided to create a new product a couple years ago in the cloud, like everybody else, right? We called it OpenLogic Cloud Swing. It just came out in beta last month and we're having a beta 2 go out this coming Monday. OpenPASS, so the idea of what it's trying to do is to get people up and running very quickly in the cloud of their choice with the stack of their choice, kind of how they want it. And the things that are underlined there are the bits that are already implemented in the first beta today. So that's what we had to build very quickly. And by quickly, I mean, I created a prototype late last year and we wanted to get that prototype rewritten from the ground up zero to production in two months on all those technologies that we hadn't used yet. So that's the crunch time. So some of the problems were that there was a lot of excitement around the prototype that I wrote. We won a CloudConnect competition in March at the CloudConnect conference. That gave us a lot of attention. We have a lot of Fortune 500 customers over the last six, seven years that were interested in what we were doing with the cloud and so they were ready to jump on it. Marketing sales said, can we sell this today before we wrote the first line of production code? So the usual problems with a quick prototype. And all of our existing SaaS products, although we've been in production for four years now with Rails 2.3x, it eventually started at 2.0, 2.1. We used some jQuery, Redis and MySQL. We hadn't been on the cloud yet. We hadn't used Amazon or Rackspace or anything like that. So a lot of this stuff was pretty new technology-wise and we didn't drop everything. We still have all our existing products, existing customers that we have to keep making happy, keep adding features and building, and it's not a big team, right? So we had to juggle a lot. So what we did, the approach was to partition the team. So we said, okay, we're going to have to carve off some time. It's too hard to put all of these things in one big bucket of stories or an agile team kind of scrum-based. We need to isolate to make sure that we progress on both the new stuff and the existing stuff no matter what. And so the way we did that is we actually split the team and we asked the team, we were kind of nervous about how this would work. Would we have to, you know, as management, just make the decisions and say, well, you're the best fit for here, you're best for there, or should we ask? And we started by asking and the good news was they basically self-selected exactly like we wanted. So it was perfect. Some people said, I don't want to give up on the old stuff, although it was making us money. And it's very important. They saw it as, well, I've got lots more to do there. I'm not done with that product yet. I really want to make that thing better. I want to push it. And then some other people, I really want to jump into the cloud and I don't know any of it, but I'll learn it. Let's go for it. So that worked out really well. So try that. If you run into this kind of situation, leave it up to the team and see what happens and then you might have to deal with some consequences but it's worth a try. So we just kind of jumped in the deep end. And what that means was we essentially just did just-in-time research on each of the bits of technology to see how they work, where the holes are, you know, kind of where's the quicksand. And I'm usually the kind of scout the commando on the drop-behind enemy lines. So I kind of looked at each one of those things just ahead of the team. So at least I could say, well, here's a major pothole coming up, or this just does not work. That's not even used out. Let's steer this way instead. So that smoothed out a lot of bumps and kept us from wasting too much collective team time, all kind of working on one nasty problem together. So it smoothed the road a little bit. So we did that for the UI, for Rails, for CoffeeScript, Couch, the Asset Pipeline. I've got more to say about a bunch of this stuff later. Insecurity, and that's mainly encryption. So in transit and at rest, because in the cloud that's definitely a best practice for security. We encrypted everywhere. So all those things required testing new components, seeing what worked and what didn't work very well. And we did tweak our agile process a bit. I think we heard from GitHub yesterday about this. They did something similar, or do something similar. We used to be all story-based, and we moved to more feature-set-based. So it's kind of clumps or clusters of stories that make up a very small but cohesive release that we can put straight into production. So that might be one story. That might be five or seven stories, but it's kind of in the ballpark of, you know, hopefully a week or less of work. So we're not pushing the production 40 times a day. That sounded interesting. Gave me a heart attack a little bit to hear that, but we do try to push a couple times a week if we can. And the other thing we did is we split out DevOps work. So there's a lot of stuff that has to be done for a product like this that's not code features sort of directly visible to the end users. So things like setting up our DNS, global DNS, round robin-ing, round robin-ing, SSL certs. How are we going to send e-mail? Do we want to use a service for that? Do we want to set up our own e-mail agents? What do we want to do with that? How do we want to deploy to production? Somebody's got to write all the rake scripts to automate this. All the monitoring configuration, all that stuff. We decided let's carve that out into a tiny DevOps team of two sort of part-time people over a couple months to kind of attack those issues and integrate with the team and bring back any extra development that might need to get done to them but at least pave the way and do most of this work on their own. And that's actually worked quite well. Okay. So now to talk about the tech, what is all this stuff? So kind of a brief poll. You can raise your hands and keep them up if you applied all these. Who's already been using Rails 3.1? Okay. We're about Rails 3.0. Okay, so most of the room is on one or the other. Coffee script. Decent amount, maybe half. Counts DB. Handful people. A couple dozen. What about jQuery? Everybody, yeah. Prototypes dead, right? What about SAS? Decent amount of people using SAS. Okay, so good mix. Maybe you haven't put them all together quite yet. So I've got some things to say about this. Hopefully it's something new. So on Rails 3.1 side, some interesting new bits. There are page streaming, which we just heard about. Kind of the concept of doing things more dynamically, more asynchronously. In this case, page streaming in Rails 3.1 says, well, let's go ahead and start pushing the page out to browsers so they can start loading JavaScript, loading style sheets before we even render the bulk of our page, before we've gone through the whole Rails stack. So it gives a kind of the sense or the appearance or performance, perception of performance, which is just as good as a real thing as far as a user's concerned. Their page starts loading faster. That's what we want. The asset pipeline, really nice way to sort of pre-compile down your SAS style sheets, your coffee script and a JavaScript, et cetera, and make that easy to push into things like CloudFront and other content distribution networks with your Rails apps so you get good fast load times for users everywhere in the world. So it's a good start. There's definitely some issues with this, which I'll talk about. Rack, cache, e-tag, and conditional get support is all baked in by default now in the default middleware stack. So it's nice. So you get good caching capabilities baked right in. You don't have to kind of write a bunch of code for that like we had to do pre- Rails 3.0 and 3.1 days. Engines are now back. They kind of got drummed off the stage early on as being kind of nasty, and now they've made sort of a full comeback because they have good support with the refactored Rails 3.0 and 3.1 code so they can have their own sort of isolated, almost like an isolated mini app inside of your app where they can have their own configuration and own views and kind of nice, cleanly separated plugins and hooks and things, but still work with your app. So they're back. SAS, jQuery and CoffeeScript are now there by default, which is great because we had already decided to go with all three of those technologies back with Rails 3.0. So I was very happy to see those choices. Backbone was another... Who's using Backbone? Anybody? A few people. Okay, more than I expected. Backbone, kind of a lightweight client-side MVC so you get Model View Controller in JavaScript on the client side so you can have more of a robust and separation of concerns kind of way to run your app if you've got a heavy or UI-rich front end and then synchronize with the back end, almost treat the back end of front end like peers as opposed to one being sort of the lesser child or lesser brother. CoffeeScript, I think most people know, but it's essentially a new scripting language that translates JavaScript or translates into pure JavaScript and it's got a lot of flavors and features from Ruby and Python other languages baked in. I'll show a little bit of that in a while. jQuery, kind of de facto standard. And SAS, if you haven't used it, it's a superset of CSS3 with all kinds of awesome stuff like variables and functions, overrides, inheritance. It's very dry. It's exactly what we've kind of always wanted. And then Couch. Couch, if you're not familiar with CouchDB, is a document store with map-reduced views. Definitely not anything like a relational database. And I'll talk more about the differences later. The one kind of nice feature is you get a safe shutdown via kill-9. Your data is pretty much always going to be safe just the way Couch works under the covers, which is good if you're putting things in the cloud like on Amazon EC2 where things just disappear. Okay, so pros, cons, and WTFs. We'll take a look at each one of the technologies and see kind of where they're good and bad where we had issues and what we think of them. So first of all, with Rails 3.1, back at RailsConf, it was just coming anytime now. On May 22nd, we got the great news that, hey, Final Version's going to be out in a couple weeks. Should be good stuff. Like, okay, good, this fits our timeline perfectly. It'll be out in production. We might even get a patch afterwards before OpenLogic has to go live on it. That's all good news. And then two and a half weeks later, June 8th, they said, well, here's another RC and should be out in a couple weeks. Everything goes well, great. We still got plenty of time. Not too worried about that. And then two weeks go by. And another two weeks. And another two weeks. And I'm starting to get this money pit feeling. Anybody remember that movie? Fixing a house. When's it gonna be done? Just two more weeks. Two more weeks. Anytime you ask, it's two more weeks. So I started to get a little nervous. And two more weeks go by. And then two more weeks go by. And then it actually came out two days ago, if anybody hadn't picked up on that. Woo, it's finally out. Unfortunately, it was well after our first beta release to production. So we're still on an RC. Good news is it's out now. Bad news is now we have to figure out, well, what's changed and what's broken. Because basically every RC either randomly broke and fixed things that worked or didn't work or was pretty much all the way along. Especially in the asset pipeline, which we'll talk about in a minute. So anyway, it works. And the first thing I asked our team, I kind of did a little team interview for this to put some information together about Rails 3.1. And the very first comment was bundler sucks. Or I love bundler. Everybody been through that yet? The love-hate relationship with bundler? Just love. There's no hate for bundler in the room? Hate, there we go. We got having babies, love, hate, you name it. So there's some of the quotes I got from our team when I asked. It's great, it's terrible to paint in the ass. I love it, it makes things easy. So you kind of get the roses and grenades kind of thing. Tries too smart and fails miserably. It's a good quote. Get some file in the locker, DSL are good, everything else sucks. Get repeal usage is great and terrible at the same time. In fact, you can point straight at GitHub and get some point-in-time snapshot of code. It is awesomely powerful, and also will screw you hard. Which is interesting, because everybody on the team, whenever they go to install, is going to have a different code, right? Every time they do a bundle update, well, I did it yesterday, oh, I did it today, it didn't work for me, oh, that's what it was. There's this commit that happened. So powerful and painful at the same time. So it turns out that whether you like it or love it, it's kind of good for you and you kind of have to use it. So, you know, enjoy. All right, other good things, or good or indifferent things that came out with Rails 3.1, there were a lot of good improvements for Active Record. Unfortunately for us, using Couch didn't help a lot, but the split out of Active Model from Active Record definitely helps a lot. It means you can step back from a relational database, sort of fixed view of the world, and a lot of tools that they now focus on Active Model can work also with CouchDB, with CouchResq Model and some other layers in between. Now, the sprockets or the asset pipeline was a great theory. I love the concept, I'm waiting to see it work perfectly. Kind of very finicky, lots of moving parts, and again, every single beta sprocket went through what, 15 betas or something, and they're kind of changing along with the Rails 3.1 RCs and they would break and then fix and then break. A little bit painful. We had to do something like this, perform casting goals, true to get our Unicorn, NGNX, EC2 environment to actually serve static assets at one point, and that's actually the point where we're in production right now. So hopefully the final will fix some of that, but just be aware that the concept is awesome in the implementation, might not be 100% yet. Laundry list of other stuff we kind of ran into. Stack traces are now incredibly deep, as one of our guys put it. If you ever do a puts caller or sprockets kind of barfs and you look at a stack trace, now you get 10x, the stack trace used to get kind of bore through. So the stack trace silencers are your friend, but they're still a little bit of a pain. Nice new router syntax, everybody likes that. Tested form, handling improvements, everybody likes that. Now respond with has been one of those love hate things, depending on who you ask again. So respond with, if you're not familiar with that, you know the respond to where you take a format and say, well, if I'm serving HTML, do this. If I'm looking, if I need to return JSON, do that, or XML, do this. Now respond with can replace all that, boilerplate and say, I'll magically do the right thing for you. It's kind of another Rails magicism. And if you walk this straight narrow path and you do it just as they expect you to use it, it works well. And if you tend to color outside the lines or have a little bit of a special case, it's actually more of a pain than it's worth. It doesn't work very well. So again, based on who you ask, love or hate. And there's the asset pipeline right now, the way we saw it. Hopefully it'll be nice and clean and shiny with the final release. Okay, SAS on the pro side, SAS is awesome. It's what CSS3 should have been. So you get things like nice nesting. You get inheritance, you get overrides, you get variables and functions. It's all good, all goodness. And so some of the quotes I got from the team on that, SAS is freaking awesome. Never want to work without it again. However, we couldn't get Compass to work with sprockets with the RCs. So as of, you know, a whole couple weeks ago. If you're not familiar with Compass, what that gives you is, and I know I'm really downplaying all the magical things it does, but the coolest thing it does in my mind is gives you a nice little library of cross browser compatible CSS generation. So in my SAS, I can say, oh, for this element, I want a border radius of four pixels, and this guy will automatically spit out the big blob of stuff that's well for Mozilla to do this and, you know, for this browser to do that and all those, that junk that clutters your CSS, auto-generated, hidden behind that one little clean line. Totally awesome. So we're going to have to try again now that Rails 3.1 finals out. Sprockets 2.0 is the final outcome. See if those weirdnesses with the asset pipeline are resolved. But very good stuff. You're not using SAS. You probably shouldn't even listen to the rest of the talk. Starts to start using it right now. Okay. Prototype versus jQuery. So we've been in production over four years now with prototype, of course. But we have used jQuery. It started slipping in off and on over three years ago, just because libraries were moving towards jQuery. So we've been using both for a long time, and there's a clear preference on our team for jQuery. And not just for the libraries, just for kind of the feel, I guess. It's kind of hard to describe, but that's kind of the way it got reported to me. I like it, too. It's become a standard. The default in Rails Run makes it even easier. Now, one of our guys said that the unobtrusive JavaScript seemed a little weird on the way it works, because it's kind of the Rails way. There's a little bit of magic, where if I just sort of decorate an element and put this data attribute in it, then the unobtrusive JavaScript will magically come along later and do the right thing for me to attach some action and make that browser independent. Good, but it's one of those... You know, there's no direct tie in your code. Just because I put an attribute there, this thing over here happens later. It's kind of like aspect-oriented programming if you've done that. So it's very Rails-like, and it can be very powerful and a little painful to debug and track down, when things don't work, when there's not that explicit relationship sometime. I'm still a fan, though. Okay, CoffeeScript. CoffeeScript, pretty cool stuff. You get nice variable assignments. You get statement modifiers, like if and unless the Ruby... Easy ways to define functions. So I can define a function that takes one argument, x in return, x squared. I get list, easy to define. Easy to find hashes. I get var args type support. I get nice ways to ask if something exists, because JavaScript makes this a bit of a pain, right? You get a check, well, is it undefined, is it null, is it false? All this kind of stuff. So CoffeeScript is intended to fix a lot of those JavaScript wards, and all this generates pure JavaScript code. And then I get Python-like list comprehension. So you get a lot of stuff, and this is just the tiniest taste of CoffeeScript, so check it out if you haven't looked at it already. A lot of good stuff there. So it does translate to 100% pure JavaScript. It's supposed to run just as fast, or sometimes faster than handwritten JavaScript code. It does have a bit of a weird syntax, if you're not used to. Especially our team is not a Python team, so the fact that there's whites-based sensitive formatting is a turnoff to some people. So a little bit of hard to get used to, some strangeness. So we find that there's a little bit of inconsistent usage so far in our own code base of some people saying, well, I just need some really simple JavaScript here. I don't want to use CoffeeScript. Maybe I don't want to learn it. Maybe it's just a little bit more overhead, four characters JavaScript there, so I might as well expand it to 50 lines. So maybe some laziness there. We just got to tighten that up a little bit, but that's just to give you a feel for what our team's been through so far. So some developers don't really see the benefit of CoffeeScript. They kind of say, well, it's just a new syntax, or it just fixes some JavaScript words, but I already know JavaScript, so why do I need that? And I think that we might all like it more a little bit when we have a more complex UI. So as we continue to expand the product past the first couple betas, and it gets more and more rich on the front end, I think CoffeeScript will become more important when we have real classes, objects, hierarchies, and not just a collection of functions in JavaScript that work together. Backbone, client-side JavaScript MVC, like we talked about, we haven't needed that complexity yet. We decided to use it in advance when we had a need for a problem that looked like X, and we haven't run into that X just yet. So I think what will happen is because we've got tight-led deadlines, it was a lot to bite off. Oh, it's yet another new library to learn. It brings a very sort of complex set of new concepts, especially when you're already new to CoffeeScript and maybe jQuery for some of the team and everything else. It was a little bit much under the time pressure, and if we didn't have a strong need to actually use it now, we just didn't. So we may come back to that. Right now, we're happy with what we've got. Okay, on the couch side, great stability, it's cloud-safe. So one of the things I really liked about it is you can kill dash nine at any time. You can have safe data the way it, you know, right ahead, safe B trees, the way it basically stores all its data on disk. Essentially, you're not going to lose anything. Nothing is going to get corrupted, which if you're running on EC2 and you've got so many experience there, you know that things can get kind of funky at any point in time. It can disappear. You could lose a disk. EBS has issues, et cetera. So I liked it for that kind of point of view. It does affect everything, as you might expect, because every data store-related tool we have typically understands either SQL or ActiveRecord, if you're in a Ruby and Rails world-like device. And so it restricts our choices a little bit to things that understand CouchRest model, or now there's sort of a series of ORM mapping tools and shims that are all over GitHub on kind of ways to wedge in access for Couch to other tools. But it means you're going to spend a lot of time googling and looking for blogs and docs and other people's experience and experimenting and tweaking things. It's not as clean as it would be nice to see. Not a relational database, so modeling takes some time. Views take some time. You have mandatory optimistic locking. So that's kind of a big change if you use the MySQL or Postgres and aren't already using optimistic locking. Because whenever you update anything in a Couch database, you have to pass along the ID of the document you want to change and its current revision number. And when you go to say, make my change in Couch, Couch says that's not the latest revision barf 409, deal with it. So you have to realize that if you've got a lot of concurrency, if you've got something updating like some auto-update field or you're polling or anything like that and you're updating some little, even just a timestamp and you're going to have multiple places concurrently, now you have to deal with this issue. Not necessarily a bad thing, but it's a mindset kind of change if you're used to pessimistic locking. And as an example, we're used to MySQL and Postgres as a very senior team. We've got Solar and H-Pace and all kinds of stuff running around. Moving to something like Redis was a no-brainer. Moving to CouchDB definitely takes effort. More work. So because we're in the cloud, we decided to go with Big Couch. By the way, they don't have any logos, so I decided to come up with some good ones for them. There's the Big Couches. It's a clustered CouchDB. So it's based on Dynamo concepts, like read and write, write quorums and replication. So it's just what we need. It's open source. But it was a little more work and expensive we wanted to buy off initially to bring up our own cluster of full-time in the cloud to handle that when we found, you know, Cloud and the guys that created Big Couch offer a service for hosting Big Couch. Now we do have some performance reliability issues that we kind of worked through early on and are still working through a little bit because it is slower. You're clustering the database now. So if you want it to be really safe, you've got to write to multiple databases, get a response before you can assume the right happened, which of course is going to slow anything down. So it's a little slower. You've got to think in terms of documents, not rows. You want to minimize roundtrip because now we're using a database as a service, essentially. So even though they also host in EC2 East, which is where our first cluster is, you're still going out across machines and going through Amazon's network and there's some overhead in the stacks there. So you need some caching. You need to think about minimizing roundtrip. Other things we use, just kind of bonus items. I couldn't cram that first title. We do use device. Who's using device? A bunch of people. If you're not using it, it's really quite awesome for authentication and setup. It supports a zillion standards for OpenID and et cetera for single sign-on. Documentation is not perfect and it takes some work to integrate with Couch. Again, there's some shims and other layers that we got working, but we wound up submitting a lot of patches and things back to the various communities to smooth things out. Machinist. Anybody use machinist? Not very many people. Kind of a competitor to Factory Girl for better fixtures. We did use Factory Girl for a long time and decided to move to Machinist. It's kind of like maybe the grass is always greener a little bit. Close, but not exactly identical to real object testing. Capybara and WebDriver, still a little impedance mismatch there between real browser testing. All these are changing. None are perfect. I have a little picture of bugs there because they're not big bugs, but they're little bugs. They take time. Let me wrap up kind of quick. EC2 and Rackspace, great for firing up new servers in a clean state. Really easy for developers to test stuff. A better VMware than VMware is what some of our guys called it. Disposal snapshots. But things like fragmentation is really annoying, the fact that you can't use one image across regions. Things like that are really kind of aggravating. Also load balancer issues across availability zones. The documentation says one thing, it actually does something else, and what it does is not as good as the documentation. Ugly, things hang, themes won't start, random slowness, inconsistencies abound. So you've got to expect when you're living that worldy design for failure to take it right in. So kind of final thoughts. It's all business as usual. This is what we do for a living. It's always a tight deadline. There's always a lot of pressure. There's always a bunch of new stuff, and it comes together last minute. No surprises there. But if you haven't experienced that world, if you haven't been doing this for years and expecting all those weirdnesses to happen, don't expect smoothness. Weird stuff will happen. Cloud services can help. They sped things up for us like cloud, and things we're looking at, like a single worker for async job management, dying for distributed DNS. It just saves you time if you're in a hurry. And of course this wouldn't work without open source. We use everything we can get our hands on. So if I already blown my time, we're gonna have any time for questions? A couple of questions. Okay. Anybody? So a couple of comments there. One is the compass support for Rails 3.1. There's a new alpha. The new version always fixes everything. A new alpha that'll magically make all that paint go away. And then respond with, it sounds like it may be dropped entirely for Rails going forward because it's low in A. Any other, any other questions? Okay. Did we look at MongoDB versus CouchDB? I'll make a good controversial statement here. I'm using CouchDB because I like my data. And I prefer when it's actually the same as this and doesn't just disappear and get corrupted and share like that. The Mongo half the room is not clapping. I don't know. Mongo is very fast for a reason because if you don't do things like, you know, bother to commit to disk and have things like that, it's really fast. So are hash tables. Yeah, total fun. It really requires quite a deeper answer than that. Until very recently, however, Mongo, I know had all those issues, absolutely. Now, as of the latest release, major release in theory, these things are changed. Why I'm a little nervous about that is because it is a new major release and there's no time yet. So those problems may be completely gone. I don't know. And so I'll prove it otherwise I'm suspicious. Any other questions? Okay. Well, thanks a lot.