 Thank you everyone. Can you guys hear me in the back, okay? Jim, can you hear me? Good? Okay, cool. So as Josh said, my name is Brian Helmkamp. I'm the CTO of a company called Efficiency 2.0 in New York. And we have a pretty cool project we work on where we use Ruby software in order to help people be more energy efficient, especially with their home power bill. And we're also hiring Ruby engineers in New York, so if that's interesting to you, I'm finding. The slides for this talk are available online already through this Bidley link that will take you to SlideShare. My Twitter handle is brianary and brianary.com used to be my blog, which is now 400 page, and it will be back up and running at some point soon. So that's aspirational. So this is the agenda. I've given this talk once before at a Ruby group, and I think it took me about 45 minutes. So the only, I had two weapons at my disposal. The first is coffee to try to make this go quicker. The second is to try to get a sense from you guys about what you're most interested in learning about, and also sort of what your familiarity with the subject material is so far. I'm going to kind of start there. How many people have heard of Aero? Okay, so that's pretty much everybody. How many have used Aero not through an active record? So, you know, looked at the code or manipulated the objects correctly? Okay, so just a few. So that's kind of the audience that I'm going to try to target is people have heard of it, but haven't maybe used it directly and show off some of the power that's available on active record 3 now that Aero has been integrated into Rails 3. So we'll look at what is Aero? We'll look at a little bit about relational algebra, the conceptual framework that Aero is built on. We'll look at how Aero is integrated into active record 3, pop the hood on that, look at some of the implementation of how that works and some of the features which are maybe a little lesser now, but that you can take advantage of in your Rails 3 today. And then finally we'll get into the rainbows and unicorn section of the talk if there's time. I'm looking at some pretty interesting stuff that I think might be possible in the future now that active record has this relational algebra base. So what is Aero? It has nothing to do with mermaids. It is an object-oriented interpretation of relational algebra written in Ruby. That's how Nick Callan described it when he was originally working on it. Nick is the original author of Aero. And this project actually goes back a fair bit of time, which most people don't realize. As I was preparing this talk, I went through the Git history and the first commit on Aero is actually in December of 2007. So while Aero is getting presented as a relatively new piece of code that you can start to leverage, it has existed in some form or another for some time and the concepts that it's built on actually go back even further than that when you look at things like namescope, which we're going to talk about in a little bit. So one thing I wanted to emphasize is that Aero is not an ORM. Active record is not a relational mapping engine. Aero could be considered a toolkit on which to build an ORM that's not an ORM itself, which means if someone, let's say, data mapper decided they needed a better way to integrate with relational data scores, they could use Aero and that would be just fine. And that's kind of an important point in this project. It's not just like, you know, I don't think the name can be confusing in this regard. It's not just a sub-project of Rails. It's its own unique project which hopefully will have applicability far beyond active record over time. Although active record is the first ORM production-ready ORM to implement Aero and that's a great accomplishment, which I think is one reason that active record is awesome of many. So if you're interested in learning more about the origin of Aero, Nick Cowan has a blog post on his blog called Natural Scaling Sprinkles about why he wrote it and recommend you check out that link. So I wanted to emphasize that Aero is a largely community-contributed project. There's a lot of people, including people in this room, who have done a significant amount of work on making Aero production-ready. And without that, we wouldn't be able to take advantage of this today. So there's lots of contributors. I want to take a second to thank all the Aero contributors, Nick, for being the original author. We wouldn't be in this position to be benefiting from this functionality of looking for guys like these. So how does Aero integrate with active record 3? This is a kind of high-level architecture diagram. Active record 3 sits on top. That's the interfaces that you know and love, which are an evolution of the Rails 2 active record interfaces. Below that, you've got active relation, which is split up into two components. You've got the algebra, the code that implements our relational algebra engine, and then also the engines, which allow you to execute those relational concepts against the database. Finally, you need a connection adapter or something. Somewhere in the stack means to be able to talk to your MySQL or Postgres on a socket. In this case, we're still using the Aero connection adapters, which are largely unchanged from Rails 2. And then that also sits on top of the database, which runs a separate process, as we all know. So relational algebra 101. Let's cover the first part first, which is what is a relation? A relation is composed of two components. There's a header, which is basically a list of columns. This shouldn't be familiar to everyone who's seen a database, a diagram before. Each column has a name and a type, and then it's a bunch of tools. So you can see here that this relation has four tools associated with it. And you can start to see how this maps to SQL concepts. This isn't strictly a SQL concept itself. You could represent a relation with an array of arrays in Ruby, and sometimes that's really useful. But if you think about how the concept of a relation maps back to things that you're used to in SQL, you can see that this looks a lot like a table, like we said. This looks a lot like a view. And even just a query result set, which you run on an ad hoc basis and then you throw away, it maps to a relation in a way. In an active record, the concept of a class and an association, you can think about those mapping relations as well. So what can you do with these things? Selection is one of the most common operations you would perform on a relation. This is analogous to the where clause in SQL. You're going to restrict a relation into a new relation, which is a subset. Projection limits the columns associated with the relation. So let's say you have a user's table, which has 15 columns, and you're only really interested in the first name and the last name. You might say, select ID, first name, last name, from users. That's called a projection in relational algebra terms. There's joins. I'm not really going to go into a deep dive on this. There's lots of stuff here that's pretty interesting, but just know that joins are part of the relational algebra, and they work largely like they do in SQL. And then you've got your set operations. There's many intersections and difference being the primary ones. These should be pretty clear to everyone. They're even implemented on the array clause in URB. So what's most cool about relations is the concept of closure. And closure means that if you take a relation and you perform an operation on it, you get a relation out the other side, which means you can continue this string of transforming relations into new relations basically infinitely, and you still get all the nice things to work with. And this slide illustrates how this is manifested in code in Rails 3. So this is real Rails 3 code. If you call Ruby, this is an active record-based class, mapping to a table. You can get the relation out of the call.scope. You can add joins, where, limit, and eventually you can run the query, and it works just like you would expect. This is implemented in the code as a composite pattern, which is pretty cool, and relations are also unbeatable. So that's why I have to do this assignment on each line in order to make sure that we're capturing the new results out of the other side. So what's in this? What's in it for you today? The fact that active record is now based on ARO. The internals of active record got cleaned up a lot, but I'm not sort of perceived on the assumption that you largely don't care about that. You're not an active record maintainer, but the way, in cases where the implementation of active record can be cleaner and be more useful to you, that's interesting. Also, just generally, there's a nice new query syntax, which is probably the number one thing that people have seen when they've heard about ARO, that's in Rails 3M. I'll cover that first. So if you go back in time, originally there were hash queries, and this is a Rails syntax for, as long as I can remember, you would find all, and then after that you have this hash, and everything you need to care about in the query needs to be represented in keys on this hash. Conditions being one of the most important. And conditions itself can be a hash when you need to do the quality lookups, and you don't really care about order. But as soon as you need to do something more sophisticated than that, for example, looking for an age greater than 18, you have to, the hash doesn't work. There's no way to represent the concept of age being greater than 18, and then a standard would be hash. So, how many times have you written a query that started with the conditions hash, and then pretty soon you just have one thing that you can't express in the hash, you need to rewrite that whole query using the array interpolations syntax, where you write out all the SQL and then substitute in the values, right? That happens all the time when you're writing an app, at least to me. In Rails 2.1 we got name scopes. This is a kind of a convenience where you can declare a query to be named and go back and reuse it later, but it still suffers from the same problem that the original syntax did, and this is actually a supplement, a replacement for the original syntax, in that you're using, you're forced to use hashes for all of the definition of a query. So, fast forward to today. Rails 3, as you all know, is shipped. And this is an example of a sort of syntax you can leverage in Rails 3. So you've got a Rubyist, I'm gonna say I'm gonna find Rubyist where the city is NYC. If you just look at that line of code, to me this feels almost as expressive as this concept can be translated into code. There's really not much to cut here in terms of boilerplate, like symbol conditions, anything like that. I'm not sure how much better we can get with this given your current Ruby parser. And if you look at what comes out of that, you're going to get a relation class. Which we'll talk about a little bit more as we go. So, as we saw, relations have the property of closure under most operations. This can apply to your active record code because you can chain different conditions for other query forms on top of each other and build up a query over time in your code programmatically executed. So, this is just a simple example where you restrict a set of users to go modify another criteria. You can also reuse relations later. So, you can create a relation which represents some concept and then go do something else with it to run one query and then go modify it in a different way to get a new relation to run a different query without having to rebuild the server description. One of the coolest things about relations in Rails 3 in my mind at all, positively affect your code as soon as you upgrade, is the concept of laziness. So, this is near and dear to my heart. This is the common pattern where you're going to run a query in a controller or at least you want to get a reference to a result setting controller. It's probably the best way to describe it. And then you need to iterate through that result set in your view and generate HTML. And sometimes that generation of HTML is slow. So, in Rails 3, when you say rubius.wear city is NYC, the query doesn't actually execute. And when you say nycrb.each, at that point you've now iterated over the result set and the Ruby code has to go back to the database and get a result set at that time. So, the concept to remember here is that at the point you iterate over a relation is the time the query will execute. And you can think of calling dot first or dot last as a form of iteration because the only way to know what the first or last thing is is to get some idea of the set. Now, that was all fine. There wasn't really a problem in the previous slide in Rails 2, but what if you, you know, it takes a long time to generate each list item for each Rubyist. You might hear me use the fragment cache, right? This is kind of an idiomatic Rails way to improve the latency page. And in Rails 2, at the time, if there's a cache hit, the block doesn't run. But in Rails 2, the query will still be executed in the controller. In Rails 3, with an active record 3 built on Aero, the select never happens because the each never gets called. And the each method is the method that would actually trigger the query. So, you don't have to get to these weird situations where, you know, you've added fragment caches but now your queries are all still executing. So, yeah, you've saved your HTML generation time, but you've got these extra queries. Now what do you do to get out of that? You introduce presenters and that sort of thing, but sometimes the conceptual overhead of adding another layer like a presenter isn't really necessary. You just don't want to run that query if the cache is hot, right? So, you can get that for free with active record 3. We saw the finer method where a few times I had limit in one or two of my slides. But basically, every one of the hash keys that you can use to compose a query in Rails 2 corresponds directly to a method that you can call on either a relation or an active record base class in Rails 3. So, you've got where, having, group, order, limit. All those things that you're familiar with for Rails 2, passing in as keys, you can now call a method and get back a new relation that has that restriction applied to it. So, one thing that's really cool about relations in my mind is that they quack like an active record base, right? So, that means that if you have a relation object that you've built up over time, you can call a method like update all on it and the exactly what you would expect will happen. A single query will be executed updating all of the records that match that relation with the updates that you described. You can do a delete all, you can find records by one or more IDs in that set. You can check if anything exists in that set. You can already use in Rails 2 on active record base, but you have to give them a big hash to make them understand what records you care about. In Rails 3, you can now have an object that represents the relation of records that you do care about and then call these things on it later. So, that adds some pretty nice, it turns out to be a pretty nice way to clean up a lot of code. So, next, I wanted to run through kind of a deep dive on an example of how the integration of the AREL into Rails 3 has simplified a lot of concepts and made things a lot more accessible. I know name scopes, at least to me when I first heard about them, were kind of a scary concept, right? It was a little bit difficult to get your head around how all of this query information was getting assembled together into, you know, a query that actually ran against your database. If you pop a bit on the code, that was kind of reinforced by the, you know, some of the gymnastics that had to be done in Ruby to make all that stuff work. So, let's look at what happens when you run scope NYCRB to define a name scope, now the method called scope in Rails 3 with a condition that the city should be NYC. So, the first thing to keep in mind is that the code on the previous slide is exactly the same as the code on this slide, right? Self is the implicit receiver and since we're in the scope of a class, the self actually means the Ruby is class. So, you can add in the self and get a better idea of exactly what's happening and then, if you replace self with Rubyist, you get this line of code which does exactly the same thing to the Ruby interpreter. Rails doesn't know anything about those variations because it's opaque. Ruby will run all of those things exactly the same. So, now you can get a sense that you're creating a relation just like you always would and then you're calling scope with a symbol to name it and you're passing in the relation and it gets set up so that you can reuse it later. Now, one nice property of that that becomes apparent is that you could build up the relation in any way that you need to and then just pass it to scope at the end. You don't have to, on the right side of that method in vocation, use a method like where you can just give it any relation. So, if it takes four lines of code to build up exactly the relation that you care about, you don't need to cram that all on to the end there. You can just create a local variable, build up the relation using the four different calls that you need to and then call scope with the name that you want to assign it to in the relation and it will just work. So, what happens when you call scope? This is kind of like one of those black boxes of Rails magic that might inspire fear in the hearts of men. But I want to show you that it's not that magical. So, this is the Rails 3 code verbatim or scope at the time I was putting together this presentation. It's a fairly big method by my standards but it handles a few different types of inputs and we're going to sort of strip that out, strip out anything that's not related to the path that we're running with the code that we just looked at and see what happens. So, we're going to move all the code which is totally necessary for the block form of scope. That's what you call scope NYCRB do and then you put a bunch of stuff in the block. We're not worried about that right now so I just stripped that out. Second, I'm going to strip out everything related to the hash form. So, this is kind of a carryover from the Rails 2 days. You could pass a hash of a bunch of query options to scope but we don't really care about that. So, this is the essence of the code that needs to run when you call scope in the syntax that we just saw if you select that. So, you can see here, it just grabs the name it validates the scope name valid scope name with a question mark actually I think raises an exception if the scope name is not valid so that's kind of confusing. Either that or it's a bug because nothing checks the return result of that method and then it creates there's scopes which is a hash and the values in the hash are, you know, lambdas which are formed just by taking scope options which in this case is the second parameter of scope and our specific example is a relation and it merges that in the scope. Now, scope is, as we saw on the first slide, verbius.scope just gives you back a relation representing the entire table. So, it merges the default scope with this new information you've provided in the lambda and then it really just calls singleton-class.redefine method with the name and that lambda. So, really, if you sort of boil it all down and you call scope NYCRB where city is NYC you could really just write that. So, you know, it's convenient to use scope as a method to define these things, but there's nothing that magical going on now because introducing ARL into active record 3 has more closely aligned the internal representations of these query objects with what you care about in your code and as you can see a relation object which you create in your code just gets carried down the stack until you define a method which returns it and when you call NYCRB now on Rubyist like any other relation. So, this is the part of the talk where I have to put up a big caution flag. What I'm going to do now is look at some of the ways you can use the ARL API in ways that are quickly unsupported by Rails. So, for the sake of the Rails core guys and me, please do not report bugs on anything post this slide to the Rails like us. And, you know, obviously this code may be subject to changes in the ARL API which changes at different rates than the Rails API. So, we saw a lot of active record relation objects in the previous slide but we never really saw anything namespace under ARL so where are the ARL objects? What I wanted to show is first there's an ARL table method on every active record base class is return the ARL table. An ARL table is just a subclass of the ARL relation of the relation that's backing that class. So, ARL table is a really useful method and every ARL relation defines a square bracket method to take the name of attributes and return an instance of attribute. So, in this case we're passing in a simple population and we're getting back an ARL object that represents the population attribute which is an integer. So, we can get that back by calling square brackets on the ARL table. And finally an attribute method defines methods like an attribute object rather defines methods like gt for greater than which we'll return predicates. So, in this case we say give me the attribute for the population of a city and then create a predicate for greater than 750,000. So, we're going to do a few more exercises using this functionality but in order to make the code a bit more terse I'm going to apply this patch to active record base. All this does is provide a convenient way just given an active record base class to go through the ARL table and return the ARL attribute without needing to tack on dot ARL table on every one of our calls. So, let's take a look at how this expresses itself. At this point we're going to run a query which we're going to run against the city's table to pull back all the cities with population greater than 750,000. So, this is nothing new. You can do this today in Rails 2 all the way back to, you know, Rails 0.1 but this does give you a way to do that without having to drop down into simple strings at the drop of a hat when you need to introduce any sort of query semantics that are greater than. ARL supports these predicates. It supports things like equality is a value in a set or not matches is based on regex less than greater than all that great stuff. You can use all those methods and they just sort of work as you would expect. And that lets you do a lot of really interesting things. So, when I walk through this example of building up a query that is a compound query it's got an OR and an AND in it you can do without having to drop down to SQL in Rails 3 today if you're willing to use the ARL API directly. So, first I'm going to create a predicate which represents, you know, a city where the name is Detroit. I'm going to create predicates for big cities and awesome cities and then I'm going to try to find cities which I might consider living in and that would be cities where the name is Detroit because it's my hometown but it's decidedly not awesome because there are 750,000 and they're awesome. When you run that, it actually does what you would expect and you can get back your results. So, that's something you can do today. Future possibilities. Unicorn and Rainbows. Now, we're getting a code which will not run anywhere. This is a code that came out of my head. But, because ARL is a solid foundation of the latest model that Active Record is now building on, there's some interesting concepts which I'd like to explore in my timeline for something like Rails 4.0 to consider if there's better ways that, you know, we can improve our object relational mapping functionality. So, the first thing that comes to mind is mapping a relation to any class, right? So, how many times raise your hand if you've created a form which needs to insert into multiple tables when you put one button. Almost everybody. So, a signup form is a great example. You need to write to the user's table and you need to write to the email address's table and, you know, currently, this is a little clunky. Like, Active Model makes us a little bit better and before that it was pretty horrific. But, now that there's ARL you can consider the idea that perhaps Active Record would let you map a class to any relation. So, currently in Active Record, how much functionality is there for deciding what a class, what relation a class maps to? If you really think about it, it almost distills down to a set table name. That's all we've got. You've got classes and you've got set table name so that you can, you know, map them to a table that's named differently. But that's it. If everything is based on relations, it's possible to create a class called signup which maps to the user's table joined with the email's table and when you save a signup it inserts to both. Now, this sounds really hard and it is hard but it is possible that once in other languages, for example, Python, they're able to do this. In fact, they're able to even figure out that because of the foreign key constraints you have on your database, I need to insert into the email's addresses table before I insert the users. So the email address I need and I put on users will actually be valid at the right time. So, this stuff is possible. There's a question of, you know, if it's worth it, but it's something to sort of expand our horizons when considering the RM layer as an extension of that, what if you could map any relation as an association? So, right now, the flexibility for mapping an association is fairly limited. You can pass some things like conditions and you can specify which class you're mapping to and what you're naming it, but you can't do something like map an association which requires a subgrade or at least you can't do it easily. So, you know, there could be a concept that a user needs to be able to access that esoteric, but in order to do a last order in SQL, you need to do a little bit of, you know, SQL magic to get that to work. You can define that in a method today, and then, you know, every time you and you can write custom SQL to go and get those last orders and put it all back together. But there's the possibility that active record could enable you to do this without any real backflips. And, you know, this is an example of a work where has one after the name of the association took in a relation and then set up any time you call user.lastOrder and knows how to traverse that relation and get the right data. In fact, you can make active record smart enough to be able to eagerly load all of the last orders for a given user set. Again, this is something that's actually possible in Python. In fact, a blog post about how to do this in Python is one of the original things that got me interested about relational algebraics, and that's pretty cool. You can just define that, you know, you've got this fairly difficult query to get the last order then, but you can eagerly load that and have just one query run. All of the objects get instantiated correctly for both the user instances and the order instances, and you can just go on with your business. The next thing I wanted to talk about in the, you know, future possibilities section of this is custom engines. So we saw that when we went down into Algebra engines, and Aero actually ships with two engines. It ships with a SQL engine and a memory engine, and each engine implements CRUD. So given a relation, an engine has to understand how to create records, read records, delete records, and update records. So for SQL there's a two SQL method that Aero will add to things, and basically that's how it does it, but you can in Aero today, there's a memory engine which can just read and write out of arrays or arrays of arrays. So, you know, we were talking about that back when we were talking about tuples. But since Aero already supports modular engines, you can imagine that Aero could eventually support engines which are not just SQL in memory, which are things like no SQL stores, memcache, which is really just memory in another process, and things like the most venerable data store of all, the file system. Now, that's kind of all well and good, but what would that actually get me? So, cross-engine joins may be an answer to something really cool we could do to this, and I want to ask you a question. I'm not sure what our results are going to get. How many people have cached or stored, as the primary source, a list of IDs, database IDs, in either memcache or redis or something similar in order to do an optimization or this sort of thing like that? Raise your hands. Okay, so that looks like maybe a third. So, I've done that before. It was a valuable performance optimization. We stored a redis vector of database IDs, and we wanted to go render a page. We did a read in Tourette's, which is very fast, and then we did a query based on primary key to the database, which is also very fast. Now, if there is a redis engine for Aero, you could imagine that someday you could define and find a way to tell active record that, let's say, the association in between users and tweets that they should see in their timeline is actually stored in redis. And then when you say user.timeline tweets to get a result set that you can iterate on the page, active record knows it needs to go get the IDs from redis, and then go to the database and do a primary key lookup and render those all out. So, that's all I've got. Thank you very much.