 So our first talk this morning is from Rusty Birchfeld of Z-Vents, along with Josh Tyler with him. And they're going to be talking about some really massively large storage solutions for working with Active Record. Okay, take it away. Hello, thank you for coming. I'm Rusty from Z-Vents and basically we have things to do in your area and I'm going to be talking about hyper-table and how we use it. And to start off with I'd like to just show you, give you an idea of what kind of stuff we do with it. So basically here we have a report for the past month. This is hourly data that's queried directly from hyper-table live. So I could actually change this to say, okay, I want the 16th and execute that and that's pulling back a full month's worth of data hourly and displaying it and I can compare that to the previous month. So that's thousands of records retrieved and rendered into the web here in just a second or so. And I have a couple other examples here. This is the same thing but daily and here's for a movie, lots of data and some performance numbers. We actually, this is actually an old benchmark but on a cluster of nine machines we achieved one million random inserts per second sustained and that's using AOL query log data that was released a long time ago. So what is hyper-table? It's an open source big table implementation. If you're not familiar with big table it's used at Google. They released a paper on it a number of years ago and hyper-table is available now for anybody to use rather than big table which is of course only for Google. The basic premise behind it is that it's basically a big sorted map and you can, using the keys you can represent multiple dimensions. So there are five parts to the key and each of these parts you can think of as a dimension. So the row key is the main index, that's the only thing that's indexed, that's part of what makes it fast and then the column family is what you would think of as a normal column in like my SQL or Postgres. And the column qualifier is data actually. So you can store data in the key and that allows you to create some interesting data layouts where you could have like a tag name in the qualifier and then the number of occurrences of that tag for instance in the actual cell. And then timestamp and revision so you can actually store version data in there as well. So you could have several cells with each of the versions of that cell over time. And currently it only stores strings which is usually not a big deal. This is the basic architecture. You have the master server and a bunch of range servers that are all running on different machines usually and all of those machines are talking to HDFS, the Hadoop distributed file system. It works with other file systems as well. And basically you have rows represented by a row key and that row may have many cells in the database. It's far so you don't have to have data for every single cell in that row. And those rows are in ranges, row ranges and the ranges are distributed amongst the range servers and the master knows where those ranges are so when you need some data you ask the master where's the range for this row and it will tell you which server to talk to and you retrieve the data directly from that server. If a range server goes down then the master will bring up another range server somewhere else and tell it where the data is on HDFS. So that's sort of an introduction to what HyperTable is now for some information on how to use it. The primary interface is a CAPI but that's not as interesting as the thrift broker interface which actually provides access in a bunch of different languages including Ruby of course. And that's actually what we use a lot at Z-Vents. There's actually a hyper record interface which is an extension of ActiveRecord to support HyperTable. So this is actually what powers those graphs you saw earlier. And basically we had to subclass ActiveRecord because there is SQL in ActiveRecord and HyperTable doesn't support the same SQL statements. So I have an example for you of a small application. I had to restart my laptop here so if we look at this currently there aren't any pages listed in here. This is just a small Rails app that's using HyperRecord but I'm actually going to load up the hyper table shell here and you can see there aren't any data in pages but I'm going to load in some data. This test.out file here has the first 10,000 articles from Wikipedia. So that's the title and the body of each article. So we can load that in. That happens pretty quickly. This is on my laptop here so it's not the speediest thing ever but it's approximately 150 megabytes of data. And that just loaded in 14 seconds at approximately 14K per cell, 600 cells per second. And now if we go over here this will probably take a moment to load because there are 10,000 pages now and actually we could do, well here we go. So if we look at the page for A, there's all the data from Wikipedia. I can't render it but lots of data. So basically there are a bunch of considerations, things you have to take into account when you're dealing with a database like this. It's not your standard relational database. It can't handle joins or secondary indexes. So a lot of times you have to denormalize your data sort in the format that you intend to query it rather than in this normalized format and then doing a whole bunch of abstract or arbitrary queries on it. So there are lots of technologies like MapReduce and so forth to allow you to aggregate your data in various formats and we actually make use of those as events. And those are really useful for providing those denormalizations. Column families and qualified columns using these different, using the qualifiers you can store data in the column name. So for instance, I already gave an example on that but basically the data that we are looking at here, the format for this is the row key which contains the type and the event and the ID. So in this case it would be like movie underscore or whatever the ID is. And then we have a column called detail views and then there are qualifications on that column for each time that we're dealing with each hour and then in each hour's bucket in each cell we have the value for that hour. So here you can actually see the qualifier for the column and then the value for the cell. And then of course you can store version data as well and HyperTable even uses that to represent deletes. So to make a delete fast instead of actually going and removing all of your data it just inserts a new cell saying that this cell has been deleted and then a maintenance job comes back later and cleans up all of those deletes. So basically that covers it. Questions? So the question was how do we break down by hours on our chart? So well the data is stored per hour and for instance like the daily data we're actually aggregating that on the Ruby side. It's much less efficient to do it that way but HyperTable has been so fast for us we haven't had to create a new denormalization for that yet. Okay, yes. So that string is the column qualifier and so since HyperTable doesn't support anything other than strings even though there's a timestamp field there the timestamp is actually stored as a string and it can be lexically sorted so that it actually comes into the map as a key that sorts reasonably. Sorry, what was that? No, I'm not storing the images. I just took a sequel dump and pulled out only the title and the body. All those inserts you saw, so that's 9,999 cells. All of those cells were the bodies of the articles and then the row key is the title. So the key size here 15 bytes that's the average number of bytes in a title and then the 15k is the average size of a body. Is that answered your question? So this was just a crude example but if you wanted to store the images then you could just have another cell that would be like the image cells. You'd have the body cell and then like, or I guess, so an image, I see what you're saying, images are just of another page on Wikipedia. So in that case the body could be the actual image data if you wanted it to be. So it does support a certain amount of the things that you would expect from sequel. It does support limit, but it is somewhat limited. Most of the focus has been going into like the back end and then opening up interfaces that are like the thrift interface and stuff like that. Well, the primary thing is for our metrics right now we have a ton of reporting and then we also, well we actually use cascading a Hadoop framework to load all of our, to process our log data and convert it into a format that we then load into hyper table and that's how we actually get that data. So basically with cascading instead of having to specify maps and reduces individually, which you would have to do if you're using map reduce directly, you specify the operations you want to compute. So you say like, okay I want to do a filter, I want to do some arithmetic or whatever and then I want to group in some or whatever kind of aggregation you want and then the framework will schedule that and optimize it down into as few map reduce jobs as necessary and run those on the cluster independence order. And then basically we take the results of that and those go into hyper table and one of the things we're actually working on right now is a thrift connector for cascading so that we can connect our map reduce jobs directly to hyper table and then we can load data in from hyper table and directly back out into hyper table and that'll allow us to do a lot of neat things with deormalized data. So if we want to aggregate up data from hourly to daily buckets, we can then just run a job to collect all that and stick it right back in without having to hit the file system. But I do stuff. There's uses of hyper table that we use to see this, one of the things that we can actually, we're really good at, oh sorry, yeah, yeah. The data line, we have a section here. Well here's some, this is like the code that generates the detail stuff. So basically there's, this is actually Java code. I guess I should have said that. The cascading stuff is in Java. There is a groovy implementation and then somebody took like JRuby and recreated the groovy DSL and JRuby. So that's kind of cool too. But that came along after we already started this stuff. But basically the idea is that you can do things, I guess a lot of this stuff is abstracted out. But this is a sub assembly that does a bunch of work, like filtering and stuff like that. But here's joining two keys together, two fields together and then here's a group by and this does a count and then basically it gets spit back out on the file system. So Rusty, one other thing I was going to mention is maybe our use of the change log. So one of the things that we were really excited about getting hyper table into production for is, you know, it's events we deal with a lot of user created content and a lot of times things change and we don't know why. We have a lot of, you know, malicious or inadvertent kind of changes that happen into our core production database. And so we wanted to build this thing we call the change log, which basically logs everything that ever happens. It's almost like if you took your web logs and just wrote them into a database that you could query arbitrarily. And so this gives us an incredibly powerful tool. I don't know how many writes a day we do on that one, but an incredibly powerful tool to track everything that happens from, you know, somebody added an image, somebody changed a description, somebody logged in, whatever. And so I think, you know, that million inserts per second, that's something that we haven't quite reached in production. You know, that's a level of success we can only dream of as a web property. But we're prepared for that volume of data, which is really exciting for us. I don't know if you have anything else to add about that. Or if you wanted to say anything about how, if people are interested in getting started with HyperRecord, what they could do. Yes, so the code's all up on GitHub. There's actually both HyperTable and HyperRecord out there. And so HyperTable itself has actually gained a pretty good following in the open source community. ZVents is a sponsor, also Baidu, the Chinese search engine has come on as a sponsor. And there's a community growing around that. HyperRecord's doing pretty well. Obviously, we didn't build too much salesmanship stuff into our presentation. We were more focused on getting that demo working. It's, there's a lot of stuff going on behind the scenes just to see that one little thing where it inserted 150 megs in five seconds. But, yeah, we encourage people to get involved and try it out. It's a very active community there. Any other questions? Can you describe some of the things you were trying to do, I'm assuming, with relational database technology, with SQL database bursts that basically fell down and maybe decided to do this, or was this based on fast experience like that? Well, so we didn't actually have like a database keel over on us. We wanted to sort of cut that off. And they, we started HyperTable development like two years ago. Yeah, this is a long-term effort for us, actually. And basically pulling something like 3,000 or 4,000 records in like a few milliseconds or whatever. I don't know that that's something that... You know, it, yeah, we didn't actually hit a pain point where we did croak. It was more like we saw that we were going to have to do something like this to get to the level of... It's really about analytics for us because a big part of what we do is driven by search relevance. And we really want to build in a lot more sophistication into what we do there rather than just, you know, looking at tags and time stamps and all that stuff. We have a, this tool is going to give us the ability to build a lot more into our analytics engine that drives local content because the search for events is actually interesting. It's not a free text search. It actually has many dimensions. There's time, there's location, there's content. And then there's all this user data that we want to feedback into it. So things like the change log, how many times did this thing get accessed, how many times did it get changed, all this stuff. We want to build that into building a better search result. And so we saw, you know, there's going to be more data than we can handle with any relational database, even if we do all the, you know, replication charting, whatever. We wanted something that would really stand up in the long-term. I just wanted you to see what kinds of problems you think are well-suited to hyper-table versus a relational database. Like, are there, you know, is it now that you're using it for this to use it for everything, or? So we're trying to get our site moved over to it. It's kind of one step at a time. But sort of the canonical example for a use of hyper-table is a crawl database. And there's actually an example of that up on the hyper-table wiki. And if we go to the wiki, basically the amount of data that you can store in it and the flexibility it gives you with storing that data is really powerful. And I think it's in. That's the performance data. So the crawlDB example, basically you have a title and a content column. And then there's an anchor column. And the anchor column has qualifications. And by putting the domain name here in reverse makes it easier to, it sorts better. So you can look it up faster. But basically you can figure out all the anchors that point to a given page very quickly. I guess what Rusty is saying is that if you wanted to crawl the web and put it into a database rather than just into a text file that you would have to post-process extensively, that's a good canonical example. Also what we do with the changelog, anything where you've just got mountains and mountains of data and you want to be able to preserve some semblance of real-time interaction with that. There are limitations because it's not truly a relational database. So you'd have to, I mean, any particular application, you'd have to see if you'll be okay without those. Well, thank you. Yeah, please talk to us about it.