 I can hold it so I just I want to I want to make this really short because I'm going to talk further on this topic at the Next NYC RB, but I want to give you guys kind of a brief overview of chef Which is a dependency management system for installing Packages on servers. It's what powers the engineering yards solo and flex and it's pretty cool I think so the way that chef works is you create this what they call a cookbook which is essentially a directory full of recipes so a recipe is what is used to let's say install memcache or install a bunch of gems or You know my sequel essentially what you want to do is you want to iterate out kind of what are all the dependencies that your app requires So that you can easily bring up your server in a deterministic fashion So you define that in a JSON file that they call DNS DNA JSON. I don't know why they call it that So what you can do is you can say these are my list of packages So for bringing up we play on EC2 us plan around with we want get live my sequel the client We need a whole bunch of gems Then we've got a the repository and then what you tell it It's a little hard to read, but you tell it what recipes to use And so what a recipe looks like and I'll go into this more detail in two weeks on a Tuesday evening at B way for NYC RB but a recipe is essentially just a couple sets of Ruby code where you can say for each repository In git I want you to do some stuff like execute a git clone or do a sub module in it and For your packages, I want you to install a package and For memcache. I want you to install a couple packages and get some files So chef's really cool come to NYC RB if you want to hear more. Thank you Okay, who's next my name is Pat Nakajima I was talking earlier about the fact that I wanted to give a lightning talk But I didn't know about what I could give a lightning talk And so I decided to write some software to help me do that so basically it's called thunder because before lightning comes thunder and If I type in my github name here I can see here my projects from the github and just kind of rule them out one by one saying oh, that's not that interesting You know I want to see that and basically Since I have a lot of them it's gonna take about a click through them off So I'm gonna make fun of Josh Knowles right now and go to his because he does not have any So we've got Josh's projects here and you know, we rule them out one by one until now We know what we're talking about That's the app that I wanted to show you guys and also make the point that if you ever have any sort of little Picky-need you should write software to fix it. Thank you So I'm not gonna demo a particular product or anything. I'm really I just want to invite you to something I work for the Sunlight Foundation and we recently ran a apps for America contest where we invited people to Build and submit Applications based around a number of different APIs sort around the money and politics and influence fear And as got briefly mentioned earlier today during Ben Stein's talk the the Office of the CIO's chief information officer of the federal government has recently launched data gov Which is a it's a directory They don't host the feeds but they provide an organized way of finding all sorts of different Pieces of raw data coming out of different places in the government So it ranges from stuff like the world's copper smelter locations to the FBI's top top 10 most wanted to things like Airline like quality information over the last 30 years, so it ranges everywhere And to from the banal to the interesting and so we we're running a second contest Just like the first one it is it features a cash prize Of about $15,000 who are being sponsored by Google and O'Reilly and tech web the There's actually gonna be quite a bunch of different prizes given out There that you can see some mention there we when we ran the first contest the winner was a was a Site called filibusted filibusted And it was a it was a rails app. We actually had a very healthy Rui participation level in that contest, which was really awesome I know I'm sort of going over a bunch of things here not into too much depth as if it's a lightning talk But if you want to know more about the contest I mean the the contest is build any app Around any of the data coming out of data gov and that can really be that the you know your imagination can range all over the place We actually got one submission within 24 hours Which was unexpected Which somebody built Called the FBI fugitive concentration using the half for mentioned FBI widget so you can Try to identify Different FBI fugitives now at first this seems really absurd until you realize that you actually are really learning to memorize the faces of some of You know great dangers to American lives So really I mean this is a public service and this application is now The only application in the running for a prize of $10,000 So if you think you can come up with something even more awesome or relevant Please please do so Yes, that that's what sunlight does and that's what the government is doing Everything is becoming a lot more transparent and open and everybody here loves API's and building stuff on top of it You guys are prime candidates to Kick the pants out of this contest and help your country. So thanks guys Eric mill. Thank you Hey everybody, my name is Aaron Quinn. I'm from Brooklyn So Yeah, I have two really quick things to talk about one a couple months ago or two months ago is at this other conference called Go gay Ruko in San Francisco, and it was pretty awesome, but there was an incident We're not gonna talk about it like we said, but what happened from Which is it What happened from the conference was some really awesome things Something that I got involved with is this organization called the rails bridge Started by a bunch of people who felt like hey, you know either even if Whatever the incident, you know, whatever we want to get more people involved in the rails community and more people talking about Issues of diversity and issues of teaching kids and teaching other people about ruby and about How awesome it is and even teaching kids computer programming and stuff like that so a while ago when it happened I started a little fund I put a little money in my own in there and I got some other people to donate and it's going on until tomorrow And what's gonna go on with the fund right now? There's about $900 in there I'll hear it's at Pledgey if you just go to quirky comm slash blog you'll find it But basically what we're trying to do is just raise some money $500 of it is gonna go to this thing called the Anita Borg Foundation, which is raising awareness of women in Programming and doing some really cool initiatives and the other half of it or actually just splitting in half However much we get is gonna go to this rails bridge organization, which will hopefully fund initiatives to get computers into Schools maybe and teach ruby and there are some cool people doing Arduino stuff with kids and stuff like that Anyway, really cool. Please if you can just donate ten bucks. It'll be great These guys are doing awesome things So the second thing I want to talk about really quickly is I wrote this thing and it's not ruby, but I'm really excited about it I'm called Sammy and what Sammy is you can't really see it because it's small But what Sammy is is it's kind of like? Sorry, it's kind of like Sinatra which a ruby library that I'm kind of obsessed with in JavaScript And that's like a kind of silly way of describing it, but for rubious it makes sense And what an app looks like is something like this So you find an app you pass an application you give it a selector and you define some routes and what those routes do is act on The hashtag or can even act on the URL and then execute stuff like forms and stuff like that Unfortunately, this isn't my computer so I can't really show you an example, but I want you to check it out I just started a mailing list to there's a bunch of people who are kind of excited about it And the most exciting thing for me is this guy released this and like a week later this guy Alex Lang in Germany Figured out how to embed this in couch DB in a couch app So it's an entire application with Sammy this kind of Controller running as the client controller and it's really cool And that's on github and you can check it out and this is just github.com slash quirky or Code dot quirky comm slash Sammy, please check it out. Thanks everybody All right, so So it's my opinion that so we've all heard about Hadoop and MapReduce. Excuse me we've all heard about Hadoop and MapReduce and so on and Have some understanding of it and I think that it's actually Underestimated underappreciated in our community or at least I haven't heard or I came into a new appreciation of it in the past year that Even having heard about in the past. I didn't Fully appreciate it So I wanted to maybe if others are experiencing this help them along also with new developments like Amazon Elastic MapReduce it's a practical tool That's really available and can scale out for you even without setting up servers and so on so it may be useful to you Now in a way that it hasn't been in the past. Okay, so again, this is about Hadoop So when I was first learning about Hadoop I saw a bunch of word count examples like to where it seems like that's the only thing you could really do with it is You know count up words and logs But the reason for that and the problem is they were using the streaming API Which is the the simplest way of using MapReduce. You write any language And but you only get one MapReduce pass the more interesting This is a pretty simple operation. It's just like a ruby code that maps and then injects You can only do so much in that block of ruby code. So it's reasonable to see that It's not so expressive To get more expressive and more advanced operations You have to go into the Java API or use one of the DSLs that people built on that Which include pig and cascading and from there you can do chaining and redirecting and recursing of MapReduce jobs to where the maps MapReduce is a sub-component of A larger aspect sort of like middleware which you connect and interrelate through this common interface so here's an example these are operations from pig which allow you to filter and group and so on using a higher-level language and Where I think this is really interesting is it allows you to tackle problems that Would be otherwise difficult or impossible to tackle for instance large graphs graph traversal is one of the reasons MapReduce came about was for mapping the web and in order to do that you you can come into this this job API and so on and So with this Hadoop gives you tremendous power to to deal with huge data sets they just set the world record for for sorting a petabyte and This is something which enables us to find answers in data that that's out there that wasn't possible before And I really think it's something to consider and look into for your own data for instance that's generated by your user's actions or Out there on the web somewhere So get to it, you know two gems that are really small that I wrote The monitor is not going oh We're not mirrored. It's always mirror Mirror displays, okay That's why I could see so much. All right These two gems are called abstraction and freeze-ray Abstraction is a gem for making abstract classes in Ruby Ruby doesn't have a native notion of an abstract class and all an abstract Class really is is a class that can't be instantiated Which seems weird, but the reason that you want an abstract class is because you might not Make you might make a superclass Which is not actually instantiable, but provides a basis for subclasses So the the case that Here's a case where we have a user model and Maybe the user model is fine for a while But eventually let's say this is a an actual rails model eventually we say that we want two kinds of users We want a super user and just an ordinary Joe Schmoe and they're both users But there aren't any users which are neither so it's silly to make user still useful So what you can do in this case is you do the refactoring you need to get your code into these two classes and then you say that and you have abstraction available and now I've done this right if you say User.new You can't make it It's an abstract class. It can't be instantiated, but Super user new Doesn't work because I didn't set it up as a subclass, but That's my fault, but I swear to you this code is used in real life a question Okay So that's the next point right why do you want a class that can't be instantiated and the reason is test driving your code? So at Dropio, we had a class that we had to split up into two subclasses and the problem was that the first class the original class Wasn't gonna be used anymore where you're just gonna use the subclasses, but it was used in all the tests So we wanted to make sure that nothing in the tests or in any of the code that was driven by the tests actually made an instance of it That's one reason What's that We need the class to be implemented so there were things that Were common behavior so that we wouldn't actually quite hit that It's hard to explain it without actually showing you code But the other reason that you want to do it is because this will actually prevent rails from instantiating it And if you have a nil in the type column because these were STI single table inheritance If you have a nil in the type column it will by default use in this case user And you want to make sure that those things never actually happen to your code. So you want to fail fast So it's the gem is called abstraction. It's available. The other one is freeze ray and freeze ray I'm not gonna show you the whole thing is gonna be too hard to set up But it fixes dirty tracking in active record this pissed me off after record dirty tracking doesn't work And it doesn't work for one reason. That's that you can do something like user dot name Foo and if you do that What dirty tracking wants to do is see when you've changed an Attribute of a model object so that you can ask what's changed since the last time you saved it It works by looking at the setters So if name equals was called it would see that it changed But in this case we're actually taking the string object itself and mutating it We're changing it and the user object never knows about it. So it gets confused and you have these problems So the trick is or the solution is you install freeze ray and then you say Out of frozen Name and what happens now is Name the string is always returned frozen a frozen object in Ruby You just called dot freeze on it and it can never be changed. So this line is not picked up by Dirty tracking, but it throws an error So this is only gonna work if you never have a need to do this But really there's never a need to change something in place. You just want to make sure that it doesn't happen by accident So if you really need to change something in place for some weird reason, this won't work for you But this gives me the peace of mind to know that dirty tracking is completely consistent in anything that I can legally do Thanks. Yeah, I'm gonna in a five-minute talk. I'm gonna show you how to get a 10x plus performance boost by fixing Ruby threads Thank you If you want to see more stuff like this like low-level hacking interpreter stuff check out my blog at time2bleed.com And this talk is only for x86 and x86 64 Okay, so what is a Ruby thread the short version? Thread is just a schedule set of registers a Set of Ruby VM state and a copy of the thread stack. That's it Ruby has green threads as a lot of you probably know in 1 8 6 and 7 green threads There's only thread state in user land. There's no kernel context. This is good and bad It's good because creating lots of threads is cheap And context switching is supposed to be fast, but in every 1 8 6 and 7 that's broken. So it sucks Green threads are bad in general because if one green thread does blocking I o all your green threads die And you can't take advantage of multi-core because the kernel doesn't know about all these threads that exist in user land Okay, so they're broken So why are they broken when a context switch happens between green threads and the Ruby VM if the thread stack grew Then the Ruby VM will call realloc and for those of you who are familiar with malloc implementations Most reallocs will call mem copy internally if they have to move the block of memory to a different area Then it was previously. So that's one mem copy that might or might not happen the other two mem copies always happen the entire thread stack is copied to the heap for the current the current thread stack is copy to the heap to save it and The next thread that's going to execute its heap is its Thread stack is copied off the heap onto the current process stack That's really stupid because if your thread has lots of function calls and have a bigger thread stack It has a bigger thread stack the mem copies will be bigger and if the mem copies are bigger you have a slower Ruby So the solution is don't do mem copies. So do that. It's actually pretty easy You just create readings of memory by calling mmap you point ESP or RSP depending on your architecture at the memory region You kick off the thread and then you're done You have constant time stack switching and you only need to copy and update a few registers So benchmarks I use the computer language shootout thread ring benchmark on the naive benchmark You get about a 2.3x speed boost on Ruby 186 that naive benchmark has no like the thread stacks are tiny so it's not even realistic So I added a growth red function that grows the thread stacks a little bit as you can see It's almost 10 times faster And then if you try a naive hello world with Sinatra Mongol you get 1.26 x faster Obviously if in your handler you did something more than just say hello like you know you sequel or do some other garbage You'd get a much bigger savings And if you want to talk more about a little of optimizations find me after time to bleed calm cool And maybe I have time for one question What's up? Yes Was there another question I don't know you I don't know anything about Ruby so I don't know Okay, that's it. Thanks Cool, so I wanted to talk real quick about Google product. I found out about recently. It's called Google perf tools Some of you might know about it actually if you're using Ruby Enterprise Edition They use TC malloc which is part of Google perf tools But one of the cool things perf tools does is it has a built-in CPU profiler and it generates these really nice graphics That show you what's going on in your code So of course I wanted to hook this up to Ruby and see what happened so First thing I did I maintained event machines. I was like, uh, let's see what's going on in event machine It's actually pretty cool. You can sort of tell what's happening in the code You can see all the different event machine functions here's you pull It's reading from sockets. It's accepting sockets. So pretty standard stuff, but it's cool to sort of understand You can see a lot of Ruby internals to defining classes and making instances stuff like that So that was pretty cool and then I figured maybe I can try to fix some bugs with this. So we had a long-standing bug with the event machine when you use threads Alongside ePoll they were big performance issues. And so I ran this profiler and you can see in here. It's pretty big They're right to save context and restore context were taking forever and this actually Was related to the mem copies that Joe just talked about. So that was pretty cool. We ended up fixing that bug So then I was like You know, I wonder what's actually happening in Ruby and so I actually ran this in production and one of the cool things about this is because it's a You uses a kernel timer to basically profile the application You can actually run your production. You can tell it how often you want to be checking What's going on in your process? And so this one was running in a high-traffic production environment. It got 6,000 samples You can sort of tell what's going on Ruby and one of the big things in here was this Match exec which I was able to trace back up to stir sub bang and none of my code used that But it turned out date dot parse uses that because it runs a whole bunch of reg X is round To parse out dates and we were pulling out a bunch of dates out of the database And so I was able to basically put a gdb breakpoint on this figure out where it was and get rid of that code path So this was really useful and I thought, you know, it'd be really cool if I could make graphs like this But instead of C functions, it actually showed me Ruby functions And so it turned out not to be that hard actually and a few patches to perf tools and a patch to the Ruby VM And I was able to do this So it's pretty interesting I can show you some this is rails. It's very big as you can tell There's a lot of function calls, but it's very interesting to sort of see what's going on in rails you can compare that to Merv for instance and Merv it's much smaller Sinatra is even smaller than that You can see there's only there's a bunch of rack Stuff and then only a few method calls that take place inside Sinatra itself So actually last week we were able to use this I was playing around with the Redis Ruby gem and there were a bunch of performance issues It was much slower than the Python the other languages that implemented the clients and so I Ran this profiler on it and sure enough they were using a system timer to Set up timeouts on blocking reads off the sockets and you could tell it was taking a huge amount of time And so we got rid of that timeout and you could see it was much more normal-looking and it was as fast as the other clients And then so this is pretty cool. You can check it out. It's on GitHub. I just pushed it a couple hours ago. I Will go back to it But team and one slash perf tools rb There's a bunch of ways to use it. Basically You install this gem one the one caveat is you have to pass your Ruby installation with this small Patch and I tell you how to do it But once you do that you can just install this gem It'll download perf tools that will compile a patch a blah blah and you can either run it through Ruby by calling This function or you can just set some environment variables and it'll just profile your entire application without changing it and So yeah, you can fix a lot of bugs, but the other cool thing you can do is Just sort of understand how stuff is working So Ruby gems for instance if you just want to you know understand why would gem slow or what is it doing? How is it written? It's really nice to just be able to profile it and get this really nice graph and say, okay This is what's going on and you know Ruby gems in this case is taking a lot of time Basically checking the file system and doing a dirt I glob checking if certain files exist and positive directories That's it I'm Eric Odle. I'm from Seattle Ruby Brigade With the oldest Ruby Brigade on the planet and the most ass-kicking per capita Show me show me one that has more ass-kicking per capita. Am I mirrored here? I guess so There we go. Okay, so I'm here to talk about packaging gems and how you are all doing it wrong So The first thing should use the latest Ruby gems when you're releasing a gem you get better warnings about what you're probably not doing Right better errors for when you're doing something wrong and there's some cool new features that have been relatively recently added So it's one of the newest features is you get new lines in your description so you can make a nice pretty one if you forgot to type something in some of the Gem packaging tools have like fix me or fix or to do and they throw them in there I go and look for those and say hey, you need to fill this in and There's a relatively new attribute licenses so you can go and put in the license names for whatever you've got Not required yet or anything So when you do filling all this out and you get an error It's because you're doing it wrong and somebody did it wrong before and I got mad because I was trying to do some Statistics on the database like find out all the authors and how many gems they released and somebody filled the field out wrong So if you do you got an error fix it If you get a warning you're probably doing it wrong, but I'm not sure so stuff like not filling in your ruby forge project Oh, maybe you haven't gotten one yet. So I kind of give you a break there and just complain at you and So when you are building a gem, I got some guidelines use ho because I maintain ho and so I know it's it's the best The summary in the description are supposed to be different the summary is short It shows up in in gem list dash D with details. The description is long that shows up in the the gems in RSS feed Go ahead and fill in your contact data because people like to contact you about bugs or find your bug tracker on your home page and If you get lost there is documentation you may need to get the latest Our doc to go and generate it when you install Ruby gems So I have a hall of shame for people who did stuff wrong and I'm not meaning to Point out anybody in the room if they happen to be here, but if you are here. Well shame on you So, you know this one here this one the description HTTP caching for rack. I mean come on. You couldn't have written some more than that This one here. Yeah, that's that's totally useful rep description But then there's there's this Romobile dev person who just didn't bother to put one at all and so those are all in that Gems dot Rubyforge dot org index RSS And so this one here is a great one. So you got you know, you got the author name here You got like paragraphs. I mean I kind of cut it off, but it's great And so your lessons upgrade Ruby gems one three four. I released it this morning our doc to four use ho and read documentation when you get lost And one more thing the latest our doc and the latest Ruby gems now have their own Tasks for rake. So you instead of using the rake gem package task You can use gems package task and instead of rakes our doc task. You can use our docs task itself. That's it all right, so this is a Rails plugin I've been working on it's An active record observer replacement and it's called jit observer for just in time Observer I really like observers. I like though to be able it's to sort of think in terms of aspects So we've got like notification observers and Things for activity feed on we play and so that's the net and those are all activity observers It just sort of helps a segment functionality. This is just a contrived example here that's a cheerleader observer and You've seen this work and so the thing that what that what happens though when You load up this class and the way observers work is that rails will call that instance on them They're singletons calls that instance at initialization time And so it's going to basically load up every observer that you've got In doing so it's going to load up the class that it observes as well Which is typically an active record model which has some overhead to loading that up And so at first when I was trying to reduce our start the startup time in our app I said oh well, you know, I'm referring to The active record model name is a constant there truly to her of course. It's going to load it up So I said are no problem. You know, I know that there's this other syntax I can pass to observe that probably know that I hoped would defer the loading until it was actually used unfortunately active record observer doesn't defer the loading in that case and still loads it up and we've got a Lot of models and a lot of observers and so basically almost our whole app was getting loaded up at startup making for a slow startup Now how you know how much of a difference can this possibly make a benchmarking different active record model loads, you know Are very small models maybe 10 milliseconds or largest one our user class is a hundred milliseconds to load up And if you take that over say call a 30 milliseconds over a hundred models You're waiting three extra seconds every time you start up a console every time you want it you do a test run And it's believe me. It's painful So the idea of jit observer is that when an observer starts up If there's any class at the class that it's interested in observing is already instantiated already defined rather Then start observing it as well as its subclasses if not Basically just register itself as being interested in that class should have come into existence And then so we extend observable to basically look at the registry when Something inherits from when the class is inherited from observable. I look at the registry start observing that subclass and The usage is thankfully very similar just inherit from jit observer instead of active record observer and That's it Think I'm so the reason it's it's early I'm still working on it's because in dev mode and class reloading has some issues and I get multiple observers still attached to Multiple instances of the observers still attached to a model so I'm gonna clean that up if anybody wants to help it's on github and Thank you very much