 Okay, so this talks about jump cutter the next step in gem hosting So I was originally thinking of when I started This talk it was going to be a pitch and hopefully people would leave the talk and say oh I'm going to try out this new service and things have changed a little since then so I Thought a different name for the talk would be best So I thought of Michael Bay inspired a jump cutter revenge of the gem sources However, I don't know this didn't work out that well in theaters this year So maybe a more popular one like the gem cutter saga But uh, I figured no, I don't really like fake vampires. So Ended up going with this is a story all about how my gem hose got flipped All right, so my name is Nick Caranto. I'm a fifth year software engineering and computer science major at Rochester Institute of Technology Yeah, it's a really cold gray one I'm a bills fan. I don't care what you say about it even though we just fired our head coach and I'm a rubious that heroku when I'm not sleeping or in class If you want to get a hold of me, I am you rush or crush however You want to say it at Twitter and I'm at litany against fear.com. Yeah, I like If you want these slides to play along at home, they're at next n e x t dot heroku dot com Okay, so I figured I'd fail early and show a demo of the site So this is Jim cutter and its current incarnation and live It's basically the next generation of gem hosts. It helps people find gems easier because it's actually a search function and I think it's really helped with Just bringing out information about gems and it makes publishing them ridiculously easy I don't know if I should try it out right now, but I can we show the end result? So, let's see That's probably not gonna a it worked. Okay, so The end result is after you do a gem push The service accepts built gems so you do a gem push and then you get this awesome page with just the information you need So basic information on how to install it copy it a Simple information about what it is what it depends on where? assorted resources live and This is awesome. There's also a directory and since we launched. Let's see if this works come on Wi-Fi. I Think it's getting close to 1 million downloads on gem cutter. So I'm cutters been very successful So sorry, I can't do a demo. This isn't my system. I was gonna do a push and everything But it's okay So yeah, the history of the project. I originally started this in January of this year I Started hacking on our project called Jekyll Jekyll is a static site generator that runs github pages I originally found Tom Preston Werner's blog about it and I was sick of wordpress and I wanted to use this brand new system So I forked the project on github made some changes and then I clicked a little button and Got my fourth gem. That was great But something didn't feel right about it and I started talking to him and Josh Nichols the guy behind jeweler about What could be made better about the situation? It seemed that people had stopped using Rubyforge and that seemed really weird to me because Rubyforge is like the default Gem host is where everyone gets their gems from no matter if they like it or not. So it seemed like there was this problem where Like why aren't we using this this site? That's at the center and I wanted to fix that So came up with all these crazy ideas and I basically spun my wheels for a few months until RailsConf And I started really asking people it would this work. Am I really this crazy and Ended up that I really wanted I had to just try it out. So I went home I started hacking on it and I would say in August we launched and by launch I mean we got featured on Ruby inside and I Had hoped to have some sort of like beta period for the site and that's just and when that launch happened It's like there's no point the sites out there people are using it. Let's just roll with it So I'd say we've been really running since July, but August is when people really started using it and then in October Start I came together with the Ruby Central guys on figuring out how to make this the default Had a little proposal that said let's keep Ruby for July because we need to people depend on it for a lot of stuff that I never want to put in Jim Cutter and I Wanted Jim Cutter to be the default host from the very get-go because that I felt like it wasn't going to survive as a Competing service and that's really not why I started it. I started it to improve the whole situation so I covered finally got together with the Ruby Central guys and we figured out how is this going to work and Now Jim Cutter.org is gems that Rubyforge.org. So this is huge It's live right now And that's awesome, and we're going to be moving the site towards Ruby gems.org Ruby gems.org is going to become the community gem host. Hopefully that's what we're going to rebrand it as It's going to look exactly the same just a different title. So hopefully it'll make sense that this is a community supporter project That on everyone can use it is a lot easier so this has been a lot of fun to work on so far and I Don't know. I guess the thing is that everyone sort of has to use it So That's what's happened so far. That's the background. So I want to talk a little bit about the motivation behind starting the project So everyone knows Rubyforge. It's the default gem host It's what we had to deal with right At the very get-go when I was looking into gems with Jekyll There was product approval and that's been fixed recently But I didn't like the fact that someone had to say yes, you can do this I just wanted to go ahead and do it So that was one of the big gripes I had with Rubyforge and another big issue Which I think is more important is it's written in PHP and I didn't like that I didn't like that a PHP site was at the center of the Ruby community So I wanted to fix that so the next big deal with gem hosting was the github gem server and he looks really sad on a dark background but So this was the new hotness people obviously had problems with Rubyforge and they wanted to move on to something else so I think the The forked gems that github encouraged was great and they are a great way to Release code. I think Not having to dive through the bureaucracy or even ask people hey I have these little changes to your library and they'll probably say like no or oh, that's awesome But if they say no then you're screwed and I think having a way to easily publish little modifications or for gems is great and github provided that At the same time like that's an awesome workflow, but the real problem is that it creates all these Fragmentations, so we've got at least for some huge projects like Jekyll Delayed job. We've got all these projects that have tons of forks And if you even try to look at the github gem repository, they're just filled So someone coming into the community who doesn't maybe know about sites like Ruby Toolbox or the other recommendation type sites They're not going to know which one to use so I feel that You know them for great, but there's that sort of fragmentation problem that we can't really help That needs to be fixed somehow Another big deal was has my gem built yet. Org so There's this delay between when you would give your code to github and when it would be available to download And I just thought that sucked. I didn't want to wait or continually refresh something until my gem was ready I just wanted it to be ready. So I fixed that So The way gem cutter works now is that it builds gems in under or at least indexes gems in under a minute And I think that's a lot better It used to when you would gem push it would Upload your gem to S3 Rebuild the index and then dump you back into the terminal and then people started telling me I don't want to wait five seconds for this to occur. So Now we just upload S3 and then we pick off a background job to rebuild the index and that usually goes in under a minute so I'd say it's Almost instantly available So the real goals I started jump cutter with where I wanted accessible info you guys saw the gem show like the gem view I wanted I really modeled it after dr. Nick Williams new gem page So it's got a the version number. It's got what it is how to install it a little bit of information about the gem and like some other related info as well and with Ruby forage you really didn't get that you got a list of files and With github you sort of get it, but It had a required effort. He had to create a read me and it wasn't standardized So you might some gems might have a lot of information and other gems you don't get So I wanted one place to get all the information about your gem The next thing I wanted was publisher API At the time I had no idea how this is going to work I was just sort of sick of the Ruby forage gem scraping html in order to publish or in order to post forms and get Version numbers and github simply didn't have an API. They just had a checkbox So I wanted a rest API to push gems. So you would build a gem give it to the service and it would take care of the rest Finally, I wanted to be open source Sort of goes without saying but I wanted people to make sure that they could contribute to the Ruby community site using Ruby So yeah some tech behind the site This is mostly going to talk about libraries I used and why I think you should use them and look into them So the first big thing that jump cutter is based on is AWS S3, and I'm sorry that's in the AWS S3 is a Ruby interface to Amazon simple storage service as I moved jump cutter away from my 256 meg slice house instance, I realize the gems had to live somewhere else and Downloading a few gigs of gems on that machine didn't go very well so I Decided that S3 was going to be the best place to put them mostly because I was moving towards Roku and her oku says go there so I Was at first fearful of this I never worked with S3 before and it ended up being really really easy And that is way too small Isn't monospace Man really? Now we're in the dead zone. All right. We're good. Okay, so sorry. It's not all pretty but so the way this is done is you Subclass AWS S3 S3 objects you set the current bucket, which is basically a giant folder that S3 dumps files into And then so for jump cutter jump cutter production We obviously have switches for like staging and other environments and then in order to put things in the S3 All you do is you call store on that class So store with the key and then the value so the data that we're pushing up to jump cutter And then a bunch of options like public read just to make sure that everyone can get to it So I really didn't think it was going to be this easy to interact with a service that Scales so well for file downloads, but it was Yeah, please look into that the next big deal was Sinatra I started jump cutter off as two little Sinatra apps. I didn't have rails in mind at all Sinatra's been one of my biggest influences that as a Ruby programmer in the last year or so And if you haven't looked into it yet, and you're just doing rails development Please do it works so seamlessly with rails now that it's almost silly not to look into it So I started jump cutter office to Sinatra apps one was for the gem server. So I sort of had to Look into how Ruby gems served up gems and I rewrote that Sinatra instead of I think it's just like a web brick handler now and The other part was the site so as I continued to move forward with development. I realized I needed users So I dropped Sinatra for that I Know there's a rack-based authentication plugins like warden and I guess there's the vise now But I didn't look into them enough and I worked at thought bot at the time and I really like clearance So I wanted to use clearance and I wanted to get the site on rails so the nice thing is that Sinatra is still a big part of Jump cutter every gem that's downloaded now goes through Sinatra and it looks sort of like this Like I said, I had two so not I had two Sinatra classes really and when I moved to rails I literally dropped it inside of app metal and it just works That really blew my mind. I didn't have to do any configuration. It just dropped it in and it was almost spooky So this is one example of an action in Sinatra and this is the giant index of gems, so it's a huge G zip Marshall array of every gen that's available on our service And we set the content type to application ex G zip and then we serve it based on if we're Talking to s3 or the file system because gem cutter can work off of file system as well, which means you could run gem cutter Behind your firewall at your company if you want to you can run it anywhere you want really it's not tightly bound so having the gem server as a like a separate component has been a huge win and What a blog post recently about putting gem cutter in a maintenance mode, so The deal with this is that now that the gem server is a separate component We can load the hostess without loading rails So then we can run huge migrations if we need to and gem serving will still work So I won't kill the entire Ruby community for a night So this is basically what the rackup looks like and it's getting cut off but we bring in the bundle environment which We're on the rails environment, and then we load up rack static which is in the rack library This is basically a Way to serve up static files with rack And then we bring in the hostess which is our gem server, and then you can't see it But we bring in rack maintenance by David dollar which basically says serve this file Whenever any other URL that's hit so basically if All the nice maintenance pages aren't hit Then rack then the maintenance page shows So this has been great because now I don't have to worry about people yelling at me when I'm sleeping Can't see it Okay, so the next big part of Gem cutter are gem plugins gem plugins are a relatively new feature of Ruby gems that I feel aren't being used enough This is how gem push works I originally had no idea how people were going to interact with the API for gem cutter, and I read a blog post on gem graph. I think it's basically like a dependency grapher that It was implemented as a gem plugin, so I looked through that and I said yes. I want gem push I like it so much. I'm gonna do that and the funny thing is that we've been getting a We've been getting issues on github that people are trying to do gem push origin their gem But uh sorry so the way that works So the way that works is you have a live ruby gems plugin file in your gem and Those files are loaded before any but any other file in ruby gems and that implementation is sort of weird It's great because it lets us do commands like gem push and gem graph But the problem is that anything you put in that live ruby gems plugin will be loaded for everyone that has that gem installed One example of this. I had a URL all caps constant in that file So if you try to like load any other App with ruby gems that URL constant would be there So you just have to really make sure that any libraries or any constants you're bringing in at that point are Delayed as much as possible. Otherwise you're gonna get an agent trouble like I did So in that file you basically call gem command manager Register command so you say here's the command I want and then you make sure you've brought in a subclass of gem command And all it really needs are some other extraneous methods, but the only important one is execute So when you call gem push it calls execute So that's really how easy it is and I really think that if you're building something that hooks in the ruby gems Or at least has to do anything with ruby gems. It should be done as a gem plugin. You shouldn't need a separate binary Okay, so the next big part of gem cutter is delayed job like I said I originally didn't have a Maybe I didn't say I originally had people push gems and it would instantly like Upload and index and that was a bad idea. So I needed some sort of background processing and heroku supports Lay jobs, so I wanted that and it's really been a pleasure to work with I know there's a ton of block a ton of job keys now, but this works really well Here's the way it's used in jump cutter so we have a gem cutter class that it's not really active record, but there's a save on it and It basically writes the gem which is another right to the file system or s3 And then we use the lay job to in queue the same class So what happens there is that the lay job serializes this class as YAML Thumbs it in the database and then shut and then continues on so then we say hey you registered your gem Thanks, and that takes seconds instead of minutes Which is I think a big win because you get to learn immediately if you're gem messed up So then there's in the background. There's a delayed jobs worker that runs along finds the Entry in the database pulls it out DC realizes it and then called perform So perform is where we update the index so this has been exceptionally easy to do I really like that you can sort of queue up the same object. You're working with I know this is whole way with delayed job of like making job structs or something But I think this is the most conceptually sound way to do it You know the other thing we use the lay job for is download processing so every gem that's ever downloaded we keep track So hopefully we'll see some cool graphs eventually. So yeah testing testing is a huge part of gem cutter It's a huge part of all of how I develop. I definitely haven't been accepting any patches without testing and I won't So I was drinking a lot of thought about Kool-Aid. So I use shoulda and shoulda is great. I love it I also use factor girl. I don't want to go into fixtures that argument What I do really like is double R or our R. I don't know what the right way to say it double R All right, so It's a mocking and stubbing library that I feel is the most ruby way to do mocking and stubbing And I'm probably using it wrong because I just learned half spies So you can go in there and yell at me if I'm doing it wrong But I think our is an awesome library that if you haven't looked into other mocking libraries and if you've seen mocha and if you've seen the tons of other ones you need to look into our and Finally cucumber cucumber has been pretty much my saving grace for this project I started with cucumber from the very beginning and The cool thing about it is that I wrote a feature like push gems Right when it was too little Sinatra apps and then as I moved the Sinatra apps to rails I could still run those same features and I could know that the site worked So I knew the transition to rails was done when the test passed again. So having these integration points like where the API hits the site Tested by cucumber. It's been extremely helpful and this is gonna look horrible. Oh, yeah, of course you can't read it So this is the cucumber feature for pushing a gem So given your sign up and confirm with an email You've got a built gem on your system and you've got an API key with our service when you push it Then it should work, right? You should be able to go to the page see the see the gem and make sure that the right version is up there And I'm sorry that you can't see it in the back, but uh, these cucumber scenarios. I think really They helped me sleep So I encourage you to look into cucumber if you haven't So yeah other features There's a lot of really neat features in Ruby gems that no one knows about and unless if you're like diving through Ruby Gems code, you don't learn about them. So Maybe you do and that's great, but I'm going to tell you right now So the first big deal is pre-release versions and that's in the So pre-release versions are basically a way to specify alpha or beta releases. I know you've talked about it a little so basically you tack on a a Bunch of letters to your gem version name So you can do dash pre or dash a or dash bananas kites, whatever you want and it will be counted as a pre-release gem So this is a way that instead of like forking a gem or instead of having like your own personal gem server Just use pre-release versions When jump cutter it gets a hold of one of these It places it in a separate index and you can only install Pre-release gems when you tack on dash dash pre or dash dash pre-release when you're doing a gem install So this is a big deal that I don't think enough people use yet Another big deal is development dependencies So I'm pretty sure we're all familiar with defining dependencies for gem specs So this is for example rails depends on active support action mailer so on and so forth but you can also you can also define what Testing libraries and what mocking frameworks and so on and so forth your library depends on so that way If you want to find out what you need in order to hack on a gem You've got it right here So the nice thing is that gem cutter displays each of these really clearly on the gem view page So you can see really easily what the gen depends on and what you need to get So this is not working yet just disclaimer so One of the biggest things I started I knew I had to do when I started gem cutter is that I had to link somehow to outside resources This site is clearly not Rubyforge We are not going to do code hosting and other things and I had needed a way to say this is where you get the code this is where the bug tracker lives and the way I did that right now is there's one form on the site to fill out and That's the only form on the site besides logging in and signing up and forgetting your password So I want to do everything possible to kill that form I want the site to be read only as possible so the gem spec is the interface So the final piece of the puzzle is pushing all those links into the gem spec which hopefully I want to do soon It's talking with some of the Ruby gems guys about it and they seem okay with it So the way I would like to see it is that there's a hash of links So it could be that easy. I'm not sure when this is going to work, but that's what I want okay, so No, you can't see it, but I'm going to talk about some of the things that are up and coming for gem cutter The first big thing is our doc info if you haven't seen the site, please go to it This is a automatic gem or automatic documentation builder that runs the yard And I really think yard is the next step above our doc. It's got it's our doc. Okay it's our not compatible and it's got a lot of really neat features on top of it and I really think that it's the way going forward and I'm going to be pushing it as much as possible so this site builds documentation for github projects right now, so you give it like your github username and The project name and then it just builds the documentation and keeps it up to date as you commit So I want the same thing for gem cutter. So when you push your gem It automatically builds doc documentation for you and I think that this is going to be a big win Just because it'll force people to actually document their libraries Another cool thing That's sort of a service hook and hopefully we'll have like a general service hook System like after you push is a caliper which you can't see but this is metric food what caliper does I think there was a lightning talk about it last night. It's basically hosted metric food So I want when you push a gem metrics will be built So I think this kind of thing that like after you push a gem all these other services get activated It's going to be big going forward So another big deal is a read me and search You may have heard of our pen One of the things I want is a much better Searching feature for a gem cutter right now. It's really bad. It just looks through the titles and descriptions but I want to I want it to be much improved and just haven't had time to look at that yet but another big piece of this is putting the read me's for every gem on the The gem page so you don't have to go to GitHub to find it or you don't have to go to some other site to find it It's all the information is all right there and for searching that content to that'll help people find gems a lot easier so hopefully Hopefully jump cutter it can become This sort of central site where people will go for all sorts of information regarding them. All right. Finally, there's gem forking so this is sort of a big controversy as The Hold the buckle with github gems was winding down Sort of realize that we needed to support this same workflow just because they stopped building gems for people mean doesn't mean that we can't Support for I guess the first thing to say is that we do support for gems like you can take a gem spec Tack your name onto it so I could have like you rush my gem and then push it And if that gem doesn't exist then you own it so that sort of works in the github fashion right now But the way we're going to go forward with it is have people register subdomains on gemcutter.org so This should be really neat because we're gonna give you a blank index of gems and then you can put whatever you want there so it could be really cool because people will be able to put whatever they want up there, but it could be really bad because You might have your own crazy version of rails that does crazy things that no one No one should ever do and No one should no one else should ever install it and if you have it in your source list then that will get installed So I really want to push that Ruby gems that are moving forward is going to do the canonical repository that we can trust and the only way we can do this because I Don't know about you, but I can't trust anybody else I want to trust the gem signing and certification stuff that's faking the Ruby gems right now And I think as a community we need to learn more about it. I don't know enough about it right now But I want to be able to say I know who this gem was from and I want it so I think going forward more education documentation and Just learning about that in general will be big, but as for gem forking I still think this is going to be a huge win no matter how much of a mess we get into with different weird versions of gems being on people's subdomains and So yeah, this will be a fun feature to do and hopefully we'll be coming soon So yeah, I just really want to emphasize that this is our gem host now. This is the community's gem host It's an open-source project. I've been running it like Rubinius So if you submit a patch you can ask me for core access and I'll give it to you. It's that simple I've been trying to really instill a sense of trust among all the people that have access and all the people that are just hanging out Watching jump cutter go on and try to emphasize that this is our project now If we want to see it evolve in ways that we've never imagined before we can do that We can it's not locked into some crazy system or it's not some other language. We don't want to work on it's ruby Let's hack on it. So yeah So just want to emphasize that let's make it awesome. This is our gem host. Let's do it So yeah, thanks The site is obviously there if you want the code It's like a hub and feel free to talk to us on freenode when I'm not at the conference and have an internet connection All right. Yes What's one of my favorite things about the CPAN is that they're sort of name spacing and categorization of CPAN packages and there's nothing like that in Ruby and when I'd love to see the search get better It would also be great to say like I need a gem that does something like this I'd be able to like look at all the options and see like category of 5 gems that might solve a problem in a specific problem Okay, so the question was is gem cutter CPAN and I haven't honestly used CPAN I'm not a pro hacker, but I think moving towards some sort of neat recommendation system Or just an easier way for people to find gems is important tagging. There we go. Okay, so let's make that happen Josh So it's great that it's not the source project and you want people to contribute. What are you using to manage the roadmap and issues and Organizing Definitely, I think we should move to a legitimate issue tracker and I actually have milestones and releases and stuff So yes, I will All right I just want to say tagging doesn't actually work in scale like look at Rails's bug tracker tags It's a really awesome early idea, but it totally fails when there's like a million Okay You mentioned you'd like to see one pretty graphs It's not available now. We're tracking every single download for every gem. So it's in the database We just haven't exposed it yet, but I'm definitely open to Evolving the API right now the API is pretty primitive and I think before this is one thing I should have mentioned so we're merging the gem cutter plugins So gem push gem owner and whatnot into Ruby gems proper. So part of that effort among with figuring out Ruby gems and how that works has been a Moving the API for the site to legitimate like API v1 something something So I think moving forward like growing the API and making stuff like that available. It is definitely yes Yes Actually, so the question was should we publish RDF data and honestly, I have no idea what that is I'm sorry, but I'm definitely more than open. I'm more than open to it Any other questions great if you're free to come up and bug me about this. So thanks guys