 Video equipment rental costs paid for by peep-code screencasts. Hi everybody. Let's get going. Okay. Well, first off, Fernan, this is Keynote. Yeah? Yeah, this is Ruby Conference. Yeah? Cocoa's in the other room. Oh, sorry. So we need you to do this as a Ruby presentation. Okay. You turn me off, Jeremy. All right. So what Fernan's doing here is something that we do all the time, which is install gems in our framework, well, not really our framework, but in our work environment. And then we start stuff up all the time. So how many people use gem internally, not just like third-party gems, but use it within your own applications? Raise your hands. One. Come on, bring them up. Looks like... Okay, close here. Excellent. Good job. Okay. So I'm going to try it as a gem, as a mobile application, actually. Hopefully it doesn't look too bad. We never tried it on the projector yet, so it looks pretty good. So first off, Fernan and I work for a company called Collective Intellect, and we basically collect blogs, message boards, news feeds, other social media data feeds, and combine it all together, provide a topic analysis of it, a little bit of sentiment analysis on it. And then we provide these in various formats for our customers to actually consume in either our GUI or their data feed or anything along those lines. So let's get going. So first off, here we are. First bug. First bug. I'm Jeremy Heingeidner, and this is what I do a lot all day long. I shovel data muck around, move it from point A to point B, take somebody else's muck, convert it into our muck, and then keep it going. And you do your makeover job. Try to do a heck of a job. I also have a few open source gems available. Copious free time is what they're all under. Amalgalite, rabble, keybox. Anybody heard of any of them? All right, we got four users. We have four users. Awesome. The other thing I do is, come on, where is this? I shoot a fair bit of photography, so I'm going to take a little, see how many people we have here. All right. OK. That's actually a good question. I spend most of my life trying to figure out who I was. I'll tell you what I know so far. Basically, you may remember me from one of my best selling books. Get confident, you stupid. Maybe not. So I was born and raised here in Texas. Not. I figured I'd try it. You know, I'm actually from France. This is one of our TV anchor. She's a little bit easier on the eyes than Ted Carpo, in my opinion. Let's see, then there's nothing. Oh, OK. Oh, we're missing something. So I was like, basically, I also host the rare group, which is the Denver Rails-based user group. My company is Liquid Rare. I have offered a couple things out there. One of them is called DeMol. So how many people know exactly right now what folks are doing with your web application or your Ruby application that's out there? Do you know is it still running? Do you know is it performing? Do you know if anybody is using it? If you don't, if you send note to some of those questions, check out DeMol. It basically allows you to inject malls within your application code. It does have to be Rails. It can be Merb. It can be a panel of Ruby applications. And you can extract a lot of information such as performance, whether things are throwing exceptions, and also try to figure out if you're launching a new application, what kind of features users are actually using, which is really good for your marketing guys. Yeah, well, it's good to know which ones of our customers are actually using our website, which helps a lot. The second thing that I wrote was one of the guys that is here, Dolien Berry. We started up as a Rails plugin, and I converted it as a gem. It's a ZF framework that basically allows you to do charting on your, again, Rails or Merb application. It's a really nice starting package. You may approve it, but I'm not biased there. Okay, this is a, scroll that up a little bit. This is a simplified version of our production environment. Yeah, actually really simplified version. So a couple quick questions for everybody here. How many people are using more than one language in their environment? Oh, almost everybody. All right, how many people have more than one application framework? All right, how many people have more than one application framework per language? All right, so we've got two rubies, two javas. You know, how many, what's you wanted to say? How many people here are actually Java developers disguised as rubies? All right, we got a few. Come on, there's more than, no, don't be shy. Dotnet developers disguised as rubies. Okay, all right, C programmers disguised as rubies. All right, more of those. Another question we have for you is, how many people have more than one database in their environment? Okay, do those databases know about each other? No, okay. So one of our things is we have a, we have our transactional database with multiple slaves. We only put one on here right now. And we also have a data warehouse that feeds data from our transactional database into the data warehouse. And we do a lot of reporting out of it. So this is our world. It's evolved over time. So we'll start out with world one. Okay, so when Jeremy and I started that corrective intellect, it was a startup mode, code got thrown, right and left, you hear a lot about refactoring and patterns and all that stuff. But when you have a business to build and customers to gain quickly, usually your customers are on your code base so you tend to be a little bit sloppy about it. World one is exactly that. I think it was past code smell. Yeah, past code smell like rick. Not refactored, it was unfactored code as shown earlier. Yes, so there was a lot of, There we go, unfactored code. There was a lot of thatch. There was a lot of bad practices in there. I mean, we can't really criticize that much because at the time I guess it was as good as an idea as any, basically it's one directory with a mixed bag of framework pieces, models, utilities, different ways to connect to the database depending on what type of application you are running. It's a single Ruby application, it's a Rails application, different ways to deal with Active Record and so forth. Deployment, basically using Rubylib to point to various pieces of that directory. And then as soon as we started building up the team, more people on the team started looking at putting Rails application up or Merb application up very quickly. Whoa. Hello. Very quickly, this framework, this code base went to became really unmentionable and it was a very tough environment to deal with. Most importantly, and because again of our, probably our data model, we deal with millions of records in our database and basically there was a lot of, it was hard to create tests. You take a lot of time to get the right data set and figure out the right granularity for testing so therefore there was no testing whatsoever, right? So that was another really bad thing. And you can see how sporadic it was. We've got models, logging, there's a domain over here, database, plugins, shared concerns. I mean, it's just spread out all over the place. And we had SVN externals that pointed a bunch of different locations and that provided a bit of a deployment nightmare. Yeah. And to make Mario worse, and I hope nobody follows those practices. Basically, we did have Capis Rainer at the time and there was a guy in the team that was really good with Ant. Basically built Ant department scripts, which is fine. I mean, there's nothing wrong with Ant. However, when a developer deploys, he deploys his code base directly into production. That's directly into the production. Exactly. We had multiple occasions where we would have something that wasn't even checked into subversion that actually got deployed into production. And so the only locations for that piece of code was a developer workstation and our production environment. There was no record of it anywhere. Yeah. And I actually did it on purpose more than once too, which was bad. Ant, you know, kind of is, you know, greatest pass to evil because eventually you get to a point where you're afraid of deploying because you don't know what's there and how you're going to break it. Or you're not. So you go in and hand-patching the production system that just makes Mario's worth. So we learned a little bit, you know. We figured, hey, this is the wrong way to do stuff. Yeah. So basically we got into World 2 and World 2, you know, because we were heavily on the rail side of things, we decided to break things out as rail's plug-ins, right? So we decided to basically use the plug-in directory structure and philosophy and start pulling those various pieces, those various concerns that were in World 1 into a more divided plug-in world. And this was really trying to address vertical concerns. So for example, you know, when we go out to database, we also need to, you know, inflate, you know, some XML and all that stuff. So we built it into one plug-in, so, you know, nice API to basically combine, you know, a whole bunch of information. Sorry. I'm going to say this, can you tell? One of the things we would do is the models would actually get sliced. So we'd have a base model and then we'd say, oh, we want to add this application needs this functionality in a model. So that application would have a separate plug-in that would just have that thin couple of methods that would include it when that application got deployed. Well, then we started, you know, it started getting really brittle. Yeah. The one thing that I had reminded me of was a program in C that has a lot of pointers because we were using SVN externals even more so now than we were in the first situation. Yeah, because every, you know, since we dealt with different type of applications, we couldn't just do a rail's plug-in, you know, for every application. So therefore, for Ruby framework and things like that, we actually had to use externals to, you know, pull in, you know, the values plug-in. So, you know, this was not a bad idea, I thought, at the time, but basically it was all built on the wrong premise, meaning saying that there's really not that much code in models, right? So for instance, you know, the model, what do you have? You have relationships, you know, with other models, you have, you know, validation rules and things like that. And basically, you know, depending on what type of application you're actually working on, some of those relationships may not be interesting to you, right? So if you have, you know, a customer model and you have maybe a demographic type of information, may be useful for, you know, and for applications, but may not be useful to other applications. So therefore, when you build your application, you can actually customize the validation rules, the relationship, you know, models and you don't have to look at, you know, thousands of lines worth of model code that you don't really care about. So in essence, he made sense because he would simplify the code base. But it made him more complicated. This was the wrong premise. Yes, you know, you know, in models, you train to encapsulate really your domain, you know, your business objects, right? And typically it's mirrored on the database. Well, what kind of relationships do you have in the database? You have a set of rules. You have certain things that needs to be constrained, you know, Frankie constraints, certain validation, you know, what kind of, you know, if you have innumerable types or if you have values that can be, you know, one, two, three or whatever, you know, there is some rules there that needs to be applied. So when we look at the next world, yeah, basically world. Well, this is an example of our directory structure for plugins. And this is just a fraction of the number of plugins we had. Luckily, this was only about 20% into using World 2 for our system. And so we didn't get fully converted. Otherwise we probably would have had maybe a hundred or so different plugins, which would have been really unmaintainable. And then we realized, you know, there's probably about five or six of these that almost got included in every single application. So why are these all separate plugins? Why are they all separate concerns when every single application is using them? And then we came to the realization of the one true path of way doing things in Ruby, which is gems. You love it, learn it, live it, crack it, and we mean it. So all those of you who raised your hands using gems internally, raise your hands again. All right, see, these people know what's going on. It's all really good stuff. So this is where we came on to World 3. And I must say, World 2 was kind of like the red-edged step chart of the evolution path. It didn't last very long. It was only a matter of two or three months that we actually had code that was using this World 2 approach with the plugins. Right. So basically, we learned some as far as, basically, best practices, how to break out the code into separate modules. Because really, it takes mental exercise. It's not, it's easier to dump stuff in a directory structure, and you don't have to think too much about how the code is organized. But when you have to really start thinking about your dependencies, what piece of code must depend on what, you have to put a little bit of brain effort into it in order to figure out what are the right level of granularities for each of the different modules. This in itself really pays off a lot of dividends because basically that's where you actually build the system out of very pragmatic rules and very small pieces, and then start assembling those pieces to create something that's significant. And that's what was going on with these SVN externals, all these different plugins. We basically fractured our code into a million different splinters, and then we realized, well, this is too many splinters. We need to put these back together in actual real logical units that make sense. And so what we ended up with, something that looks kind of like this. All right, go for it. So we ended up refactoring all of our code into about, it's about 20 or so different gems, and each of these is important in its own right. So we start out with the primordial gem, which I'll explain about in just a minute, and it's the basis of all our system. Then we have the sentiment, which is this blue line down here, and that is essentially our database interface, all of our models, that kind of thing. When we were fracturing out all the different models into different externals and SVN externals and plugins, it was too much. There was too much interrelation. One data model needs to know something about some other data model. There's some sort of referential integrity that you have to deal with, or some sort of business logic that has to take place when you upgrade one model or you do a method on one model, it needs to take an action on another model and things along those lines. A couple of these other ones, these really tiny ones, we have message categorizer, message spam filter. These are really tiny command line applications. And so one of the things we do is we're able to deploy all of our command line applications as gems, and then down under here, Quartz, all those folks that were Java developers, disguises, Rubius, how many of you know what Quartz is? All right, we got a fair number. We use Quartz a fair bit. Yeah, we had basically needs for a lot of various crown jobs, and we had tons of servers, and crowns were running here and there, and there was not really a central way to manage all that, so if one crown job was to fail, we wouldn't find about it for a day or two later, because our data started looking funky. And so we decided to use the Quartz framework. And again, I think when you deal with a system of this amplitude, and you're dealing with different pieces that need to come together, I don't think you can be just like a techno snob, what I would call, where you say, well, I'm a Ruby shop, you only use Ruby only. I think in the Java world, there's a lot of good, really well-tested and well-documented and solid piece of architectures, and that's what we came to. We saw Quartz being an ability to schedule jobs, and we can cluster it, and we can grow it as we need to, and basically we built this little Java to Ruby bridge that allows us to come in back into the gem world to run various jobs, and we can do the monitoring, we can do the logging, and figure out when the job fail, exactly getting an alert for it. And this Quartz gem here, packages up all of our Quartz jobs. So we can deploy our Quartz gem to any machine that's actually a Quartz server. So it just pops out on the gem, all of its command lines that are available for our different Quartz jobs, show up on the Quartz servers, and then they're just made available, you know, easy as pie. So this is what I'm gonna talk about. One of the fundamental things that we realized, and this is our third iteration of essentially our infrastructure, is the primordial gem. This is a critical piece of our infrastructure, and is essentially our developer gem. Every single developer in our environment must have this gem installed on their workstation in order to actually write a single line of code. This manages a new project generation. We use a product called Bones, it's an open source gem, who all knows about Bones. All right, we got three or four. You should really check it out. Bones is a really fine piece of work. We have a project skeleton that we put inside the primordial gem. We run a command, it generates a new project for us, and in that project over here on the left, or stage left, you can see the result of generating a project with our Bones. Actually we call it ooze. Primordial ooze is what gets us going. And every single one of these files is generated out of our initial Bones template. So we've got a top level binary, dependency.yaml. We ran it our own dependencies, and I'll get into that in a little bit. We've got our basic specs. We use specs right now. We have our deployment targets for Capistrano. We've got a cruise control config that comes as part of our generate project template. And this basically provides one of the biggest things that's been a bonus for our entire environment, which is a consistent project layout for every single application. Every single one of our applications looks like this. Command line, library gem, MIRB application, Rails application, Quartz job, they all look exactly the same way. Some of them use everything, well actually almost everything uses everything. The other of the big things in here is common tasks. Because the primordial gem is installed on every single developer's workstation, all the rake tasks, thanks gem, are in the primordial gem in the actual gem directories. They're not actually inside the project. Since it's the exact same tasks for every single project, they're all stored centrally and they're all made available for every single project. Some people use Socky, which is a system-wide rake. I haven't actually played with it much, but that might be another approach for doing the same type of thing for your systems. But again, the stress here is on familiarity with something that we've learned on World Two, even the World Two was not really a failure. That basically it's important for people to come into a code base, even a new code base, and know exactly where things are at. So you're looking for code, you're going to go under Lib. You need looking for tests under the spec. You're looking for deployment environment. You're going to go in the config deploy. That familiarity has a lot of leverage within a team of developers because sometimes you're on vacation, sometimes you're not available, something fails, you need to go in and look at somebody else's code. Well, that already helps a whole lot to be able to locate where things might be. So this is really useful for us. I mean, you just do OOS, a new project, and you can just totally deploy that. It doesn't do anything, but you can deploy, run it through CUS control. There'll be a first test that's going to fail, but you're already up and running within one-line commands, so that's pretty cool. One other quick thing in here. We also use the Primordial Gem because this Primordial Gem is deployed on every system too because it has what we call some fundamental base classes for every single one of our libraries, our projects. There's a base class that everything inherits from, has some utilities in it, some library path approaches, ways of finding data out of the gem, different things along those lines, and those are deployed on every single system. Another one is schema management. How many people here have the luxury of having a DBA on their team? Yeah, that's what I thought. Yeah, it's a rough deal, I mean, we started trying to go and get fixtures type stuff, migrations to work, so in that sense, we never ever roll back on migrations from the production database. I mean, I think if you have a societal tendency, that'd probably be a smart thing to do, but we never do it, and we started there because people were familiar with Rails migrations, so we started building a new schema out of that. But very quickly, we figured that, and probably also because we do have a DBA, somebody that's really paid close attention to the database and the database performance and indexes, frankly constraints, and stuff of the kind, and you know, he didn't understand migrations, and you know, really to us, it was more beneficial to have that person manage his own schema. Another thing that goes along with this is, how many people have things that talk to their database that are not Rails applications? All right, it's about half, okay. So, if you're gonna manage your migrations in Rails or MIRB or something like that, well, that's the deployment framework for your schema, and that doesn't work. If someone else is gonna add a new table and your application's not gonna know about it, what's the chances that's actually gonna get into your source code repository under your DB directory? So, what we decided is, and we worked this out, was we have sentiment main, our database is called sentiment. Sentiment main is our production schema, managed by our DBA, and it's everything that's in production is right there. Then we have sentiment dict, which is essentially static data, lookup data, days of the week or countries in the world, zip codes, all that kind of different kind of stuff, that's static data that doesn't really change. Then we have, when one of us decides we need a new table, we need to alter a table, we'll create the SQL to do it, and we'll put it in patches. Give it a new number and patch it, and this is all in SVN, and this directory structure is actually managed by our DBA. So, they'll put it into patches, and then it'll get vetted by our DBA. They'll look at it, say, oh, okay, actually that's not a VARCAR field. What's the maximum value you're gonna have in here? He'll tune it a little bit, maybe apply an index based on what we think our access paths are gonna be, and then he'll apply it to our development system. And when he devlies it to the development system, it goes into applied patches. That means it's been vetted by him, and it's ready to go. And then, when it gets deployed into production, he just merges the applied patches into this main sentiment SQL. And so, this allows us with these common rake tasks down here at the bottom, dbclear, dbcreate, drop, load, patch, populate, recreate. We, on our developer workstations, are able to say, oh, I want my database to match what's currently in production. dbcreate, done. dbpopulate, done. Break tasks. If we wanna actually move forward from production and go to what the development system is, then we can do dbpatch, yeah. And that'll apply those. And then if we wanna be our bleeding edge, which is now what's in our development integration, distributed platforms, we can apply the third patch with just another additional rake task. And so, this has actually made our, it simplifies schema management so much, and this has been a great bone to our entire environment. Yeah, this also guarantees that where you are running locally as a developer and what's actually in production is one and the same thing. So that's pretty important. So, and it's really simple. It's just three directories, rake tasks, you're done. So, fixture. So, in order to build this infrastructure, we also wanted to inject a lot of testing. So, basically, we looked at Rails fixtures and we didn't like it too much. We didn't think it was flexible enough for what we needed to do. I mean, we do a lot of things in the model, especially when you're dealing with constant field or innumerable type fields to have actually a class represent that relationship. So, you know, you know, status equal one, you know, you don't access it directly. You never set it to one directly. You will set it to, you know, status is valid being one, you know, as representation. And basically, we wrapped, you know, those innumerations, you know, throughout our models. And basically, the big thing is that we wanted real objects. Basically, when you start thinking about what kind of data you're gonna need for your testing, and that's very important because you wanna have reproducible test case and reproducible data. We actually wanted to start building real objects, having the real objects, the variations, the relationship there. So, in that sense, you know, your testing really start when you're actually writing the test data. And I think, you know, there's a lot of advantages on doing that. We stole, you know, stuff from, you know, things that are great, you know, there's some stuff out there, you know, for dealing, you know, with testing, you know, the whole idea of, like, populating the test database once and then running the test and basically, you know, rolling back the transaction, you know, as the spec is done. So we stole, you know, this stuff, you know, from Rails and RSpec and things like that. We also have, you know, an object dictionary so that you can do, like, fixtures, you know, you can refer to other objects, you know, as... Named fixtures, essentially. You know, as Fred or whatever. And, you know, again, you know, a simple REC test, you know, REC test or just REC, within any project, we run every spec that is available on that project. And one of the good things that actually came out of this is we were able to store our fixtures as the tests in the sentiment gem. So, say one of our other applications actually needs to populate their test database with the data that's from the sentiment gem. It can actually just do it by requiring that gem and then just loading the fixtures programmatically in its spec test. So we can have the exact same data structure or the exact same data load made available from our sentiment gem for any application that needs to manipulate the data in the database or query something or, you know, it needs to do some, make some decision based on a field in the database. And we can keep the base test database infrastructure with all the different class or all the different exact test cases that are in our database available for all the other processes in our system that actually need those to make some decisions. And I think, you know, one reason in World One there was actually no testing because we have, you know, quite a bit of schema out there. There's like 120, 130, you know, different models out there that represents our, you know, business models. And I think, you know, eventually you get to a point where, you know, what should I do? You get overwhelmed, right? If you don't start from the start, then it's a downing effort to actually start regressing back and putting, you know, test data back in your database so you can actually use tests to actually check out your models. So this is one example of, you know, how we write a fixture. So this is, for example, you know, we're gonna populate, you know, three records on our, you know, customer model. We basically have, you know, kind of the notion of, you know, breakout and load, which is basically clear out all the data related to a customer and then load it. And this is just protocol really on that object. You know, the runner is actually making sure that all the objects are deleted from the database before, you know, re-injecting, you know, the actual data. And this is done once, you know, at the beginning of the, you know, test, when the test driver bootstraps, he actually will load up the test data and then that said that data kind of leaves on after that. Yeah, we load up our test data as a spec before all system. And the cool thing too is, you know, now every developer has, you know, his own test database environment. And that's really useful. We also went in and grabbed the stuff out of Rails, you know, for the script console. So now you can be on the command line. You create a new model. You can actually go from the command line and test it directly right there out of that data. So that's pretty useful. And one of the things we actually ended up with was one of the problems in our system. Go ahead and hit that. It should be the next one. Was, can everybody see that? We turned out that we had a big problem in our system one time, where we would do this test and production and it didn't work. We'd run it on test in our developer laptops and it worked fine. You know, everything was work, all the tests passed, we do it in production, nothing happened. No, it just completely broke. You've been here, yeah, when you talk into a developer and you say, hey, you know, this is not working, you know, and you always hear back, what is working on my machine? Right? Yeah. And basically, you know, you don't really know because everybody's got a different setup, right? You worked on his machine because maybe he's running my SQL 4.1 or whatever that is totally independent from what we're actually running on production. So what we have now is actually a set of specs and it's at the bottom of this text. Can you see who that is? Who doesn't have my SQL 5.1 installed on a system. And so we actually have spec tests at the beginning that run before everything else that validates that your database configuration on your developer laptop is actually exactly the same database configuration that's in production. And this is actually something that's really good. And I encourage you all to do this because then you're validating that everything that happens in your database on your developer laptop is exactly the same as how it's gonna happen in production. And it keeps all of your database configuration. We're talking the top level, global SQL mode type things for MySQL or maybe some sort of page buffering or something like that using any number of databases. So it encourages similarity and consistency between all of your different environments. Another one. Good stuff here. Cruise control. How many people use continuous integration of some sport of some type? All right, we'll look about 25% less. Yeah. Cruise control is a vital piece of our infrastructure. If people have generated their hand, is there any particular reason why you wouldn't use cruise control? What's the sad thing? I'm not a developer. No, there you go. Okay. Okay. I mean we really, in order for a framework like this to work across the board, you actually want some kind of police state or something that actually check thing to some level, right? So I go in, I'm checking in something, I'm actually responsible for that check-in because at any point in time, and because we have a 24-7 type system, we may have to do some emergency deployment, right? And we wanna make sure that things that is in the code base actually works. I mean, for example, if cruise control is failing, I can deploy because I could really hold something totally, hold something in production. One of the things we leverage with cruise control is its ability to do dependency builds. So like for us, if the primordial build, if the primordial gem gets rebuilt and we use gem dependencies to do the dependencies between all of our different gems and I'll get to that in just a minute. But we can tell cruise control, hey, if the primordial gem is rebuilt, you need to rebuild everything else too. And you get that full integration between all of your different tests. So say we changed a model in sentiment, it does something different. Well, sentiment may pass all the tests, but say we have another gem up the road, it's one of the quartz gems, there's some unknown thing and it's something that wasn't actually getting tested correctly, but the quartz gem blows up because some batch job it was gonna run fails because you changed the data type on a column and the quartz gem depended on it, the sentiment gem didn't. So this way we can see, hey, if something changes in our database, it may have side effects up the road in one of our other gems. So it just makes it really good way. We found multiple, I forget what the exact term is, when side effects, untested side effects that we didn't know about when we made a change using cruise control and it saved us hours and hours of work. How many people here work with a team of developers, not just a single person? We're about 50%. Okay, that's good. How many have a team of more than five? Okay, yeah, our current team is 14, so. Yes, they are all US based. Except for me. Except for you, yeah. I'm kidding. Well actually, that was a good question. How many people are using outsourced help from outside the country, Ukraine, France? Where are yours based? In China, okay. Yeah, so continuous integration, that's actually probably something that's helpful in that system too. If you're checking, go ahead. We do. We do. Actually, the cruise control generates the documentation as one of its final steps and it's just available for that build. So the documentation that's available in cruise control is the documentation that was generated for that build. Yeah, and we encourage lots of documentation and we used to be really bad at it and we're much better these days. Yeah. Not all the way back to customer feedback, but we do have just the generic documentation that's built and all the RSpec tests. So if you click on a, and we show the source code for the different, it's just straight RDoC. It's not anything in the wiki or anything like that. To answer your question? Okay. Okay, how many people know what that symbol means in gems? In terms of gems? One. One. Only one person. Two, three, four, five. Okay, this is actually a really good piece of work. Twiddlewack, is twiddlewacka, I think is the appropriate term for it. And this is something that RubyGems has that I don't think any other dependency management system does. Does anyone know? I don't know. Okay, what's that? Next slide. This is what it does. Oh. Yeah. Okay, twiddlewacka means, if I say twiddlewacka 1.1 in my gem spec, that says the dependency that's satisfied here is 1.1.astrix. 1.2 will not satisfy it, but 1.1.456789 all the way up will. Are you using it all the time? You have your own gems, you're doing it that way? Nice. Awesome. Nice. All right. Yeah, because I mean, the big thing, when you're starting doing gems and different application on the submachine, you really want to tighten up the dependency mechanism so that when you deploy one application with a new gem, you don't want to necessarily hose things that it's already up and running. So basically, this has, there's a catch-22 to this is that typically when we deploy top-level application, meaning at the end of the food chain, we actually want to lock down to the precise versions on which this application is to run on. Yeah, so if anyone has read on the RubyGems document, the RubyGems docs.rubygems.com I believe, there's a little essay that's saying versioning, something along those lines. I encourage you all to read it. It explains how when a major number should change, when a minor number should change, when a build number should change, and we actually try to follow this as close as possible. Okay, deployment. Okay, deployment, so in essence, we also wanted to have an easy way to deploy all this stuff. So our applications we've built, as you saw earlier, when you boilerplate a project, you actually get deployment script associated with it. Capistrano is what we use. That basically knows exactly what to do and the different environment that this thing is to go into. One thing I need to interject here is we actually run Rails apps, Mirb apps, command line apps, Quartz jobs, all these different things, they're all deployed the same way, Capistrano. So we have a consistent deployment mechanism across all our different application frameworks. So as we pointed out earlier, when we are on the cruise control, so the cruise control box, we're not only using it just for running our tests, we're also putting it to work. So at the end of a valid successful test pass, he actually will go out and build the gem for that project. So when that project test suite has actually passed, successful or green, he actually goes out and build the gem for it and then pushes it to our gem server. So that's a local gem server. So how many people, the Java people, disguised as Rubyists, how many people know about Maven? Okay, this is where this come from, right? I'm a huge Maven fan and basically that's what I wanted. I wanted a local depot where things that are locally based, things that are created by the team are actually deployed to. So in that sense, that's really what the gem server is. I equated to the Maven repository and that's what we push our gem to. So then, basically when we actually need to deploy the application, he actually will go and if there's external dependencies for the gem, he will actually go to Ruby, Forge, Git, whatever. But for our internal CI based gems, he actually will pull it from the gem server locally. So this gives us, actually, let's pull that down. This actually is a quick diagram that shows how our deployment mechanism works. So before we said we would deploy, in world one, we would deploy with an AntScript that would deploy directly from the developer's workstation to our production system. Not a good idea, not really not a good idea. So what happens now is when I'm done with the latest check-in on something, I'll commit it to SVN. Now cruise control automatically pick it up with its SVN monitoring, build the gem, if everything goes green, then it'll actually push that successful gem to our gem server. How many people actually run gem servers in their own environment? We got one, two, three, four, five, six, okay. So the gem server is critical piece because when we actually do the deployment, the Capistrano script, run from your developer's workstation, goes out to the deployment target and from there, it'll actually calculate the dependencies and I'll get into that on the next slide, calculate the dependencies that that gem that was pushed from the gem server needs. It'll make sure that that production system or the test system or the development integration system has all the gems that are required for that top level application to actually work. And so we can actually get into, now go ahead and hit it, I guess. You can go ahead and we can have our MIRB apps, our Rails apps, our Quartz jobs, our command line applications, all calculate their dependencies at deployment time. We wanted to think, one of the things we really wanted to make sure we had is when an application is deployed onto a system, when it's finally deployed, it knows it can run. Instead of actually running the application, it says oops, unable to find gem, we wanted to make sure that that would actually never happen on a production system. So how many people here use Capistrano for the deployment tool? Okay. Good number. So I mean, you're familiar with the check task that basically goes out, check the server, the remote server you want to deploy onto and say, all right, you have NRA gem out there, version 1.x, yes. Do you have this directory out there? Yes, no, things like that. So basically what we did, and that's part of our primordial Capistrano task, we extended that check task to basically use this dependency mechanism. So now you have the same code that actually specifies what this application is dependent on to actually being used at the time of deployment. So now I know at the time I need to push the code out there that those other things are required for that application. And this is how we do it. Every single piece of code we have, you know, a MIRB app, Rails app, or anything that's a gem, it has a config dependency.yaml file in it. It's guaranteed that's part of our basic project infrastructure. And what do you have in there is it can say its own major and minor number. And then the third build number for use with the Twiddle walker is actually the RSVN build number. That's what we use for our third digit. And then we can have dependencies. So in here we say this dependency, which I think is for the pipeline gem, depends on our sentiment gem, depends on our 1.1 of a primordial gem, lock file and faster CSV. Thank you, James. So, and the way, and that just does it, the Capistrano task will read this thing when it deploys that gem, it'll unpack the gem, look at it, see what the actual dependencies are required, then do all those pre-installation checks and make sure they're all there. And when it fails, it'll actually just do a gem install from our internal gem server. And then when, another thing that happens in our dependency is we actually have some legacy code still from world one that uses ActiveRecord 1.14. So one of the things we've done is we're able to say, this is one of the first lines of code that gets executed in every single one of our applications. We have a top-level base class that's involved in everything. And what we can say is pipeline.dependency.lockdegemversion. And this, what it does is it leverages the gem task to actually require the specific gem version that you need for your system. And that way, if some other extraneous piece of code accidentally requires the gem and doesn't specify a version, it'll blow up. And that piece of code we can then find easily. So, internal gem server. We have a lot of reasons for having an internal gem server. Primarily, because we need to host our own gems. Secondarily, we have vetted third-party gems. A lot of the stuff that's on Rubyforge, it moves fast. Some of it, or stagnates one of the two. And we wanna make sure that the version of some open-source piece of software that we use is the exact version that we know works with our system. So with an internal gem server, we can lock down all the versions that are made available. So we know that when ActiveRecord upgrades to the next version, it won't accidentally install it. And we say gem install ours, and the gem dependencies resolve. And it accidentally installs the next version of ActiveRecord instead of the version we want. So there's also like, we've all seen projects going in and out of scope. Like, think that's only like a good idea at the time. Somebody go in, spend some time developing it, and then drops it for one reason or another. That just happened to be one of your dependencies, right? So now that project, you go out to Rubyforge no longer there, what would you do? Yeah. So in this case, yeah. If a third-party gem, it's out on the internet somewhere, and it just disappears. Well, now you can't actually install it again because it's no longer on Rubyforge or something along those lines. And then the third, and then the fourth one here, how many people have their production systems, some of them not able to actually access the internet? One, two, three. Probably work for a Fortune 500 company, I'm betting. All right, see. Government? Government, yeah. So another reason to have an internal gem server is for this exact reason that your production data processing machine can't actually do a gem install. But you can tell gem install from an internal gem server and it'll work just fine. Yeah, I worked for an energy company back in Denver and I didn't even have access to that box actually to physically give my code to some sysadmin and say here, put that in a production. So we're actually working on a gem called Stickler. It's on GitHub right now, it's completely unusable. You can fork it, you can look at it, whatever you want, but it won't do anything right now. But it's nice though, right? It's just a pretty face right now. But what it's gonna do is be able to manage this internal gem server. One of the things that we wanted to do is when gem's installed by default, if there's no source defined, it'll go to rubyforge.org. One of the things Stickler's gonna do is be able to repackage RubyGems on itself so that the default location for you to get your gems is what you define as your own internal gem server. And one of the things I should also mention in here, how many people, all of the other processes, all the other systems on your applications are run with the package management, RPM, Debian packages, Solaris, AIX, how many people use the package management system for your system level stuff? Okay, that's another, it's less than I expected. That's, for us, we have, I don't know, right now, probably 60 or 70 servers in our own racks, and we need to make sure that every single one of them has the exact baseline done. So some people use Puppet, how many people use Puppet? Okay, Puppet's pretty popular. We actually use CentOS 5 on all of our production systems. So what we've done, and this is, we package Ruby ourselves and we package RubyGems as RPMs and a lot of our fundamental LibXML, and all those different types of things that we use. So understanding your system level package management is probably, I encourage you to know how to do it, know how to roll your own packages and whatever that system is, and then you can track everything. We have a YUM group install, how many people are RPM, CentOS? Have you ever used YUM group install? Okay, we have what we call CI base, which is a top level group install that installs SBN, a compiler, Ruby, RubyGems, Nagios, and a bunch of different, yeah, SQL, SQLite, different, SQLites installed by default, right? Oh yeah, MySQL clients and all that kind of things, so that we know when a system goes down, when it comes in from Dell, it gets shoved into the rack, the kickstart kicks it off, then we can do a YUM group install CI base, and then all we have to do is say, cap deploy and we're done, and we have an up and running system. And again, that's eliminating again the unknowns of having something in the system that hasn't been tested for, right? Yeah, so making your deployment repeatable, I mean, whatever it takes, it used to go at the beginning, it probably took us maybe a week to get a system from zero to production, now it's probably under an hour. Minute and a half, that's what you have? Oh, a minute left, okay, yeah. Yeah, well here we go, thank you. Somebody's keeping track, Tommy's keeping track. Okay, so basically again, we thank you all for coming and being here. Basically, Jeremy company is a copy of free time, mine is liquid rare, and we have executive contributors to this project, which are basically a list of our, some of our favorite gems out there, and we'll take the people for taking the time and contributing those to the open source community. I mean, we find them useful on a daily basis. So, I think we can maybe close with some. We have a little anecdote we have really quick. One of the things we found out with our system is to be able to get somebody up to speed really quick. And we had a summer intern this year who came in and with our deployment strategy, with all the rate tasks with the primordial gem, everything along those lines, he was able to understand every single gem, get going, and then actually start writing tests and understanding how everything was. And I think that was only in his second week of work. On the second week we actually wrote an application. Yeah, second week we wrote an application and was able to deploy it. So this type of thing enables you to bring up a developer and bring them up really fast. Yeah, I mean, it's, to us, it's kind of rejoicing. I mean, we totally scarred this project, right? We were like totally on the deadline, we actually had to physically deliver application. And Jeremy and I kind of looked at each other and we came to basically the same conclusion at the same time that, you know, what we currently had was not gonna work and we had to do something about it. So we scarred this operation of the gem framework and really it's rewarding when, you know, some young guy, you know, that is not, you know, doesn't know a lot of stuff, but he can come in and he can be that productive in that short amount of time, that's really cool. Also, you know, members of the development team, you know, eventually you see, you know, some happiness there, right? I mean, it's the same success as Rails, right? You know, you, when you look at a Rails application, you understand that, you know, under app controller, that's where your controls are gonna be, your models are gonna be there, you know, as a configuration file and so forth. So when you look at the gem framework that, you know, we've created here, it's very similar. So there's that notion of familiarity and then basically, you know, when people gets into the code base, there's a little bit of a curve, you know, they'll get me wrong, but there is a sense of happiness and a sense of ownership of being able to see exactly where this is going and that has been really cool to see. And I guess I wanna, you know, close with saying, this is what works for us. It may not work for you, your environment may be completely different, completely different requirements, you know, and we actually wanna, you know, talk and see, is there something we actually completely missed in this whole system? So, you know, if we've got enough time for a question or two, one question, and, or otherwise, we'd love to meet, chat, you know, we've got the evening times, we wanna know how you- We'll be here all week. We'll be here all week. Yeah, there you go. So, Jim. Any questions out there? What do you guys think? Here we got one right here. Do you have any of all this cool stuff you've got, though? Do you have examples of this anywhere that- Ah, we've been talking about this. So, you know, as you heard me mention earlier, you know, Maven, kind of a big fan, and you know, when I think about this, I think he would be a good contribution, you know, to open source community to basically build some kind of like environment that encapsulate the essence of your project. So, Jeremy and I have been talking about this. We think about it, but it's kind of a hard problem to tackle, because everybody has different practices, different environments. It's kind of hard to generalize all this stuff, but, you know, we're gonna release, you know, probably good pieces of it, you know, separately, maybe at first, and then try to encapsulate as we see that we're getting more momentum that way. Yep. So, we're gonna try, do our best, but you know, everything that comes down, you know, out of our system is basically specifically for our development and production environment. So, stickler for sure. I mean, give that a shot. Some of these rake tasks that we've got going on, we'll try to do what we can to get them, you know, publicly available, or maybe just write about them and say, this is what we've done, this is how we can do stuff. Sure. Sure. And then the slides will be available on the GitHub, very soon. Thanks. Yep. Video Equipment Rental Costs, paid for by PeepCodeScreenCasts.