 a lot to cover here so I'm gonna go ahead and get started. This is open sourcing, real talk. So who am I? I'm Andrew Evans. I work for Hired. I've been there a little over two years. I've worked for all kinds of different startups for like nine, ten years. I've been working with Ruby on Rails since about 1.8 or so. If anybody remembers some fun Twitter-related arguments back then, good times. I've been writing bugs in PHP or QBasic or all sorts of things for longer than I care to remember. So what is Hired? Hired is the best way to get a tech job. As a candidate, you make a profile. We put you live in our two-sided marketplace and then companies ask you to interview with them and they have to give you salary, equity, everything else up front and you decide if it's worth responding and having that conversation. As a company, this is the best and fastest way to get great candidates into your pipeline. But this is not a pitch talk so I'm not going to focus on that. Instead I'm going to talk about open sourcing. This is something that Hired's been doing basically since the beginning. We have 46 public repos on GitHub. A lot of those are forks where we've needed to make some modifications or thought about modifying things. But some of those are open source projects that maybe you can use. These are just a couple. We have a PubSub layer over SideKick. We have a query builder for Elasticsearch. We have Fortitude which is our CSS framework which is built to scale. It's built for teams. We even have some smaller ones. We have a rack middleware that will log stats from the Puma web server. So if you're using that, it can be helpful for gathering more info. And we have a tiny thing which will change the color of the lights in your office. If you're using something like a Phillips Hue, we turn our lights red when the master build fails. So it's a good incentive to get on Slack and figure out what happened. I'm just going to dive deep into these two so we can talk a little bit about what producing these projects were like. I'm going to cover whether it would be worthwhile or not for you to do any open source work extracting anything out of your app and making it public. I'm going to talk about some of the challenges that we faced in doing that. And I'm going to talk about how it played out across our team, like how did that affect the dynamics. And I'm going to try and get a little technical and talk about what we learned about Ruby and Ruby gems and Rails. Hopefully we'll have some time for Q&A at the end. So why should you open source things? Well, if you've got some logic in your application, then clearly every other Rails app is doing pretty much the same thing. So if we all share our code, then we could all do this stuff automatically. When you open source something, the community will fix bugs in it. They'll add features. They'll document your things. It's free labor. It's amazing. You'll get developer street cred. You'll get lots of respect and appreciation. You'll get 10K followers on Twitter and hundreds of thousands of GitHub stars. And people will open issues and say, while there's no bugs or problems with this, I just want to say thank you and we love you. So literally none of that is true. You may have bits of application code that everyone needs across basically every Rails app, in which case what's it like working at Thoughtbot? But for the most part, you may be doing some pretty specialized stuff. Most of your app is business logic, so that lives in your business app. Some real talk about our projects in particular. I pulled some data off of Best Gems. Best Gems basically scrapes data off rubygems.org every day and tracks like total number of downloads, average number of downloads, et cetera. And it gives your Gem a rank out of the 130,000 plus gems on rubygems. You can see ours are not quite in the top 100 or 1,000. We're probably not going to be on the public leaderboards anytime soon. Our average daily downloads, I adjusted and got rid of weekends and like holidays for that. But basically it's a little bit over the range we'd expect for how many times a day we deploy to Heroku. So hopefully these aren't all just Git push Heroku. Oh, look, we got more downloads. I also pulled some data off GitHub. The number of stars and followers we have is a bit more encouraging. There's probably some people outside of our company that care about these repos. But we're not getting people opening issues just saying how great we are. I found some gems that are similar on the Best Gems ranking. So one of them is a framework that lets you build bots for Google Wave. Does anybody remember that? That existed for a hot second. One of them is a command line interface to the pirate bay. I think the founders announced a few months ago that they'd been sued and thrown in jail too many times. So that's dead. One of them combines ActiveAdmin and Trailblazer. Trailblazer is the object-oriented, explicit over implicit, loosely coupled, well-organized framework that you can add on top of your Rails app. ActiveAdmin is literally the opposite of every one of those values. So I'm not sure what you get by combining them. The last one is a command line app that gives you a random Mitch Hedberg quote. I used to use that all the time. I mean, I still do, but I used to, too. So what we found is if you don't market your gem, then people don't really just find it organically. We haven't done a lot of promotion for these. I don't think we post about them on our blog too often. This may be the first talk where we've gone deep into our open source stuff at RailsConf. So we haven't gotten a lot of people that just found this out of the blue and said this is the greatest thing. Was it still worth it for us to open source this code from our application? I'll dive in a little bit and we'll see. So Reactor is our enhanced interface to Sidekick. This is how we do a lot of our background processing at Hired. Sidekick, if you don't know, is the premier background processing library for Ruby and Rails application. It was made by Mike Perham, that guy. Mike is awesome. Thank you very much. It's been a wonderful system for us at Hired and it's been rock solid. So the thing that we were looking at when considering Reactor was, when you have, say, a candidate updating their profile, you may want to do a couple of different things in the background. You might want to notify the talent advocate that's working with them. You might want to bust a cache or update a saved search or something like that. So with Sidekick, you have worker classes for these things and you perform a sync and those things will happen in the background outside of your web request. But if you're doing a whole bunch of things, then you may have a whole bunch of different workers that you have to tell to do the thing. So you face a choice. Do you want to copy and paste all of those calls everywhere that you're updating a candidate's profile? Or you can have one worker that fires off and then that calls other workers, which might then call other workers. For the most part, this isn't a huge deal. I think it's a recommended practice. But if you're like me and you tend to forget things, then you might throw a cycle into your workers and they'll spin up 100 or 10,000 jobs. However, many activate until you can shut down Sidekick and figure out what happened. So basically with that pattern, you end up with this big graph of things happening throughout Sidekick. You have to kind of model that in your head and it can make it a little difficult to track down where exactly specific jobs are coming from. So Reactor is a publish and subscribe system. Basically, you publish an event. That event has a name, which is just a symbol. You can pass it any arbitrary data. There's also an extension so that you can have active record objects publish an event. And then those objects are serialized and deserialized and available within your event code. So this will kick off a single Sidekick worker which then locates all these subscribers to that particular event name and it will in cue a job for each of those. We wrote a convenience method for our controllers called action event. This will publish an event from the current user and it will merge in parameters or any other data that we think would be useful for tracing, tracking, analytics, etc. So what do subscribers look like? You include this subscriberable module that gives you the class method on event. So you say on event, you give it the event name and then you give it a block of what to do when that event happens. So in our first example, we pull out the actor, which is whatever published the event. We get their ID and then we can bust a cache related to that particular record. You can also set a wild card handler. So this thing will respond to all events. And we use that for logging site activity to Postgres or to anywhere else that we want to have that analytics data available. There's more options to it. Instead of passing a block, you can pass it an event name. If your objects and functionality are a little more complicated and you want to unit test them, this can make it a little bit easier. You can also pass options to it like delays and perform later. So for example, once an interview request happens from this subscriber, then two days later we'll send an update. And you can do some logic in that, like figure out if it's canceled and then abort. We also have a system for creating model-based subscribers. So in this case, we create a subscriber called admin notifier. This is going to live in our Postgres database. And we define the on-fire code to say what happens when this event fires. The neat thing about this is you can have it subscribed to multiple different events from the database. So if I'm an admin and I want to be notified when these three different things happen, then I can just do that through our admin interface and not have to deploy new code to handle that. These can be a little bit magical, but also their use has fallen off a little bit. Finally, you can have records publish events based on the data in those records. So for example, if we have an interview that has a start at and an end at time, then we can tell it to publish events at those times. We can even pass it some conditions and some other things similar to validations and callbacks. And the nice thing is if the model gets updated, the start at or end at changes, then it will automatically reschedule that for us. So did it help that we took this code and instead of just building it in our application, we put it in an open source repo? Well, when you do that, you get this nice green field project and it actually changes your desktop wallpaper to the old Windows XP wallpaper. Green fields are great. You don't have to worry about your legacy code. You can just start developing fresh. And it keeps your code separate outside of Kansas out of your application. So that produces a little bit of friction to updating the library, which it turns out can be a really good thing. The code for reactor is kind of out of sight, out of mind for day-to-day feature development for us. So it discourages making small arbitrary changes. If I want to say, like, oh, this should be calling instance methods, not class methods or this or that, I'm less likely to take that on because I have to context switch and then publish to Ruby gems and go through this slightly higher friction process. Because it turns it into a lower churn section of code, it keeps the pattern and convention clear. This action event is used all over our application and basically it stays the same everywhere and it has just kept working. We had to make some updates for Rails 5 and I'm about to make some updates for Sidekick 5. But for the most part, we don't have to think about it too much. Also, when you have this in a public repository, instead of your private application repository, it's going on your permanent record on GitHub. So that might add just a little bit more pressure to do it right. You might be a little more attentive to things like documenting all of your methods and making sure that your test coverage is 100% and that your code climate score isn't going down. Finally, when you're publishing through Ruby gems specifically, you have to be a little more thoughtful about your versioning. When you're making changes, you really have to think about, okay, is this going to break anything? Is this backwards incompatible? Or is this just a feature or patch release that isn't going to break anything existing but maybe provide a new feature? Also, when you're building outside your application and you're not thinking so much about features, it gives you a chance to think more clearly about the pattern of what you're building. So, instead of thinking about what do I need to do this feature, you think about what is the minimum information that this open source library needs to do its job. So, you also have to think about how Ruby gems and Rails and Bundler and these other tools work. One thing I stumbled on to pretty early is that requiring Rails is harder than it sounds. I looked through things like device and paperclip and Sidekick and they all require Rails in totally different ways which I found amusing. Some of these libraries require Rails for all of their functionality and some of them have a few Rails specific extensions and that caused me to dive deeper into what is a realty? Is that something I need? Where do I put the required statements? How do I manage the dependencies for that? And then it makes you think about whether you want to use Rails specific extensions inside your gem that's living outside your application. So, do we want to use ActiveSupports concerns pattern which is a pretty nice way of extending objects or do we want to use the basic Ruby include and extend patterns? And then you think a little bit more about the overall design pattern. So, when we're thinking about React or we're thinking I just want this action event method in my code so I can just put events all over the place. But when you're designing a gem from the ground up it's like, okay, what's the name for that pattern? So, when you go through, you know, design pattern, blog post or the gang of four book or whatever, you say, oh, this is a message bus. So, we're going to make channels that events get passed along. These events are going to have arbitrary data. Some people get this confused with event delegation particularly if you've done a lot of jQuery, you can have multiple listeners for particular events there. But those actually form a chain and any of the listeners in that chain can abort the entire process. With a message bus, all of these subscribers get the message regardless of what each of them are doing individually. So, if you wanted to see some examples of subscribers and message buses written in Rails, then Shiny new Action Cable has a pretty well put together well documented example of this. I have a link here to the subscriber map. Or if you want something a bit more core, active support notifications has been in Rails for quite a while. If you're wondering how all of your database statements get from the application code to your development console, this is what's doing it. And this has a link to subscribers and the class that's responsible for organizing them and mapping them. So, TLDR, publishing the reactor gem didn't make us famous, didn't really help establish any of those selfish goals that we put out. But it did help our team establish a convention. It has quietly kept working for years and it's allowed us to write that code in a way that's smaller and more loosely coupled and better organized. It's also a good excuse when anyone wants to work on it to go in and look at the pattern, look at the examples and see how does message bus work, what's cool about this. The other one I wanted to talk about was stretchy. This is a composable query builder for Elasticsearch. Elasticsearch is a pretty amazing technology for full text search. It's based on Lucene. It's similar to Solar. Basically, when you want something more advanced than a relational database I like statement, then this is a good place to go. And yes, we spent an inordinate amount of time thinking about the name for this because no gem is going to take off and become popular unless you have a really clever pun as the name for it. So, I wrote the first version of this. The goals that I was thinking of were I wanted to have an active record-ish query syntax that we can use for Elasticsearch. I wanted to have these immutable query objects which make handling and caching responses really easy to manage. I wanted this to be just a query builder for indexing our models to Elasticsearch. We already have reactor. So, we can index on specific events. We have serializers. So, I figured we'd just keep our Elasticsearch gem out of all of that business. I wanted to make this flexible enough for pretty complicated Elasticsearch queries. Because this is just a query builder, it's taking in a bunch of method calls and then it's producing some JSON that we're going to send to Elasticsearch. That means we don't really need any dependencies which is pretty nice. We don't have to require all of Rails for this one thing. And the end goal of all of that is we have these composable scopes that we can use in the same way that we're familiar with from ActiveRecord. So, here's an example of what that looks like. This is a query that should get us back Lynn Conway. She basically invented modern microprocessors. So, it's a good person to find in our candidate database. You can see here we have the where method. Pretty similar to ActiveRecord. You can throw some fairly arbitrary parameters at it and it will figure out what the right query is to apply those. But then there's some more Elasticsearch specific stuff. This match query, for example, is like the most basic full text matching relevant sorting query in Elasticsearch. And then we have these boost methods. That gets into what's called a function score query. It lets you tweak how your search results are going to be ranked. When you look at the query that's generated, it's this giant messy JSON thing. When I first started working with Elasticsearch, I tried to figure out how to make these giant JSON documents composable so that we could mix and match them in different ways. It turned out to be really hard, which is what inspired this whole ActiveRecord-like approach. So, some other example methods. I won't get too deep into these, but you tell it what index you're going to search, what type you want to get back. And then you can use where for filters. You can use match for full text matching. You can or multiple filters together by using the should clause. This comes straight from the Elasticsearch DSL. And then you can add boost to tweak your relevance algorithms. And finally, if what stretchy has isn't enough, then you can go lower level. You can say this is just a query and then throw some raw JSON at it, and it will figure out where that needs to go in your final query that's going to be sent over. Elasticsearch has a whole lot of stuff in it. It has about 7 million tools for doing different kinds of searches and figuring out relevance and producing the set of results that you want. So, it became obvious pretty early we couldn't support all of that. So, the goal was just to get to a system where we could do what we needed to in the app and then hopefully expand that as we got more and more features and hired. So, how do I get started making an active record chainable interface? I pulled out this 2010 blog post that I've had bookmarked forever, RailsTips.org, thanks John Unimaker. This stuff still works. It's still pretty much the same in Ruby 2.3. Basically, you have a class that class has some data, maybe an array of nodes, let's say, and each of your chainable methods returns a new instance of that class with some new stuff added to it. So, when you get down, you're like, okay, we're building up this array and then we're going to compile it to this JSON query. What does that look like? Is there a name for that? How does it work? Turns out it's called the visitor pattern. I was looking around at how other repos do this. I looked at active record and, oh, that uses A-REL and A-REL has a visitor pattern. So, that has a collector class and then it has a tree manager for figuring out what was collected and compiling it and then it has multiple different visitor classes to take that tree of abstract nodes and compile it into a query for, say, MySQL or Postgres or Oracle, I guess, if you're into that kind of thing. This is an example of what that tree of abstract nodes is going to look like. You have a select node and then that can have multiple child nodes, like what are you selecting, where are you getting it from and then any filters that are going to be applied to your query. And then those filters can have child nodes and those have child nodes. So, the visitor will traverse all of those and compile it into the final string. So, Stretchy didn't need that full stack. There's really only one elastic search. We decided because time, we only wanted to support the latest version. So, we can basically combine the visiting and compile steps. But the pattern was still pretty much the same. Some challenges we ran into in having this query builder live outside of our repo and kind of its own thing. I was a little confusing to the rest of the team initially. If you look at other gems for integrating elastic search into your Rails app, they do a lot more. Elastic search Rails and search kick and chewy. They all tie into active model. They have some other miscellaneous helpers and Stretchy doesn't do any of that. So, other developers were like, how do I do this or that with Stretchy? And it's like, well, we don't actually do that. And it turns out when you have this API-like active record, the whole point is to make it easier so you don't have to look at all the documentation for everything all the time. But that turns out it hides a lot of the actual features of elastic search. It hides the underlying complexity. So, other engineers will look at this and say, oh, I'm just going to make this active record-like thing. But it's like maybe you're missing out on a lot of the stuff you could be doing there. When we have an open source plus our internal application queries, it can be a little fuzzy deciding what should go in the open source versus what should go in the app. Obviously, all of our business logic should be in the app. But if we're using this particular style of query for text relevance or that particular style, like, is that worth having a convenience method for in the open source package? And then early on, we had a couple of cases where it's like, okay, we want to make this feature. But then the gem doesn't support that yet. So, your open source library ideally should not be blocking any future development. So, to fix that part at least, we had to make it more flexible and easier to customize within our application. And then a little bit later, we made it more flexible and extensible so that we could optimize those queries in our application. And then later, we made it easier to customize and more flexible and more extensible. And that turned out to be a huge win, actually. You can really see the process of editing and refining to get a flexible, extensible solution because, ultimately, it needs less code. And it makes clear what the bottom level versus mid-level building blocks are. So, I actually graphed this for the strategy repo. I took each of our master commits and looked at how many lines of code are in it. So, we've done one or two rewrites. The first one, I'm not going to call it bad, but it certainly added a lot of code. And then, over time, we've been able to just keep cutting that down. And at this point, it's back to being a pretty small gem. So, the TLDR for that, by making this open source gem in this nice green field, we got to learn the cool new visitor pattern and see how do you do that in Ruby. For me, it was a great excuse to dive in and read a bunch of code from Arel and Mongoid and other, like, more mature frameworks. By having the core interface be open source, that allowed us to keep that very simple and then keep our complex business logic in the app where it belongs. And it's helped us keep a good separation between elastic search queries and the internals of, like, our models and our controllers. So, to wrap up, this whole thing, open sourcing has been pretty cool for us. Even if, you know, it hasn't made us rock stars, we don't get, you know, a learjet to fly us to RailsConf because of how awesome we are. Even if this guy is your only watcher on GitHub, you can still get a lot of technical benefits for your organization. And even if you don't get free labor, it can help you internally with your team. It can add a little bit of friction, which can be a good thing. It can be a bad thing occasionally, but we found the trade-offs to be worth it most of the time. Thinking outside of Ruby on Rails and sort of leaving behind the conventions and crutches that gives you can make your code simpler and clearer, easier to understand. Developing open source can give you a good excuse to learn more about computers. You learn more about Ruby and Ruby gems and Bundler and all the other systems for how you get that back into your application. And with Greenfield Blank Slate Projects, it's a great opportunity to be a little bit more thoughtful about the code that you're writing. So I told you we'd save time for Q and A, so here we are. Any questions for me or about these open source projects or about Hire? Yeah, so the question was elastic search changed versions from 2.x to 5 because reasons. They completely changed how the query DSL works. There used to be this big separation between queries which handle relevance and then filters which just affect your result set but don't affect relevance. So I think it took us about three or four months to get that upgraded internally. So as part of that effort, we upgraded the open source gem. That was surprisingly easy because we have this stack and tree pattern. We basically just had to change, okay, in the tree, instead of doing this, we do that. So I can point you to the pull request for that, but I was actually shocked how simple that was. Alrighty. Well, thank you all very much for coming. I'm Agias on Twitter and I've got the slides up on my website. Use hired. If you're looking for a new opportunity or if you're a company looking for candidates, get online. Give it a shot. Or come work for us. It's a pretty cool company. So, alright. Thanks, everybody.