 and I am a software developer at VitalSource Technologies. The primary office I work out of is in Raleigh, North Carolina, but we've got offices here and there. We're owned by Ingram Content Group, and they're headquartered in Laverne, Tennessee. Today, what I'm going to be doing is talking about a history of a suite of our applications organized around Common Core, and this core dates back to the very early days of Ruby on Rails. I'm going to start out by doing a little bit of stage setting. This here on this chart is a rolled up aggregation of seven different repositories that we have dating from August of 2005 all the way to this past week, showing the number of commits per week. It's not broken up by a repository, this is just like an aggregate number of commits. It presents a substantial but fundamentally incomplete view of what one segment of what our company does. Overall, as you look from the left side to the right side, you see that there's a general trend line that goes ever so slightly up into the right. Then you look just past this area around 2015 into 2016, and you see a giant spike there on the right side of the graph. We're going to take a deeper look at that a little bit later in the talk, but the nice thing is that with these seven repositories we have a lot of history starting from diversion, going all the way through our transition to get. I'm going to start with a little bit of a prologue here. VitalSource does a number of things around ebook publishing. We allow publishers to send us books to build. They can set prices, determine availability, different things along that nature, and then we will sell them on to end users, particularly students in learning applications. My start with this began in, actually I had this wrong, this should be October 2015. We're in the middle of an upgrade from Rails 2.3 and Ruby 187, and I was gaining experience across the app suite by working on this project. One of the pieces that we needed to do was upgrade a lot of JavaScript. Reason being is that the app used a lot of prototype, which was going away in favor of jQuery. We had a lot of the old RJS style linked to remote and linked to remote JS calls and a few other underscore remote methods that we were using. You have these magical pieces that weren't really JavaScript. It was these magical Ruby incantations that became JavaScript in the browser. This led to us upgrading that non-Ruby flavored JavaScript, basically consolidating on one version of jQuery and so on. As part of this process, I come across some very early code, and I run git blame, and that leads to a discovery. Part of what I see is some code from the original maintainer or the original committer on the project that I'm working on. That leads me to go to Twitter and say, hey, I've looked at some very old code today, and this was originally started by Dan Benjamin, who you may know from podcasting fame, and he actually wrote back in. It was a bit astonishing that this code that he had written back in 2005 and had worked on for a little while and substantially the same as when he had written it. I got a bit curious about the history of this. Looking at the code across several of our connected code bases and preparing for this talk and earlier, I've talked to several members of the development team, past and present, and talked to members of the business team who were basically present for the inception and the continuing behavior or the growth of these applications. That's what we're going to talk about today. We're going to talk about team growth and changes at a high level in practices, interactions between the business and between it with developers, code growth a little bit, and then how that code has changed. We're going to go into some code examples, not many, but we'll get to see a little bit. We're going to see how developers in the business interact, and then we'll spend some time looking at reflections and available lessons, and then we'll also talk about some tools that folks can use to actually come at some of these understandings and lessons that they can draw out of their own code bases. As I was putting this together, there were two quotes that came up, one of which was just kind of an offhanded comment, and that was, in the absence of logic, there's history. Then the second one came up a couple of times from two different people, and it seemed like a good idea at the time. In a 12-year-old code base, there's going to be a lot of lessons learned. There's going to be a lot of changes to best practices, what the best practices are going to evolve. You're not going to start out having all of these in hand to start with, but the nice thing is is that this code has been around for 12 years because it had actually been successful for the business, and the business has been successful. That's why we still get to work with it. I'm going to go through the history in four acts. First one is with the appearance of Rails, and David touched on some of this this morning in his opening keynote, but this was the original announcement to the Ruby Lang mailing list about Rails 0.5.0, and that this thing that he had been talking about was actually out there and people could look at it. Now remember, this is before RubyGems was a thing, this is before GitHub was a thing, this was just throwing a library out there and seeing who would kind of grab on to it and go with it. So this is July of 2004. Over at VitalSource Technologies, there's been a series of kind of cobbled together automation steps around building and assembling e-books. It was basically a lot of Java in place, but there was a sense that it was a very limited process and the folks who were working on that were kind of basically overwhelmed with its limitations. And there was also a desire to have a very visual process and so something like Rails coming along actually made it feasible to build a robust web application. So Rails comes in and Dan had actually been experimenting with Rails and he found that he liked working with that more than working with a combination of a PHP front-end and a Java back-end. And so he and another developer, Damon Clink Sales, basically started building. They started building this at the Apple Worldline Developer Conference. I believe this would have been 2005. And they skipped most of the sessions to actually start writing this new Rails application. With Dan focused on writing a front-end piece and Damon focused on writing the back-end piece. Now, looking back at this, there's a lot that we count on in Rails today that was not there. Ajax was new. Prototype and jQuery did not exist. And the Rust conventions that we're so familiar with were not actually part of Rails yet. And what people were used to with APIs were very much around having XML-RPC style or SOAP style or, you know, if you kind of combine flavors and if you look at the original Rails code base that's spun up with this, you actually get a whistle file that comes up in your Routes file, which I found really interesting. The other things that we didn't have at that time is Capistrano wasn't around. So there was, you know, deployments did not have the well-worn path. And so there's a lot of these things that we think of as fully baked into Rails or how Rails applications get shipped now that, you know, had to get figured out. But they got it figured out. And so this is the first commit message. I apologize for the readability. This ended up a bit smaller than I thought. But this is about 400 files from that first pass at the app that got committed back in August of 2005. And what the changelog has is that this was using Rails 0.11.1 from March 27th of 2005. One of the controllers that got committed here was a library controller. We still have one of these in our app today. But as you can see here, that it's still somewhat recognizable as Ruby code and somewhat recognizable as Rails code, but it looks very different than how we might ordinarily write a Rails controller today. So that's the start of Phoenix, which kind of was renamed into Connect in time, and then has more recently become an application called Manage. But there's another application, and that's P2 Services, which is essentially a services layer, and that came along in January 2006. Around this time frame, Rails hits 1.0 in December 2005. Rails 1.1, which brought in RJS and being able to write the magic Ruby incantations that would give you JavaScript for Ajax and the remotes, came around in 2006. 1.2 brought in REST, and the REST conventions that we're more familiar with today started there. Hit 2.0 in 2007 and 2.3 in 2009. I'm skipping over a few releases here. Act 2. Growing the business, the code, and the frustrations that come along with it. Rails marches forward. Heroku gets founded. GitHub gets founded. Passenger comes out, which helps out the deployment story versus... See, if I remember at the time, there was a lot of mongrel in use at the time. And Rails is kind of riding high. It's like, oh, hey, if you're building an app, you're building it in Rails. It was kind of the default thing. Over at VitalSource, the growth of these applications was actually starting to outgrow how the development organization worked. And so there's a tension between building out features versus time taken to go research business problems and answer customer questions and that sort of thing. And all of this ended up being driven, being very business driven and very interrupt driven. And so some of the developers who were present at the time were kind of frustrating to actually get work done. Developers had managers at the time, but their direct interaction with their managers was fairly minimal. The applications grew. And developers typically had areas and tasks of specialization. So it's kind of a nicer way of saying folks got siloed, but folks kind of developed areas of the code that they were comfortable with and areas of the code started developing their style. So you can, you look through different parts of the application and tell, you know, this developer was involved in writing this substantially or wrote the whole thing. The number of applications grew. So Phoenix, that original kind of front-end application and the database layer started in 2005. There were VST models and P2 services, VST models was a way of saying, hey, we're going to take our models and we're going to share them between this code base and the services layer in a third repository. In time, the migrations moved out into their own application called Goose. And then we had a couple of reporting applications for basically providing business data back and that was reporter and Uber reporter. Then a third version of our API layer because P2 services involved two different versions came out in 2010. There's a limitation along the way and that is that the suite of applications along with others that were running Rails were all going to basically be locked together in terms of versions. So everything was going to be running Ruby 187 and Rails 2.3. And the idea there was we only want to have machines that look one way and so if we need to repurpose machines we can shift them to any other application and there's nothing to shift about them. Any application can run on any machine. This started to become a limiting factor because Rails hit 3.0 in 2010. This I'm sure folks who were in doing Rails development back then remember this was the big community merge with Merb. There were also a lot of active record changes around that time. Ruby itself hits 1.9.3 and we're starting to try and get out of the perceived performance hit of Ruby 187. Ruby Enterprise Edition was out around this time too which was kind of a performance-optimized version of 187 that the folks behind Fusion Passengers sold. Rails 3.2 Ruby 2.0 Rails 4 and Rails 4.2 all followed. At the company it's now about 2013 and a new CTO has arrived and not long after there's a new development director as well. And here we start seeing some process and the team is starting to get out of the mindset of shipping everything as soon as possible and more thinking about having very like batched work building things to be more testable. There's much further to go with this and really trying to have the technical management provide kind of an umbrella for development to basically shield them from incoming requests that are important questions to answer but maybe it's like are you happier having folks answer questions or do you need features shipped and trying to figure out a good balance of that. And then we had product managers come in as well so to basically take input from the business about how to shape feature work and that sort of thing, bugs that needed to get addressed, questions that needed answering rather than taking that input directly from the business itself. And so work continued but the upgrades did not. So in time the requirement to have kind of that lockstep that I talked about where everything is going to be on Ruby 187 and Rails 2.3 that got lifted. But as consensus was building to for an upgrade to say hey, you know, we really need to catch up with where the community is at VitalSource bought its primary competitor in 2014. The competitor was Corsmart. They had roughly the equivalent size business as VitalSource did and VitalSource then had to engage in the work of basically absorbing and digesting and changing all the applications to take on this new business. I took about 18 months from 2014 to mid-2015 and that was everything from migrating users you know, picking up new publishing agreements identifying functionality that existed in one place but not in the other and making sure that it was present for what was going to be what was going to exist going forward and then beginning work on a new combined storefront with features that Corsmart had but VitalSource did not and of course Rails and Ruby kept moving and so now these 7, 5 to 10 year old applications, you know, we're still stuck at Ruby 187 and 2.3 and now had an 18 month hit on them being able to deploy. We come to Act 3 where we actually get to the big upgrade. It's Fall 2005 and most of the work of integrating Corsmart has been completed and a plan was made to get the applications upgraded and part of that was our CTO Al, basically he's really an operations fellow at heart once you had to focus on minimal downtime we're no downtime deployments, that sort of thing really avoiding operational impacts on the business where we can and at that time it was getting very hard to get continued support for Ruby 187 and Rails 2.3 you could, there were some corners where you could still get security support for them but by and large that it ended. So the upgrade process got at start and it was mostly a parallel effort across each the applications and it was basically we took it as a step wise progression from Rails 3.0 from 2.3 when that was done we went to 3.1, when that was done we went to 3.2 and to 4.0 and to 4.1 and to 4.2 we did a similar step wise process through Ruby largely interspersed with the versions of Ruby that would corresponded with the version of Rails at the time and that took I think about 7 months to get through everything and get it shipped, tested and out and stable. Some of the challenges that came along with that were mixing urgent work and upgrade work. We had a problem keeping divergent code bases shipable where if we have the bulk of the work on these upgrades happening there was further back changes that would have bug fixes applied and then we had to basically bring those bug fixes forward or we would have fixed bugs in the version and then might have the back port something and just trying to keep all of that balanced out was fairly challenging and further to the fact that these 7 applications all shared a database and so we had to basically coordinate all the cross application dependencies such that the applications could go out in the right order such that they would all continue to function when they shipped on the new code bases or on the upgraded code so some notable challenges with that. There were a lot of things about upgrading rails that kind of forced their hand at 3.0 where there was functionality that came in before 1 that got deprecated at 2 and finally removed in 3. We had to move off of RJ templates to native JavaScript. We had to transition fully to TRB. We incorporated the asset pipeline at long last and then there was a lot of restructuring of active record queries because there had been a lot of work that had gone in with active record in that amount of time and we finally got to take advantage of that. The teams also shifted so what had basically been kind of a few silos here and there there was an API and platform team that focused on backend stuff they had had a changeover in personnel and then that team then grew with some additional hiring and then for the front end applications like Kinect there were additional hires brought in and then there was additional technical management and product management brought in to help those applications as well. And then using some of the lessons learned that we'll talk about in a little bit a new application was started and it basically took database coupling versus API layers and we actually were able to say okay, we're going to not go down that same path. And so when that new product started in early 2016 it basically instead of hooking into the same database it's had the hard requirement of we're going to use an API layer instead and whatever data it needed locally it would save locally but anything that was in the core database Linux had to go over an API and would not actually directly touch the database. And because how Rails is being used has also evolved in that amount of time it's Rails on the server side but it's delivered as a single page using React on the client side and that's been a model for more of our apps that we're using going forward and it seems to be working really well for us. Let's just start getting into some technical culture shifts here and the beginnings of some increased code review culture more collaborative work which is really good. So we come to Act 4 and that is growing the code with a guided tactical approach. The review culture gets stronger earlier and here we come out of we shift our code reviews earlier we don't have a single person to review or responsible for the final sign off and so we're not waiting for that sign off to happen and potentially throwing work back. We start a version 4 of our API layer and here we're taking JSON as the first preference with XML supported for external APIs but as the second choice JSON in our opinion is far more pleasant to work with than some of the large XML pieces that we've been working with. We start returning meaningful error messages with our APIs and we're basically able to take the experience of the earlier versions and integrations and incorporate that into that design. We're doing more opportunistic refactoring now so lots of logic extractions into classes a lot of like oh hey let's set up controllers in this different way we we went very controller heavy with our code early on and so now it's probably closer to what folks would expect more of now and that's more use of surface objects use cases for some of the newer applications and then really trying to only have the controllers handle your interaction with the business object instead of directly trying to do everything as they had been doing before. As part of that we're also moving towards basically treating the API as how you're going to interact with the core data in the central database so I think in time what we're looking to do is kind of get away from having shared models between code bases we're going to get away from having shared migrations that have to run in one repository and then you have to take the copy of the database schema over to the other repositories to get that in sync and then basically anything takes place over an API transaction and then as folks have been using the UI pieces more we're also increasing the amount of work that we do for API and integration work both internally and externally the other nice thing about getting to this point is that each team has the ownership of when they upgrade to new versions of Rails and Ruby and so it's possible for them to update to 2.4 and 5.0 so basically when they feel that they're ready to and want to take that work on the team that I'm working on currently we're probably going to take this work on in the next month or two and then finally we're going to take on a UX refresh of the core application because it hasn't changed a whole lot since 2005 when it was initially written we're going to be increasing our automated test coverage basically filling pieces in development culture is shifting towards green make sure there's tests right code that is easier to test and when we're writing new features let's actually lead with having the automated testing done when that feature ships instead of trying to fill it in later and process wise we're refining we've gone from most teams have gone from Scrum to Kanban and each team kind of gets to iterate on its own practices and how they handle their business in terms of taking new work in from the business where they retrospect putting new processes or taking processes away that sort of thing and in time we're moving some of these apps towards a well-earned retirement our early service layer is due to be replaced with our v4 layer with more up-to-date authentication mechanisms and more restful APIs our two reporting applications that were written against this code base are being replaced with a more capable ETL based reporting application that's being written by another development team and you know essentially we're taking the opportunity to say you know it's okay to let these go we don't have to keep these applications going in order to have the value for the other things that we're still working on so some tools and methods that I think are generally applicable to teams that are working with code bases that are older than say six months or two years or what have you code climate if you use it has a churning complexity graph and so what it can be really neat to look at with this is basically find out what you have kind of in the upper right-hand corner find out and then figure out how you want to dig into okay why are we changing this a lot and why is the code quality suspect for the changes that we have to make and you know accordingly what kind of changes can we make to pull that complexity and churn scored down you know can we extract logic can we you know break apart some complexity into more easily testable things things that don't have to change as much github as code frequency graphs this is for the Phoenix application and if you look kind of on the far right around 2015 on the right-hand side you'll see kind of some spikes there that was where we were doing our upgrade work you'll remember if you go back I'll have it up on the screen again but the commits kind of like we had a big spike in commits but overall the code added and subtracted a whole lot where at least nearly as much as what we had back in if you look 2009 so if you're looking back through your code base you can look at oh what happened around this time frame that necessitated so many changes going on in the code base that's kind of an area to dig into and look at more thoroughly here's another one from p2 services where basically we see a bunch of spikes at different points and so those are also things that we could dig into github also can show you the complete history for individual files so if you go into a tree view for any commit hash you pick out a file that you want to look at and then there's a nice history button and I'll give you a nice graphical look at everything that has happened on that file in that repository and that can be you'll see the commit message and drive into context with the other hashes and see what's going on there I also think it's very helpful to look at project release history so we're using Rails, we use a number of gems we have our own project release history these are all things that we can draw from to basically kind of figure out what's happened in a project and figure out how those changes in our dependencies influences the direction of our project we also have git log and I use this to assemble the commit chart that I put up next but you basically have an incredibly rich tool to just both kind of like dig into your commits by committer you can get a range of commits after a certain date, before a certain date show me the last 100 commits show me commits that touch this file on the command line and it's all incredibly fast but the nice thing is that you can also do some data exports with this and so that's how I actually got to this here and that's you can basically format and export from git log which I'll show you here where you basically define the format that you want to use and then I do a little bit of a clean up after the fact and then I end up with commit lines that look like this as a pipe to the limited file and then I wrote a small Rails application to basically pull all that in and then I can throw that to a library called chart kick which actually spits out this graphic right here where you can actually see what the commits are over time and you can also do grouping by week, by month by thing and then depending on how you import that data you can do some other nifty slicing and dicing with it as well git itself is a very wonderful tool I wanted to get into this for this presentation but I couldn't quite get to it git is a time machine so I could go back to any point in time in these projects and basically start over from how those projects looked at a given point in time so if I want to go back to 2010 and see what those projects look like I could in theory spin those up and my thought there is if you can get a virtual machine using a version of Linux from back then or a Docker container with the right version of Linux you can get the right version of Ruby installed, the right version of Rails installed and then kind of set that code base up as it would have existed at that time and kind of be able to look at things and say oh okay I can put myself back in context for what happened back then that could be valuable to you if you want to say hey where did this happen why did we do it this way and then kind of the bigger piece I should say as big a piece of this for me was talking with folks who had been involved in this from the very beginning or over the life cycle of these applications both from a development standpoint and from a business standpoint they're not always going to be available you may not have access to every participant I got good responses from most of the people I contacted though and that was very helpful where you can talk to folks to get context for why certain business decisions were made why certain code is structured the way that it is that can all be very helpful I do find it very helpful to kind of approach this in a I'm curious manner not an accusatory fashion I tried to phrase my questions with kindness like hey can you tell me about what happened around this can you tell me about what it was like working on this project back when everything was brand new and some of the feedback that I got around this was pretty interesting and two pieces I got from one of our developers Tommy Stokes was that we all learn as we go and get better with time so don't look at code from six years ago as an evaluation of how a developer is now and then you can also look at the code and think that if this could have been implemented in a better fashion or it could have been tested better there's probably a really good reason why it wasn't at the time and that could be because of business requirements and deadlines it could be because of a developer's own experience it could just be general tactical limitation and so some lessons that are available with this and just a thing about lessons available often times we hear like lessons learned but I saw something recently where they said lessons available just because you weren't necessarily guaranteed to learn just that that lesson was available to you so here getting to MVP was critical because there was overwork and we needed better automation tooling and shifting gears to allow external access to the application is something that was kind of a big change for the teams and one that is still kind of rippling out and then the upgrade itself took a long time and that stalled innovation there's always more that we can do in terms of refactoring business logic particularly moving towards a single responsibility principle shared migrations and models have not served as well like for all of us trying to move things intelligently through the application that causes test breakage the apps are overly coupled for how they really could be but there's a joy in writing Rails versus other software development for Dan Benjamin when he started the project it was preferable to writing a hybrid PHP and Java solution and for one of our other developers Rails was the first moment she felt she enjoyed software development if we'd understood where we would be now in terms of allowing external access to the number of customers we would have how much stuff would be coming in we would have looked at structuring our data and our applications a bit differently at the API layer making the errors more clear and descriptive having good documentation not treating every response as a 200 okay regardless of what happens with it is a big one for us we have to keep that in some areas because that's what the expectation is but that's not one we want to keep and I think one of the most important things I'm left with is expect that my present and my prior colleagues did the best work that they could given their knowledge and understanding and circumstances at the time and so I approached this with love and kindness to try and really put myself in the shoes that they were in when this happened trying to understand this contextually and understanding that everybody learns as a project goes on, best practices evolve and new perspectives are going to come in and add distinctness as people come and go on a team but Rails has served us very well for the last 12 years and will continue to do so for these applications and it's been surprising that everything has been as stable as it has been we are changing the UX this is how it looked back when Dan started this project back in 2005 and we are recently kind of giving it a UX facelift and so soon it will look like this and this caused audible gasps when this was a demoed last week people were astonished that it was like oh wow it's changing finally and the other thing that was interesting and I've got just a little bit of time left so I need to wrap up but the business took a big risk Rails was not at 1.0 yet and so they had a business critical need and they took a leap of faith in going with Rails when it would have been very easy to say no so it's kind of with the mix of astonishment that it's gone on so long and it has more potential to fulfill so I have some thank yous to run through really quick but I got some invaluable research assistance with some interviews the software that I used to build this presentation was on Rails 5 and so on and I'm online at base 10 and these slides will be available at walscorp.us slash presentations probably sometime in the next week or so and you'll see stuff when the conference releases slides as well thank you so much I appreciate you all being here and enjoy the rest of your conference