 We can get started with our second talk today here. I'm welcoming Denise Pellucci to talk about old code bases and how to manage them. Thank you. Thank you very much. Thank you. I'd like to thank LCA for inviting me to speak and I'd like to thank all of you for coming to hear me speak. So I am Denise Pellucci from a project called Dream With Studios. Dream With is a fork of the LiveJournal.com code base. LiveJournal was started in 1998 and released to the public in 1999. Dream With, we forked from LiveJournal in 2008, released it to the public in 2009. And I know that here in a Linux conference that might not seem venerable enough to qualify as old, but remember we're talking about a web app here and web context was very different in 1999 than it is now. So what I'm going to be doing is I'm going to be talking about some of the horrible things that we've discovered and how we dealt with them. And hopefully I'm going to be able to teach you some lessons about how you can apply that to your own code base. I'd like to start with a little bit of a survey. How many people are here because you are working on a project that you would consider venerable? Okay, we've got some hands. I want to narrow that down a little bit more. And I like to do that by life milestones because it's fun. So how many people are working on a project that would be legally able to hold an account without parental permission in the United States, which is 13 years old. If your code is from 2002 or earlier, please raise your hand. Okay. How many people are working on a code base that is actually old enough to vote 18, 1997 or earlier? Next month, happy birthday. You're in New Zealand, so you can go out and drink a beer with your code now. Okay, how many people will be old enough to drink in the United States? 21, 1994 or older? Ooh, okay. Are you older than Linux? 24, 1991? Okay. I'm going to keep going until I narrow you down here. You know that saying never trust anyone over 30? Is your code base 1985 or earlier? Okay, so we finally found the top end here. That's good news, that's good news, because I don't know what to say to somebody who's like working on Send Mail. So let me tell you a little bit about Dreamwith. Fork from LiveJournal in 2008. At the time of the fork, it was Pearl56. How many people here are Pearl people or no Pearl at all? Okay, that's a pretty good percentage. Pearl56 in 2008 was old. We only ran under Apache 1.3, if you don't know web context. Apache 1.3, the Apache 1.3 series was released in 1998. It would not be officially set to end of life until 2010, but the Apache 2.0 series was released in 2002. So by 2008, pretty much everyone had moved on to the Apache 2.0 series except us. We used, as our templating language, a templating language called BML, which stood for either Brad Markup Language or better Markup Language. Brad Fitzpatrick the inventor, he could never quite decide which one he liked better. And he had created that in 1996 for his previous web project before LiveJournal. And he got it to a mature project by about 1998 and pretty much never touched it again. So that was what our templating language was. We were relying on a whole bunch of Pearl modules, usually very outdated. We required very old versions of them. You couldn't just pull them from C-Pan. You had to manually compile them. So we took a look at all of this mess and we couldn't blame anybody but ourselves because, for the most part, the people working on DreamWith in the early days had been working on LiveJournal previously. And we just looked at it and we sort of said, here be dragons. And we found a lot of dragons deep in the code base. Now, I could make this entire talk be nothing but weird things we found in our code base so you could laugh at us because we have to laugh or else we'd kind of go a little crazy. But what I'm gonna do is take you on a brief tour of the dragons we found in our code base and then give you some tips that hopefully you can apply to your own project in a different context for deciding what to do with the dragons that live down in your code base because everybody's got some. So what did we find when we went looking? We found things like comments that are totally unhelpful. Not only does this comment have like a one line, okay, yeah, this is hacky, it forces a text flush. You know, it's got this link to a read more, zilla.livejournal.org slash 906. There's only one small problem with that. That URL is six bug trackers and four owners ago. It does not exist anymore and it was never archived by the internet archive. You can't go to archive.org and put in that URL and hope to get more detail. So if you have a bug in this area of the code and you wanna know why that hacky thing is in there and it's hacky, you're pretty much out of luck. We found workarounds for outdated browsers. I'm not gonna read the entire text of this comment. It's not really all that important. All you need to know is that it starts out fixing a bug in Opera 8 and when you fix the bug in Opera 8, it then manifests a bug in IE6. And I mean, this is actually a really good comment right here. It explains exactly what the code is doing and why it's doing it. There's only one small problem. Current version of Opera is Opera 26. Current version of IE is IE11. The workarounds in this subsequent code are for browsers that are at least 10, possibly up to 15 years old. So how do we know if we need this hacky workaround anymore or not? We found really old HTML. I hear that. That's kind of the noise that I made. In our defense, this was an admin area that almost nobody looked at, except like 20 people who were working on the admin end of the web app. But I was one of those 20 people and I looked at this every day for 10 years. And if you don't have the web experience, all you really need to know looking at this is this is like baby's first HTML webpage from 1999 called. They want their tables for layout back. It's really bad. We found out of date special casing. When you sign up for our website, you have to give us your email address to create an account. It lets you confirm your email address. It lets you reset your password in the future if you ever should happen to forget or lose it. People mistype their email addresses. Then they can't get back into their accounts. This causes technical support issues and technical support questions. So we try to head that off at the pass. And somebody did a lot of work to think really hard about all these possible misspellings. It was kind of fun really to brainstorm all the misspellings that we had seen users actually type. There's only one small problem. This isn't the beginning of a find mistyped email address routine. This is the entirety of the check. Hotmail, AOL. 80 to 90% of our signups are from gmail.com now. We don't actually have the special casing and the typo check for Gmail yet. We do have a bug open for that. That's like, next month. And finally, old bad decisions that were right at the time. This needs a little bit of technical explanation. How many of the people here who have worked in Perl were doing Perl in the late 90s, early 2000s? Okay, you probably remember that Perl's early handling of Unicode and UTF-8 was a little spotty. They introduced UTF-8 handling and it tried to do the right thing and often failed very horribly. So at the time, LiveJournal was an international website. We had users from all over. People wanted to write their journal in their own native language and communicate with their friends in their native language. We needed UTF-8 support and we needed it very quickly. We looked at the state of things. We realized that we could not wait for Perl to fix the various UTF-8 handling bugs. And so what we did was we told Perl to stop even trying. We would strip the UTF-8 flag on all of the text that was coming into us and we would do our own UTF-8 processing in code and essentially hack in UTF-8 support before Perl got UTF-8 support natively. And it was absolutely the right decision at the time because it gave us Unicode support approximately two and a half years before Perl could have done it. That two and a half years allowed us to serve those international customers much, much faster to the point where even today, live journal is the free press in Russia, essentially. So it was the right decision to make at the time but unfortunately it tied us to that method of doing it from then on because what we were doing to the UTF-8 flag supplied was a destructive operation. It was a one-way operation. You could not reverse it. So now it's 2015, Perl's UTF-8 handling is very, very good now. We're still stuck with our half-assed implementation because we needed it faster than Perl could give it to us. We actually thought about undoing this back in 2008 when we forked because we figured we would have a fresh dataset. We could start over, we could do it right this time but then we realized that we wanted to allow users to import their data from live journal to Dreamwith and not have to copy over their blog and their comments manually and that if we wanted to be able to do that, we had to continue to be wedded to this bad decision from 2000. We found a lot more and as I said, I can keep going for the entire session. Outdated modules. Perl has an absolutely wonderful module library with C-Pan. No matter what you want to do in Perl, chances are pretty good that somebody has already done it and has already put it on C-Pan. It's wonderful. The only problem was when you were installing the code, you could not just go to C-Pan and check out the module and use the most recent version because in the most case, it would die horribly because the code was relying on an earlier version of the Perl module. Ancient JavaScript. Roll your own. In 1999, there was no jQuery. In 1999, there was no shared libraries. The closest thing you would get to that would be websites where people would say, oh, I needed to get this effect in JavaScript. Here's how I did it. Go ahead and use this code. Which is great. But it doesn't really maintain well. So you now have JavaScript libraries that are very well maintained, very well bug-fixed, have a lot of eyeballs on them, so bugs are shallow. Except we have all of the ancient JavaScript that we built on our own to do the same thing. And if we want the advantages of those shared libraries like jQuery, we have to rewrite to use them. We found massive, massive duplication. Code duplication. Six different, slightly different ways of doing particular methods or functions. And you look at it and you're like, which one am I supposed to use? Because of course, they're imperfectly commented. Nothing is ever perfectly commented. But there's also duplication on the front end as well. For instance, we have three separate ways to search the site. Each search touches slightly different areas of the code base and of the user data. And they don't talk to each other. So if you are interested in the cheesy MTV Teen Wolf television show and you want to find other people who list Teen Wolf on their profile as an interest, you go over here and use this search. If the new episode of Teen Wolf just aired and you want to search around and see other people who are writing episode reactions and talk to other people about, oh my God, did you see that he did the thing, et cetera, you have to go over to our full text search, which is over here. They don't talk. That sort of duplication happens in our admin areas as well when users take actions on their account like deleting entries or logging in, creating a new login session or a bunch of other different actions. They get logged into our admin data table, except we have two separate action logging systems. They only overlap by about 60%. So you can find out some things over here and some things over there. And when you need to investigate a customer service problem or a terms of service issue, you need the data out of both of them. So you have to look in both places. All kinds of duplication everywhere. And finally, we found that we had bugs that had turned into features because they had been there so long nobody knew they were bugs. I have a lot of funny stories about this. If you want, catch me later and I'll tell you the one about the icons because it's really funny. But I will tell you this story instead. About a year and a half ago, we decided we were going to rewrite the front end for how you leave a comment on a journal in order to make it more modern, prettier, cleaner, faster, et cetera. We did the rewrite. We announced the code push. We sent the code live. We made our news post and we said our usual, this code is now live. If you notice any bugs, please let us know and we will fix them. About 10 minutes later, people started reporting as a bug. Links aren't auto-linking in comments anymore. We do a bunch of text processing on user-submitted data, obviously as anybody does. One of the things we do is if people paste in a bare URL, we will auto-link it. But only if you're logged into your account when you make the comment. We don't auto-link URLs from anonymous comments. It's a spam fighting measure. So I turn around and I say, well, okay, I just tested it. I pasted a link in my comment and it linked just fine. So why don't you point me at the comment that you're seeing and I'll look at it and we'll investigate further. And they pointed me to an anonymous comment and the link, sure enough, was not auto-linked. And I said, well, links don't auto-link in anonymous comments. It's a spam fighting measure. And they said, but it was auto-linking yesterday. And I said, no. Links have never auto-linked in anonymous comments. And I had made a lot of anonymous comments on the site myself. I participate in a couple of anonymous discussion journals. I had made plenty of anonymous comments with links, none of them auto-linked. I didn't know what these people were talking about. It eventually turned out that one pathway for displaying comments, a fairly hidden pathway that took about four clicks to get to, was not applying the anonymous comment rules to anonymous comments. It was treating everything like a logged in comment. So if you took these four clicks to get to this one view, then links would auto-link in anonymous comments. And people on these anonymous discussion communities would pass around instructions on how to get to this obscure comment view. When they saw a link in an anonymous comment, they would go click, click, click, click, click, click so that they could get to the view so that the link was clickifiable and they didn't have to copy and paste it. That bug had been there for, I think, 11 years at that point. And people had come to rely on it. They were pretty annoyed at us for fixing it, too. So, you know, I can keep going on stories like this for a very long time. And if you catch me later, I probably will because I like to share my pain. But you're probably here because you have some code that's getting up there in age. We took the show of hands earlier, so let's say you're looking at your code and you think, well, okay, should I rewrite this? I can't give you the algorithm for deciding yes or no. We don't have an exact algorithm ourselves. There are a lot of considerations. There are a lot of pros and cons. You have to think about it very carefully. What I can do is I can give you some of the things that go into our decisions and hopefully that will help you apply them to your own code. First thing I'm gonna do, I'm gonna walk you through two of our upgrade decisions and show you the things that went into our thought method. You can see how we do this. The first one was when we were starting out, the question, should we upgrade Apache? Should we go from Apache 1.3 to the Apache 2.0 series? And we had to look at our costs and our benefits. You know, obviously the benefit, you know, you're moving away from software that's at the end of life. You get actual security fixes, fancy that. You don't have to immediately downgrade new installs. You know, when you're dealing with a web application, people wanna come and contribute on your code. The first thing they have to do is install the code to get a working development environment. At that point, when somebody came to us and said that they wanted to come and contribute to our code, the first thing we'd have to tell them was, okay, great, love to have you, can't wait to get you started. The first thing you have to do is make your development machine more insecure. That was really hard to explain to people. And you know, you're standing there just going, I know, I know, I'm so sorry, I feel awful. It's just, it's a thing, really. So yeah, upgrading to Apache 2.0 much, much easier to explain. And of course, you know, it would mean that we were no longer horribly ashamed of ourselves and could actually hold our heads up in polite society because by that point, it was getting a little bit ridiculous. So you can't just look at the benefits. You have to look at the downsides too. The cost of doing that Apache upgrade, it was a lot of effort. Live Journal was one of the largest projects running on Perl at the time. Live Journal had done significant performance improvements because Live Journal was running on a very small budget. And so Brad Fitzpatrick, the creator, learned how to make every line of code squeak, he tuned it so far. And a lot of those tunings were dependent on a lot of the Apache internals that were very specific to the 1.3 series. The first time we tried just, well, we thought, okay, well, let's just go and try to run it on 2.0 and see how bad it is. It was really bad. It took six months just to get it to spin up under 2.0. That was not six months of full-time work. It was nights and weekends work, but six months is a very, very long time to be working on one project. It introduced all kinds of new and exciting bugs. Anytime that you do a rewrite, you're going to introduce all kinds of new and exciting bugs and it's something you have to plan for. But as my last bullet point says here, it was a no-brainer. It was literally the first thing we decided to do to the code once we forked it. And as a matter of fact, Live Journal was so grateful that as soon as we checked in the code, I believe they pulled it upstream within three weeks. Which, yay, open source. Let's talk a little bit about something that was a little less cut and dried. Recently, we have begun to switch our front-end framework to the foundation framework. If you don't know about foundation, it's really, really nice framework. So the benefits we get out of that, it's a modern framework, it's underactive development. It gives us much better responsive design. That's what foundation is designed to do. It is designed to make it very, very easy to do responsive design that looks similar on desktop and mobile and functions well on both of them. And that's a very hard task. It gives you better cross-browser capability. As we saw on Wednesday with the browser bugs session, if anybody was in that one, browsers have lots of bugs and you're going to trip over them. And with the foundation framework, people are working on actively working around those browser bugs, it's more than just you. Foundation is much easier to make accessible for people with disabilities and people using assistive technology. That is actually one of our core principles on DreamWidth. It's something we consider to be pretty much our primary task. If someone says, this doesn't work with my assistive technology, that is a showstopper bug for us. So foundation getting us better accessibility is a major win. It's very well-documented. Previously we had BML, the old crufty template language from 1996. It wasn't documented anywhere. Even the documentation that came with the initial release was very, very, very, very high-level in cursory. So when new people came in, the first thing we had to do was teach them this project-specific language that was 20 years old and not used anywhere else. That was a bad value proposition. So by switching to foundation, we make it easier for people to come and develop with us. And finally, we get the satisfaction of killing that lurking old HTML with the tables and the all caps tags and et cetera. The switch to foundation does have a lot more cost, though. Switching to foundation involves redoing every single one of our pages because it's old enough that we have to redo everything from scratch. Foundation is good at approximating your existing layouts, but not perfect, and you cannot always do an exact reproduction. Excuse me. So no matter how closely you try to imitate the old design, you are going to have things looking different. Our users are very picky. They will notice, and I am not exaggerating here, a five-pixel movement on the screen after a code push. We will have situations where we want to increase the padding around a particular item in order to make more white space on the page, and we'll move it. We will get at least one bug report after that code push, saying that this thing has too much white space now. So in order to make changes to the look or feel of the site, we pretty much have to do a lot of advanced notice and advanced preparation for our users so they don't freak out when things look different. New and exciting bugs, which is, again, always a downside when you change things. It results in a design that looks slightly less individual, and this is another one of those trade-offs. A lot of sites out there are using Foundation. It's a very common framework. If you ever look at a web page and say, wow, this looks like Web 2.0, chances are pretty good. You're thinking that because it is using a common framework, and Foundation is very frequently that framework. It is possible to make designs that don't look like everything else in Foundation, but Foundation makes it easy to look like every other site out there. So we had to consider, is it worth losing that sort of uniqueness and individuality there? But of course, really the main cost is all of the work. Our full-time employee, Athena, started on this nine months ago. She's about half done. So when you are looking at your own code base, you're browsing through your code and you say, you see something and you say, wow, that's really ugly, and I want to fix that and bring it into this century. How do you make these evaluations? So here are some of the things that you should keep in mind while making those evaluations. Things that weigh in the pro rewrite column. Not necessarily absolutely reasons to do it, but things to consider as evidence for a rewrite being a good idea. Will you become more compliant or more compatible with existing standards or best practices? I don't know what your standards and best practices are. You know what the standards are for your niche. Every particular niche has them in web development. Web standards are very easy to find in server development or desktop development. You might not find capitalist standards for everything that you want to do other than the RFCs and stuff, but there's best practices and you know them. You also know where you're not meeting them. It's one of those things that kind of sits there in the back of your head like, yeah, I know I'm not doing this. I really should be doing this. If rewriting something will get you closer to that, that's a mark in the plus column. Will it make your code easier to install? Now, we have had in the last 10 years some amazing, amazing work being done in the field of installation. It's a lot easier to install things under Linux now. It's a lot easier to install packages. You no longer have to be sitting there staring and editing the make file and trying to compile unless you like doing that, in which case more power to you. Some great advancements have been done in that field. And if you are in the audience and you are one of those people who have been working on making installations easier, thank you very much. If your code started before all of that effort, it might not actually work very well with those new installation methods. So if your install docs contain this long list of things you have to do in order to install this code or in order to make this compile or what have you, will rewriting this make it easier? That's another check in the plus column. Will it eliminate project-specific systems like how I was talking about BML? BML is an example of a project-specific system. It's not the core focus of the project, but it's a system that was invented in order to make the core focus of the project run more smoothly. There are a lot of times when you need a project-specific system because nobody else is doing the thing that you want to do and doing it well. So you go and you write the thing. Sometimes you go and write the thing and then it becomes the widely adopted solution. Sometimes you write the thing and you wait two years and somebody else writes the thing that's almost exactly the same thing except their version takes off and yours didn't. And then when people come in, they have to go and they have to learn your particular project-specific system when they're used to the thing that everybody else is using. So if you have those things in your code base or in your workflow, not necessarily in your main code base, but in your general workflow, consider getting rid of them. If a rewrite will move you away from things that are not widely adopted and that are not your core focus, consider that again a vote in favor of going ahead and rewriting. Will you reduce your reliance on institutional memory? Literally the only reason we could make Dreamwith work was because my co-founder Mark and I had both worked for LiveJournal for so long that we knew all of the things that had been done and we knew all the reasons for it. We've been trying to document those things, both technical, social, everything, for five years and there are still times when people come to us and say, hey, I don't remember why we do this particular way. Can you explain it to me? Institutional memory is a great thing to have, but it's a bad thing to rely on because people get fed up, people get bored, people leave. So if you can rewrite things so that it's no longer needs you to go and seek out the person who's been working on the code for 20 years in order to understand it, in order to explain it to you, that's another mark in favor of rewriting. And finally, is it going to benefit your users or your team? Is it going to get you a feature that your users want? Is it going to get you closer to doing a feature that your users want? Is it going to make working on the code more pleasant? Every code base has hotspots, things that people just hate touching. And if you can go into that system or section that people just hate touching and make it slightly less painful to work on for your team, again, that's an advantage and that's a reason to go ahead and rewrite. The cons, reasons to not jump in and necessarily do a rewrite. Are you going to lose a long history of bug fixes or security fixes? This is the major reason why we actually forked LiveJournal instead of starting our own project from scratch. People asked us that. People said, why are you tying yourself to this 10-year-old code base? 10 years in web application is, you know, like 50 years and everything else. Why don't you just take what you've learned, start over and, you know, do the second system. And that leads to, of course, second system syndrome, which I'm pretty sure everybody's heard of. And it also loses you that rich history of, you know, security fixes for that one time that somebody found a really nasty way to do some XSS injection by doing this, this, this, and this. So, okay, you fix that security bug. Well, if you rewrite, you're probably going to introduce something like that again. You're going to introduce a new security bug. You're going to introduce, you know, browser display bugs. You're going to lose all that accumulated foundation of your code. Will you tie up too many of your people? Will it involve too much retraining? This is kind of the question of, is it going to take too much time? But too much time is subjective. Will it tie up all of your developers for six months? You might decide to treat it differently than if it'll just take your developers six weeks or six days. Is it going to involve you having to retrain all of your contributors on the new system? You know, we switched our bug tracking system from bugzilla to GitHub issues. It involved retraining all of our contributors, many of whom don't contribute to other open source projects, many of whom are programming for the very first time with us. So we had to retrain them on how to use the new issue tracker. That's... It's not a reason not to do it, but it's something to consider. Will you be tied to something that might not last? We all see this happen. We come to these conferences, we go to this talk on Fubar and you come out of it thinking, that's great. I love this. This is the best thing ever. I'm going to go and rewrite everything to use Fubar right now. And then two months, two years later, you come to the next conference and you're sitting here and you go into a talk on Barbaz. And it's a little like Fubar, but not really, it's in the same area and you go, oh, wow, this is doing things much better than Fubar did. And it scratches a lot of the itches that I had with Fubar. Now I want to go and rewrite everything to use Barbaz instead. You can get on a treadmill like that with new technology. Obviously we are never going to perfectly predict what gets widespread adoption and what things stick and what things are fads. If we could predict that, I would probably be making a lot of money in Las Vegas. But consider that, think about what you will do if what you're deciding to use doesn't stick and try to avoid making irrevocable decisions like dropping all your UTF-8 flags out of pearl. Is it going to require you to fork or adapt a standard or a module or something? If you want to implement something, use a particular module that is available, but it doesn't do exactly what you need it to do and you're tempted to fork it and use your own version because you know that the upstream maintainer is never going to put in that one feature that you need it for. And then you're stuck relying on maintaining a local copy of this and you have to keep them from diverging and it's a pain in the neck. So if you're tempted to do that, that's usually a warning sign that you might want to reconsider rewriting to use that particular thing. And finally, is it not going to benefit your users or your team? I showed you way back at the very beginning that HTML page with the 1999 Baby's First HTML. That is an admin page on the site, fewer than 20 people ever see it. Every time I looked at that page, I cringed because I was one of those 20 people and I spent a lot of time there every day. But really rewriting it didn't get anything for the users and it didn't really benefit anything for the whole team because it wasn't a workflow problem, it was just ugly. It only took me five hours to rewrite it, so I did it because it was really annoying me every time I looked at it. But if that had taken me five weeks instead or if it would have had these complex dependencies or whatever, I would have just lived with it, it's not worth it. So these are all things to consider. The number one question to consider in deciding whether or not to rewrite things, however, the absolute number one question, so important that it gets its own slide is, are you going to finish it? The XKCD cartoon for those who can't see it. Situation, there are 14 competing standards. 14 ridiculous. We need to develop the one universal standard that covers everyone's use cases. Yeah! And soon, there are 15 competing standards. We've all seen this happen with standards. This also happens with code. You look at your code base and you say, there are three different ways to search. We should fix this. We'll write the one that handles absolutely everything. And soon, there are four different ways to search. And that's how we have each of these duplications going on. There's something that almost gets there. But you want to just rewrite it and start there. So avoid that if you possibly can, because it's ugly. So, okay, I've given you a lot of warning signs about how to handle if your code base is already old. Well, what if you're starting out now with a new project and you want to avoid your future self looking back and going, God damn it, me, of 15 years ago you were really, really not on the ball there. Here are some very quick things that you can do to help future proof your code. Comment absolutely everything in the code itself. Do not use external references to your Wiki or your Bug Tracker. Disappear, as we saw. It's okay to leave those links for further reading, but do try to summarize the actual change or reason for the change in the comment, not just drop a link. If you find yourself wanting to explain why you chose to implement it this way, don't just say, see the Wiki page on this decision. Put it right there in the comment. Speaking of comments, make them grep friendly. Common technique in code is using your Fix Me flag, whatever your project's particular flag is. Standardize those. When we went looking through our code, we discovered that because of the various strata of people who had been working on the code over the years, we had something like six different Fix Me flags. We had Fix Me, we had To Do, we had XXX, we had Read Me, we had Later. Pick one and stick with it. Excuse me. If you have to make a workaround for a specific browser or operating system or a particular version of a module or what have you, put in a note, something like workaround or whatever, and add in what the workaround is for. If you're working around a bug in Windows 7, put in, this is to work around such and such a bug in Windows 7. Then when Windows 7, when you drop that out of your support policy, you can go to the code and search for Windows 7 and find all of the hacks that you've done, and you can evaluate whether you need to go ahead and change those. Write task lists for your future self. We all do it. Software engineering is a series of compromises. You compromise between what is available now and what might be available in the future. Sometimes we have flame wars of what we should do about this particular issue and one-side wins, but in order to win, they have to agree that we'll revisit this in another two to three years. Well, go put that reminder on your calendar that in two years, we were going to revisit thus and such. Put it on your calendar when it rolls around on your calendar. Actually do it. Don't be tempted to blow it off because you're going to, but go ahead and actually do it. Finally, if this is applicable for your particular code, regularly build it, install it, compile it, et cetera, from scratch on a clean system with current versions of all your libraries, all your modules, et cetera, et cetera. When you're working on code yourself, you're developing on your project, you know you've already got things set up properly. You know you already have all the required modules. You know you already have all the libraries that you need. You're not going to spot those problems. Go ahead and do it as though you are completely fresh and you'll spot things a lot faster when there's still minor problems before they become major problems. Thank you very much. Those are the things that I have learned and I hope you avoid having to learn the hard way or that if you already have learned the hard way, find me later. I will take questions very briefly. I would like to start with actual questions and then if we have time after that then we can go on to observations and other war stories. We do have a couple of minutes so try to be brief. So one thing you didn't mention was testing. Is there any sort of test suite or are you just flying blind? There are some tests. We can't really run them because when we run them it corrupts the testing database. It's one of those things that we keep meaning to fix. We haven't yet. There's kind of a project open for it. People keep wandering off from it because they come in they take one look at it and they go oh my god. And then they just sort of back away slowly. Here let me lay down an offering of chocolate to the monster and creep away while it's distracted and eating the chocolate. They do exist but they're in such a state that they're almost worse than having no tests at all. Questions, questions? So in the earlier days prior to the fork how effectively did live journal use version control and similar tools? How effectively did live journal use version control similar tools? Very effectively. Excuse me. Version control was handled very well. I believe at the time that we forked they were using subversion. Prior to that they had been using I think they were using actually CVS. It was kind of a history of versioning control systems over time and that's actually a good example of the other thing of should you make the switch? We went from when we forked we moved from subversion to mercurial that was state of the art at the time and then shortly thereafter we went from mercurial to get. And changing version control systems is another one of those big changes that requires lots of retraining. Particularly with Git a lot of our contributors are very new to open source. I did the statistics once. 75% of people who have contributed a patch to our code base have never contributed to open source before. 60% of them have never programmed anything before. We were their first programming experience. Also another statistic that I like to brag about 65% of our contributor base identify as women. We are the entry point for a lot of people and Git has a reputation for being very difficult to wrap your brain around. I have trouble with Git. I keep getting my Git branches into such a state that I look at it and I go how did I do that? And then I go to Mark and I say Mark how did I do that? And he said I don't know. So you know that's a cost. That's something that we have to spend a lot of time helping people clean up from Git disasters. I say we it's not me. I'm usually the one whose disaster people are cleaning up after. And that was a consideration we had to take in mind but Git is powerful enough that we decided that the benefits we got from it were worth the downsides. Thank you very much. We are running out of time. So if you have any other questions or tales from the trenches then please discuss them later. Thank you very much.