 Are you guys ready? So today we have, well, this session we have Dirk, and he's going to talk about, I suspect, the open source project that he maintains, called subsurface, possibly? Possibly, possibly not. I think I'm supposed to say g'day, right? Anyway, good afternoon. One of the things that I've always done whenever I'm in a public building, I always look for the fire exits. It's, you know, one of those weird things people do. And I love the signage for the fire exit, because it says, in case of a fire, go to, and then there is the field where to go to, and it says, the nearest fire exit. And below it, it says, if that doesn't work, where should I go? Well, the second nearest fire exit. That is maybe the most useless fire direction sign I've ever seen. Congratulations, University of Auckland. My name is Dirk Hoondl. I'm Interest Chief Linux and open source technologist. What I'm talking about today, conceptually, is certainly something that is related to my day job at Inter. But the project that I'm going to use as an example is not. It's a hobby project that I'm doing on the side. I've been doing for three years. It's called subsurface. And for those of you who don't know subsurface, it's a dive log program, which might explain some of the background images on my slides. So sustaining momentum or the gap between user requests and developer capacity is one of those projects that, one of those problems that most people who start an open source project never think about. Because when you start an open source project, well, most people don't realize that they are starting an open source project. And once they do, they're really excited and it's new and it's great and it's awesome and it's all easy. And then this happens. I initially had planned to have a few of those graphs and then decided that that would be seen as project bashing. And so I removed the name of the guilty. I removed the name of the project. This is not my project. But if you look at what used to be called Olo and is now called open hub for. Look at hub projects, you find a ton of projects who have graphs like this or far worse. Which means someone started a project, there was some activity and then there's two years of very little activity and then it peters out. And that's very, very common for open source projects. And in some cases it could be because the project is perfect. Proc Mail is an example like that. Proc Mail has received no patch in more than five years and we all know that Proc Mail is perfect software. The author has said so himself. But most projects that have a graph like this, it's usually an indication that either the developers have moved on or that the developers have given up. Both of which are very, very frustrating if this is a project that you decided to either use or if it's a library or a toolkit or something to rely on. What I wanna talk about today is some of the reasons why this happens and some of the things that you can do to hopefully not have that happen. So what is the life cycle of a typical project? Can I turn this without breaking things? Oh, good. So as I said, you start. You're very enthusiastic, rapid development happens depending on your talent level. Within four days you have a working resource control system or alternative six months later you have something that compiles somewhere in between. And one of those initial first hurdles is what I call being marginally useful. Because if you want to get other developers involved unless you are extremely convincing or have the ability to order people to work on something with a corporate relationship or something, usually you have to have something that is at least somewhat useful before anybody else will look at it. So this is kind of all these very first hurdles that you need to get to. In the example here, Linus is the one who started Subsurface. He is admittedly a somewhat talented developer and had something marginally useful in under a week. I hate the guy. And he doesn't even show up for this talk. But marginally useful is usually only enough to get a few other people who by nature are developers involved. And I'll talk about this interesting, so who are the people who are involved and what does it mean on the next slide? But so you go from marginally useful to a first release. And I can list a few projects who missed the ordering, who do a release before they're marginally useful. It's a judgment call. But the reason why I point out the first release is there are so many projects who've been out in the wild for a little while and their 1.0 release is not always very prompt. Who of you remembers when a network manager had their 1.0 release? Pardon me? Couple of months back. Couple of months back, yes. And how old is network manager? Yeah, 14, yeah, yeah. So some project take a little while to get to 1.0. The Linux kernel famously took three and a half years, not two and a half years. I can't do math. Two and a half years, March 94. Linus jokingly said with Subsurface he was gonna be much more aggressive. So Subsurface 1.0 came out pretty quickly. At which point it was already quite useful. Already had, because of his fame, I guess, a lot of users. But for a lot of projects, for a lot of projects that are started by less famous people, a lot of projects that are started just to scratch an itch. Someone has a problem, a little thing they want in new desktop environment, for example, or the 55th Excel incompatible spreadsheet program for Linux because we already have 54. Very often the first release is something that is the first time other people pay attention and maybe other people get involved. And then there is this other milestone that I have listed in the life cycle. And that works for me. And that's a very critical life cycle milestone because very often, at least some of the initially developers at that stage will go away. Because we all, well, the vast majority of us have ADD, attention deficit disorder. Ooh, there's something shiny over there. I could do this. And how many of you are developers? How many of you work on only one project? Wow, we have a person in the room who actually can focus. It is awesome, it is awesome. Alternatively, it could be very lazy. I'll go with you, can focus. So what that means is that works for me is very often the stage where not only maybe the main developer, but a lot of the developers working on a project kind of lose interest. Because let's face it, this is all about me, me, me, me, me, right? We are not doing open-source software because we are heroes and we are into charitable things and we are thinking about the other people. It's all about us. Anybody who claims differently is either different from me or a great liar or both. Now, having said that, a lot of time you will get to this point where it works for you and you still decide to continue to work on something. Linus famously said that Linus has been doing what he needs a decade and a half ago at least, but I guess it's more like two decades ago. So something then motivates people to continue to do this. And I often suspect one of the reasons is that you like pain because once you add users to the mix, most open-source projects turn into just that. I don't want to sound that negative, but there is an interesting observation here. So when you start with a project, you by yourself or you and your two best friends or whatever the group is. And oh, by the way, I should have said this in the beginning. All the projects or everything that I'm saying here is with the mindset of an open-source project that an individual starts. This is not inter-corporation deciding we're going to do this new open-source blah because inter-corporation has the very advantageous ability to say, here are 120 developers who will work on this. Most of us don't have that. Who of you has 120 people working for them? I thought so, yeah. So usually it starts with the developers as the same group as the users because most people start with something because it scratches and itch that they have. Subsurface, at the time it was started, there was no good dive-logging software that would run on the Linux. And that's why it got started. And so Linus and I were the first two developers and we are also the first two users. And then in the next phase, what usually happens, once you have marginally useful two first releases somewhere there, you have a phase where the number of developers that you have is in some way shape or form proportional to your user base. So you may have 10 developers and 100 users and then it's 15 developers and 150 users. And this balance is very healthy because as you get more users and as they have ideas of what should work, what bugs, as they have new requests, you also grow your developer base and often what happens, some of the users turn into developers and it's a very easy relationship. Well, that's not overstated. It's a manageable relationship. And then something really bad happens. You have success. When you have success and people start using your software on us, you suddenly are in this last scenario where the number of users is really big compared to the number of developers. And that's this moment where the topic of this talk sets in. How do you maintain momentum? So I spent the last two hours trying to get the latest version of Subsurface to build on Ubuntu 12.0 or the second to last long-term support version of Ubuntu, which of course doesn't support Qt5, doesn't have the correct libraries that I need for XSLT processing, doesn't even include Libdive computer, is missing all the pieces that I need to build my software. Why would I support that? Well, on the scuba board, there's this very nice guy who's super enthusiastic, wants to use it, and that's what he uses. Answer number one, we'll get the different OS. You can tell that to a developer. Telling this to a user means you just decremented your number of users, most likely. And the more users see you say this in public, the fewer people will be interested in engaging with you, which may or may not be a problem. I personally try to be nice to people, which is also a difference between me and Linus as he established this morning. And I was going to say this with him in the room, so... The second thing, and then that's what Qt said to me earlier, is, well, just tell him to build it himself. Yeah, it's kind of... Most people who want to use dive lock software are likely scuba divers. Not sure how many of you are scuba divers? We have half scuba divers. So you're a scuba swimmer? And no of... Pardon me? Really? Oh, yeah, that's... Yes, I was going to say that. This is one of the wonderful things why I love to come to LCA, because it's the middle of winter at home, and it's nice here, and I'll be flying to Fiji tomorrow to do some testing of the software. It's just... It's selfless sacrifice to be able to test the software I'm working on. Yeah, and if you believe that, I have a bridge I can sell you to. My wife is not confused about this, nor is my management team, so no, my day job. Anyway, so where was I going with this? Yes, scuba divers. Scuba divers, in general, are not necessarily the most computer-affined people. This sounds weird, but it's all those things when you talk to scuba divers, and then whenever I go diving, I have several dive computers, and people look at you and go, what are you doing? I'm like, okay, all right, dive looks off. It runs on a computer. And the number of other divers on the dive boat, is blank when you say that. Computer. Oh, that's one of those things my wife does Facebook on. So no, telling a scuba diver that he should build this himself or herself on Ubuntu 12.24 is not going to work. And this is one of those challenges. So when Linus started, this was, of course, running on whatever was the Fedora version that he had on his laptop. And then we got it to run two more flavors of Linux, and then I ported it to Windows and it supports Mac. And of course, who was in my talk last year? No one. Excellent. Last year I talked about migrating the whole project from GTK to QT, and so we have better Windows and Mac support. And so what happens is that this surface of supported environments goes up with your user base. And at the same time, the surface of supported hardware configuration goes up. And the surface of supported dive computers and for this software supports not just the hardware it runs on, but it also needs to support the other end, the dive computer from which you extract the data. And so as your user base grows, the amount of work that is needed to just stand still goes up dramatically. And so right now when I do a new build, I do builds for Windows 32 or Windows 64. On Mac, I have a single build, finally, that works on both. We do builds for Fedora 20, 21, OpenSUSE 13, 1, 13, 2, Tumbleweed, two versions of Ubuntu, one version of Debian, a version of Linux Mint. I'm missing a few. There's two architectures. I think it's currently 25 different platforms that we support. So just to stand still, just to do a little thing and make sure it works everywhere. It's a lot of work. And then of course comes the next version of Operating System X. So you go from Windows 8 to obviously Windows 10. And you Fedora 22, no, that's it. No one laughed at that one. Wow. So what happens when your user base grows is that the cost of just standing still goes up. But at the same time a lot more people means a lot more ideas how this should work. So two years ago when we were in the very early stages of the program we had a few hundred people using it. The users and developers were like one to five in numbers five to one I guess. And the number of feature requests, every single feature request half the developer says, yes I know it's on my roadmap, I just haven't had time yet. Today the vast number of feature requests we go, huh why would you want that? Or that's interesting. I have no idea how to do that. So the more users you get the more challenging it gets to actually implement what they want. The more complex it gets the more fragile your software gets. And in so many ways what you want to say is oh that's a good problem to have because the alternative is you have no users and they're who cares. But it is it is one of those interesting winning means your life just got harder challenges. And at some point in this process and most likely without you ever noticing it you suddenly moved from doing a project to doing a product and that is a massive, massive difference. In a project what you get is a lot of works for me. Yeah it just works. It compiles in my system what's your problem, right? It supports my dive computer. You have a different one? Too bad. In a project you experience a lot of 80-20 and oh by the way in a volunteer based project where your developers are volunteers you will find a lot of people who are super excited about a hard problem and they solve 80% of the problem and they're done. And if you want the problem actually solved the final 20% that's you the maintainer. And I will say this very honestly I have several developers on the subsurface team that I really love that are wonderful people and whenever they say oh I'm working on this feature in my mind I'm planning time on my calendar and I can finish that. Because I know whatever I get will be kind of sort of mostly what they had in mind and I'm sure it works for them. But it doesn't build on half of the platforms the UI is like it was developed by me and I'll talk about UI's I hope it's somewhere it should be on there. And it breaks and it yeah. So you always have to keep in mind unless you have the luxury of a QA team and a lot of manpower behind you. Being Linux is nice because there are so many companies who do all the testing for Linux and a lot of the kernel developers will jump in and do the final 20%. But in most other projects that is what at the end of the day the maintainer has to do. And then of course the other thing most of you said you are your developers who of you is good at writing documentation? Brave man, okay. Oh, half. Who of you likes writing documentation? Oh, wow. No? That's sad. Come on. But it's fine. But there's a big activation energy. Yeah. So for me the activation energy is near infinity. I absolutely hate writing documentation. And the documentation that I write is usually completely incomprehensible because it assumes that you know so much about how all this works that literally when the very first version of the man page for subsurface is like one paragraph, like everything else is obvious. Thankfully we have in the team with a wonderful gentleman from South Africa who is just like you who loves writing documentation he's really good at it and very patiently keeps up with the work that the developers do. The other thing localization, which is a massive pain to set up the infrastructure but then before every release you go back out to your we have 23 languages I think at this point you go out to all your translators and say do you have some time because I would like to do a release and then of course your developers say oh I have this bug fix and it changes seven strings and every time, reliably every single time before every release someone finds something that requires new strings. I don't know why that is. Support when I became an open source developer I never thought that one of my biggest time things would be support. I'm a developer in a normal development organization support is somebody else's problem right? You do your own open source project guess whose problem it is? It's yours. I have a fairly active developer team we have 67 just this year we have 20 people who very very steadily work on subsurface yet the number who reliably answer support requests yeah something like that one two three it's usually me and the other thing oh by the way users are really really stubborn as to where they want to post the support request so like every good piece of software we of course have a facebook page you have to have a facebook page the next people who sent me support requests on the facebook page oh send private messages to me on facebook for support request or there's there's a big scuba diver forum scuba board people send random posts in some group in scuba board with support requests every once in a while I search for subsurface on this forum and like oh there's a support request awesome we have a huge user forum link on our website that was too easy I guess infrastructure that's another one doing an open source project for free in your spare time is an expensive proposition because someone has to pay for the domain and for the server and for all these things and that tends to be me doing releases why do releases it's actually a good question because as you said you just have your users compile top of master and it's all awesome right so this whole thing of sustaining momentum I have learned that it is incredibly important to have reasonably regular and frequent releases the perception of users of when a project is abandoned strangely not strange enough is not based on the number of commits to get most of my users do not check our get history so I do it most of the users look when was the last release and so if you don't have a release at least every 3-4 months people say okay it's gone and so I quite aggressively start with very early betas for my releases because I can announce the betas most users are not touching that but the fact that it's out there shows them that there is continued development I have now started to do pretty much daily builds where whatever is in master assuming it builds now you should try and have master build just we automatically create the latest build for all of these 25 different platforms which comes back to infrastructure by the way because for quite a while that was a embarrassingly manual project for me until at some point I said this is so dumb I enter the same 6 commands on 5 different terminals and I finally built my own little CI system so by now everything is built with a single command except for Mac of course because that needs to be different but everything else is built on one system a single command and it triggers everything but in the problem of continuing an open source project and continuing to be successful continuing to grow your user base having regular releases and having releases that ideally let's be optimistic fix some of the bugs people report add some of the features people ask for and in general improve things an open source project that I used to be involved in X3D6 which was then forked into X.org X3D6 would do releases for years after it died and you do the diff between the last release and the current release and there are version number changed and three files had minor edits and that's it, that's the new release okay now you actually need to continue to work on it to continue to listen to your users and you need to continue to figure out how you can address what they send you the bugs, the feature request the platform request and what not and then that last one again something as an open source developer very few people think about marketing I guess it depends on whether you care if your project has users well if you're here talk with the title sustaining momentum I kind of assume that you care if the project has users but that's the reason why there is a Facebook page the Google Plus page and we have a user forum and a website and our own domain and all that because if you search for dive lock on Google and and if the person who started your software is not Linus Torvalds then 50 commercial projects will come up first and then hopefully your your little project will pop up we had this wonderful benefit that Linus has I don't know 2.5 million followers on Google Plus and so when he started talking about subsurface that really helped our search rankings quite dramatically so when you look for dive locks we now are fairly high in the search results but it is something to think about because how do users find you assuming they're not highly technical and are people who know how to find you who know about you how did they know about you target group are scuba divers who they talk to their dive shop they talk to other people on the dive boat how do they find you Facebook has actually been surprisingly good at bringing new people so you do all this you have created a team, you have developers you're building products you're documenting, you're supporting you're making releases you're doing even some marketing that still leaves you in the situation that almost always you have too few developers and too much work to do and I'm very proud how big the development team for subsurface is but I'll be honest if any one of you has spare cycles I have a great project for you to work on one of the hardest things for me is saying no so if people come and say I would like Ubuntu 12.4 should I invest the time what is the opportunity cost of me spending three days trying to get this to work or should I just say nope I have more important things to do it's incredibly hard and unfortunately I don't have a good answer on how to do it I can tell you you have to do it I say it more often than I like but there is absolutely no way that you're assuming you have users assuming you're not it's just you and your three buddies and you're all happy but if you have users that are not developers you will have to say no and some people are very comfortable saying no excellent at saying no for me it's hard because there is someone out there who has something that it's kind of a reasonable request he's on an old OS whatever Windows 32 bit binaries for Windows seriously but there are people out there who running Vista that's so funny to me we have like we have a system now where you can fill out a user survey and it detects which OS you're running on and when after a week or so I pulled the first statistics and I had 25 people on Vista I completely started giggling I understand the people on XP it makes perfect sense you have the old netbook nothing else once unless you have XP but Vista really anyway but yeah so you have to figure out where do I say yes where do I say no you have this issue with generational change in the current version so we have released 4.3 about a month ago we have 375 commits since of those 375 commits 4 are from Linus that's what I call generational change he wrote you know 90% of the first version and by now he's just hanging out with us we have other developers who came did some great work and then disappeared we have people who come in for one single release they have one thing they want to implement they implement it it works they got inclusiveness so in the last couple of years a lot of conferences talk about diversity and the fact that it's all white guys I'm a white guy in case you didn't notice I need to be accommodating to the blind people in my audience please be nice people so it is very very hard to in a small team to actively work on diversity it's very hard to say in a way that isn't sounding weird or offensive I'm looking for female developers that sounds really really stupid yet it's something that I do very very actively so whenever whenever people in the first commit I try to be super nice and to respond and to not say you know who threw up on that page no hey great patch thank you very much and oh by the way maybe you could read our coding style document because it has a few ideas that two spaces is not enough indentation things like that but try to be nice to people try to reach out to people try to be inclusive of anybody who comes whether they are phenomenal brilliant developers or whether they are people who are still learning I have two developers on my team who are super productive contributors and when they started on subsurface two years ago the code that they sent me made my eyes bleed but patience has totally worked out and we have really good contributors we still unfortunately have a single steady female developer which makes me very sad we have most of the continents covered we have Africans and Asians and Europeans and the token Americans and the South Americans and everything but if you're trying to find new developers besides being nice and having an interesting project and encouraging people to do there are a few initiatives that people do a few ideas that people have how to get new developers one is bug bounties those are fantastic if you're googling you can pay a million dollars other than that I have not seen them work because the amount of money that I could put up hey a hundred dollars if you find a bucket fix it hey that's a lot of money for me and B who really would work for that I mean that doesn't work pay for feature now that I have done I have paid a designer to re-design my own page I have paid other people to do a specific thing that I wanted I just paid this out of my own pocket open source projects don't tend to have a lot of money the last one is phenomenal you all saw Carol speak I think it was yesterday the Google Summer of Code has been awesome for us we participated last year for the first time I had four students two of them did what you expect they disappeared the day after the final money was paid but we have two new developers who have been super productive and students one from Egypt, one from Kenya and just great guys sadly no female student I will once again try this year to be more aggressive there so if you want to maintain your content if you want to be able to grow with your user base what you need to do is you need to start thinking about your project as a product you need to start thinking about your user base you need to start thinking about what is it that they want what is it that they need what can I put on the roadmap what do I have to say no to and how can I engage in a way that makes them want to contribute that makes them want to come back that makes them want to do more and that's the quick and easy summary I have like five minutes left no I can't count two minutes left questions yes I think for the video we want the microphone I'm not a diver my name is Birger and I want to ask about sponsors what about sponsors for your project or any project so I quite intentionally talked about what I consider the typical open source project the typical open source project has a very very hard time to find sponsors because unless you are really big or do something really special very few companies will support you so we get support from some of the dive computer manufacturers and that support usually is if you have a dive computer please support it mixed blessing we love it, we're very happy I have literally arms full of dive computers which is great but we don't have anybody who comes in and sends us developers or sends us money or anything like this we have great support from companies but not in a way that would solve the problem that I'm trying to describe here so I have a lot of contributors in the open source projects I'm maintaining meaning people who send e-pads or maybe two and then they're off any ideas on how to rope them in and get them more involved yes, so this is one of the statistics that I track quick follow up question maybe because I have to hand off the mic if you have worked on this for a long time on your own you have developed a sense of quality that you would like to maintain building trust to people to give them commit rights once you have rope them in so this is a statistics that I track how many people send me one patch or three patches and disappear and if they do it I always connect back with them so a month later they have disappeared I'll send them an email hey what's up, it was great to get this patch from you why are you gone and you'll get the people who say no no this was just the one thing there's the guy who sends these new desktop files the new standard that the GNOME people invented and he apparently sends a commit to every single open source project on the planet adding this one file, excellent or it's people who say it's just a bug that bothered me I just wanted to fix it and a lot of people say yeah you know no this is just too much work one of the things that I do is that I especially with new developers I try to be very responsive letting people hang for a long time is demotivating so I try to be very quick in responding and say hey thanks for the patch here are a few things that I would like to see differently I will intentionally lower my quality threshold for first, third, seventh, fifth patch at the beginning and basically take the patch and then do a follow up patch that cleans things up simply because I know it's very frustrating they send you a code and then you say this sucks and then commit rights on subsurface the only two people with commit rights are Linus and I and Linus hasn't committed a single thing since the day he handed maintenance over to me so I'm the only one who commits the project is small enough it's 300,000 lines of code it's small enough that it isn't really a big bottleneck to have a single committer but I do have developers whom I trust pretty blindly so Linus sends me a patch, I apply it Robert sends me a patch, I apply it there are a few in the team I just I'll do a test build hopefully and a lot of others I do a very careful line by line review and I go back and forth and I try to help them get better but again it's a question of how much time can you think into that I have two questions the first one is about motivation, so when you start a project you're building it for your own learning and to create a solution for yourself, it works for me phase once it's in success and you're the primary maintainer what is your motivation to be running it my second question is training because there is a lot of flux as you were saying of people coming on board and then leaving so every single time you are going through the training process and what I'm noticing is there's a lot of expectation of a little bit of hand holding so how much of times do you schedule and is that what you typically get and how do you deal with that so the first question why does anybody want to be a maintainer it's just love for pain I think and you get to go to conferences and it's reputation so my day job I don't get to write code anymore yet I love programming, it's what I've done for the last 30 years and so for me this is a wonderful outlet where I can say now I'm done yelling at people that's what my children say is what I do for a living let's write some code let's do something and being the maintainer of a project is this ongoing commitment of doing things and being responsive and so it's kind of the excuse to keep doing it so I can tell my wife I really need this I need an hour where I can just sit down and have fun so that is my motivation I don't know what's the motivation for other maintainers I mean everyone has their own drive their own thing that makes them continue to maintain projects and we see projects where the maintainer moves on Linus said okay that was enough somebody else take over, he's done that forget, he's done that for a subsurface your second question was about training so we have tried really really hard to create documentation in the README there's an install file that has step by step instructions how to build the thing we have very clear directions how do we want commits to be we've done we require a sign-off byline we have a lot of information on the web page for the Google Summer of Code we did even more and have that on a wiki so we try really hard that whenever we do what you describe as training whenever we help people to get started every single time we try to make it scalable so put it on the web put it in the file check it into the sources make sure that you just can point the next person to it and if the next person says oh I didn't understand that well then you explain it and you change the file and so the secret here really is to try to make it reproducible I still spend a lot of time on completely stupid things I mean maybe I'm just overly sensitive to coding style but the number of arguments I have about white space are just ridiculous on the flip side I want my code to be readable, I want my code to match my own pattern recognition so I want it to be folder we have a little script that pretty much can clean it up but I also want people who send me patches to kind of understand this is how it's supposed to look well I think we could have many more questions for you but we're out of time and thank you very much for doing the talk and here's a gift from Melcia thank you, thank you everyone