 You guys, do you have a lab up here or do you need to give them? Okay. Okay. Okay, you got one of those beer? Yeah. Okay, I'll flash. All right. So you good to go? Do you have any audio or video with your presentation? Nope. Okay. All right. Lab is off for now. I'll turn it back on. You want to start talking? Level seemed okay and I'm loud anyway. With you ready, we're supposed to start at 10.40, but I'm going to probably give people another five minutes, just let everyone filter and think, you know, you can still hear someone talking. I don't know if they're still going or not. We will talk about this one super long anyway. Do you have any questions here in the course of the talk or later? Please signal your desire to have the question and I will take a minute or two so you can be recorded. So people might still filter in because it sounds like there's still chatter out there, but let's get started. So that's me in all of the places. I'm an infrastructure engineer at a company called Ride Cell and I'm here today to talk about open source sustainability. Before we talk about sustainability, let's lay out a little bit of scope. A lot of people have very specific opinions about what constitutes open source. I'm going to take a much more general stance away from the specifics of what precisely makes something open. I'm also not going to wade into open source versus free software at all. So that other way, let's look at a few types of projects of what I consider open source. The majority of projects buy the numbers. One person writes some code to do a thing and they put it up on GitHub because why not? This is how I got my start in open source writing plugins for a bug tracker called track. But the more visible projects are the big ones. Is a project so vast and omnipresent? It's easy to just think of them as part of the fabric of the universe. Some of those very big projects eventually actually have to build legal entities around them, usually projects that are partnerships between multiple large corporations. But a newer venue for open source projects has been VC backed startups. So the company acts as a steward for some form of open source project and they sell stuff on the side. We'll get to the selling later. What I call mirage projects feel like an offshoot of that so big it's part of the fabric of the universe idea, except these often have frighteningly few core maintainers. We all saw this during Heartbleed where the entire world sort of simultaneously realized that there were a single digit number of volunteers maintaining this enormously important core infrastructure. Some older projects, especially big Linux packages, tend to feel a little bit stuffy compared to the newer wave of GitHub generation projects. But these projects have a very important history. They've been designed to stick around for the long term as individual contributors come and go. Some open source projects can only be described as crazy ridiculous moonshots, usually led by a single person and sort of organized around, we just have to get this done. And in my very wide view of open source, there's a lot of things that aren't software too. Some of the legal specifics differ on how you license and maintain these, but the overall sort of contributor dynamics are basically the same. It's clearly not a complete list of every type of open source project out there, but it gives you a taste of the kind of breadth that I'm going to be talking about. So that's two words from the title, what about the third? Literally speaking, sustainability means taking a project that exists today and ensuring it will exist tomorrow, and the tomorrow after that, and so on. Money and time are important, and we'll get to talking about those in a few minutes. The fundamental limiting factor on most open source projects is the rate of burnout of contributors and maintainers. Most projects fail not for technical reasons, but because people don't want to work on them anymore. The most common symptoms of burnout in open source are these three things, a cycle of feeling guilty about letting down your users, leading to dread about working on the project, and finally bitterness towards the project itself for sometimes the users of the project. Open source projects have truly taken over the world. In almost every facet of technology, communal resources have pushed us further and faster, and if we want this pace of progress to continue, we need these communal and shared resources. But with this cross-pollination comes interdependence. While open licensing helps to ensure the possibility of someone picking up a fallow project, it doesn't help us if there's no maintainers there to pick up the fallen banner. And if I can digress for a few moments about why I think open source itself is important, I think we can build things together much better than any one team or organization. A lot of problems on software, there's so little value to the market that it's hard to define how the market would even come up with a solution for this. But collectively, all of us together can and we're all better off for it. So open source rules the world and I think that's great, but clearly I also think there's problems with the system or I wouldn't be giving this talk. How do we end up with a worldwide ecosystem of software projects built on the backs of a relatively tiny number of volunteers? If we want a globally simplistic answer, infrastructure investment has been lagging across the board. Putting time and money in things like bridges and dams doesn't show returns immediately. If a return can even be quantified, if your focus is only on the earnings report for the next quarter, things are gonna slip through the cracks. But a deeper structural answer, at least in my view, is in how we make things. I can only put my brain towards building one thing at a time, but while I'm doing that, I'm using probably hundreds of other projects and tools. Even in the best scenario, there's always going to be more users than contributors and that imbalance has spiraled far out of control for most open source projects. Even a simple project like a Ruby on Rails web app. I just sort of made this up off the top of my head. The Rails app at the top of the list sits on a mountain of projects and libraries. As we build better and better abstractions, which is great and we should keep doing that, it becomes easier to forget the shoulders of the giants on which we stand. But why all this interest and sustainability lately? Way back when, open source was just some crazy college students, several of them actually from right around here. Basically everyone that consumed open source also worked on it. So maybe I worked on GCC and you worked on Bash, but we all used the same sort of suite of tools and we all benefited from each other's work. Over time, this balance eroded. More people pull value out of the system than are putting it back in. As the third ish wave of open source developers are reaching middle age and deciding that maybe other pursuits are more important, often there's no longer fresh faces willing to take up the banner. And so here we are. It's hard to pick a patient zero, but it's probably either the lamp stack or Ruby on Rails that got the ball rolling on corporate use of open source. VC funded startups, usually in the early 90s, realized they could switch to friars and beer tools and save money. And the VCs didn't know about this, so the checks stayed the same size. And when being cynical, these startups used that money to just add their own profits. If I'm being nice, they used that additional money to double down and figure out how this whole web thing was gonna work, because that was still pretty new at the time. But all good bubbles must come to an end. And by now startups and big companies alike know that zero costs, software stacks exist, and many are of extremely high quality. Budgets have all adapted to this. So companies often have no choice, especially small new startups. They can't afford anything else. At the end of the day, open source and capitalism are at loggerheads. Capitalism requires, or at least strongly encourages, privatization, and open source is the exact opposite of that. As long as the bulk of companies continue to draw more value from the commas they put back in, we're going to have a sustainability problem. Much of the interaction between capitalism and open source is related to intellectual property, but I did a whole talk on that last year. So that URL will be up again later if you wanna check that talk out. So that's the general answer to why things went wrong. What about some specific projects that were ultimately unsustainable in one way or another? Two that are very close to my heart, React and RethinkDB. These were both originally VC-backed startups, but they were open source projects from the beginning. The founder of RethinkDB put up a fantastic postmortem on what he thinks went wrong with the company side of the equation, and I think it should be required reading for basically everyone at open source. But roughly, it was an extremely saturated market, both in finance and in mind share, and they just couldn't get a foothold. Whatever you think about MySQL and Postgres and whichever of those you prefer, they've set an extremely high bar for what is available for free and people are just not really willing to pay for databases anymore. Big player in the Android open source world, Sanagen, and their former OS distribution Sanagen mod. In 2016, there was a big management shakeup of the company and they looked at their books and they decided the benefits of being open source in terms of contributions they were bringing in were vastly outweighed by how much they were spending on user support and the amount of money that they could pull in from people that would pay for it if it weren't free. So they decided it just wasn't worth it anymore. Two notable individual cases, why the lucky stiff from the Ruby community and Mark Kilgore from the Python community. I say this not as a call out or critique of either of them, but it's cautionary tale if you put too much of a burden of a single community on one person, it rarely works out well for either side. And a more general example, drive by contributors. People that just sort of show up, they do one commit and they leave. Drive by contributors bring really important fixes. They tell you when your user experience is bad. They fix weird rare bugs and sometimes you can convert them into regular contributors or even maintainers eventually. On the other hand, it takes a lot of time and energy to shepherd that first contribution and doing that over and over and over again can be extremely draining. There's not a great answer here, but just be aware of the problem and step back from mentoring first time contributors if you just don't have the energy to deal with it anymore. That's okay. So that's things to track from sustainability. How about things that improve it? A big part of any project is how it's run and that can have a similarly large effect on sustainability. A benevolent dictator for life is where most projects start from, projects created by one person and more contributors slowly join, but that one person remains in charge of figuring out the overall direction where the project is going to go. This can work very well for a young project, but eventually having to run all those decisions through one person can slowly create more and more friction and frustration over time. So some projects that started out with one BDFL eventually fragment into a number of these smaller BDFL like maintainers that all just sort of exchange patches. Linux is the canonical example of this. There isn't actually one Linux kernel. There's like 40 of them and they just all work with each other strange ways. This can reduce friction for the maintainers that are exchanging patches, but it can be intimidating for a new contributor trying to figure out how to get a patch in and where it even goes. Some projects have a more rigid hierarchy for decisions. This makes it really easy for engineers to work out disputes. There's very clear dispute resolution processes, but it brings a lot of social overhead with it. Others use a federated council approach. Usually these are called SIGs, but that's sort of the newer term. Each group has final say over their area of expertise. And sometimes when the contributor base is stable enough over time, you can forego basically all the rules and just say, work on rough consensus. When enough people agree, just do the thing. And if we need to change it, we'll change it. This can feel great for long time contributors, but it can also really easily result in hidden power structures like the one person that is in charge, but is not marked as in charge. And that can be utterly incomprehensible for new people. And there's a lot of other models. As with the project types, I can't possibly hope to exhaustively list these. Every project isn't gonna be only one of these. They're gonna pull together a set of governance rules that matches, hopefully, their specific circumstances in people. Conway's law states that the output of any team is going to eventually look a lot like the org chart of that team. It's intended to be a bit of a tongue-in-cheek joke, but it's also remarkably accurate. Choice of leadership and decision-making structure doesn't just affect the humans. It affects the software that those humans produce. Unfortunately, it can be a bit hard sometimes to tell which direction to move in to improve sustainability for your project. For some projects, more structure can help to reduce the burden on overloaded maintainers. On other projects, giving them more paperwork to fill out will push them even further into burnout. So when you're playing with governance structures, always make sure to listen to the people that you are governing and make sure to change things if it's not working out for them. As I mentioned before, open licensing is an important component of open source. It's not directly related to governance model, but I needed to put this somewhere and one often influences the other. A good license ensures that even if all of the contributors from a project move on, at least in theory, the source code will remain available. It'll still be up somewhere and people can use it, they can fork it, they can continue the project. What about cases where the license can be more than just a backstop? Full intricacies of GPL or copy left general licensing are better left to that intellectual property talk I mentioned before, but the idea of copy left is in its own way sustainability. GPL doesn't actually care about the money side of sustainability, but we'll talk a little about dual licensing later, which sort of gets into that. Exactly how it works depends on the GPL flavor that you use, but the overall idea is that if someone is going to modify your code, it has to be available to the users and thus probably to you as the maintainer. One failure mode of those really big projects I mentioned before is that if they use a permissive license, someone can fork it, change it to keep the patches themselves, and in theory that can siphon off resources from your project onto their fork. Again, in theory, if you put your project under copy left license, it heads off this whole cycle because you can always get their patches and hypothetically if you want them, you can introduce them into your own project. But really how often has this happened? Definitely not zero big projects, notably the Linux kernel itself definitely have issues with proprietary extensions causing friction for both users and maintainers, but my personal view is that this is probably not going to be a thing for your project. I think that copy left probably for most projects causes more problems than it solves, but maybe your project is different. But okay, in the end, maintaining open source is about time and time is money. The simplest funding model is to just not have one, be a volunteer project, and those are great. Some people just like coding for its own sake, some people want to improve their professional portfolio, but remember, volunteering your time is an act of altruism. I speak from personal experience, it's real easy to end up in that cycle of guilt and bitterness that I mentioned before until you just never want to see a computer again. The next step up is working on open source on your normal work hours. This might be a project that your company maintains, that you're improving, or contributing a bug fix back to a project that your company uses, and it's a great option when you can do it, but a lot of companies don't really understand why they should pay someone to give something away for free, so not always an option. The next step up from one person using company time is everyone using company time. It's not a long-term solution, but it can be a way to get a new or nascent project off the ground. You'll need to look into some of these other options to eventually repay your investors. That's the idea at least. So now we have to do the round-up projects that are trying to be self-sustaining, to bring in enough income to support at least some of the time that you are spending on the project. By far, the most popular method is to sell support contracts, but this can be really hit or miss. Small companies often can't afford anything that takes away from their launch window, excuse me, and big projects just think we'll support ourselves internally. Examples of support contracts, there's a lot of them, but MySQL and Java are two particularly popular ones. Sometimes the project itself, or the community that stewards the project, isn't really set up to be a commercial entity, so the maintainers will sort of split into a bunch of smaller companies, and they'll sell support contracts and other services through those, but again, with the same idea of funding their work on the original project. This is the dominant model in the Postgres community, for example. There's no Postgres corporation, but there are a sort of fleet of smaller consulting groups around Postgres, and a lot of the contributors work for those. After selling support, the most common models are based around having an open core and then selling related close source components or differently licensed components that add additional functionality to cater to specific use case, usually big enterprise software. Sometimes there's clear places to add these paid add-ons, like big enterprises love Sammel login, they love cross-team reporting, et cetera, but other times it can be difficult to find a place to sell these features without harming your open source users in one way or another. It's been very popular in the DevOps space, so Chef, Pop and Salt and Ansible all sell commercial add-ons. Dual licensing is a special case where you sell the same software they could get for free, but under different license terms. So the common case is you have a copy-left-based project, somebody wants to fork and modify it, but they don't want to make the patches available, so they purchase a license from you that doesn't have the copy-left restrictions. It's not very common these days. QT is probably the best example of somebody still using this. Similar selling support contracts is professional services. So this is usually some mix of consulting to help them best use or deploy your product, customizations making custom plugins, custom modifications to the project for that customer, or hosting if it contains a server component. Again, examples too numerous to list, but Pivotal Software is a big one, they sell services around Cloud Foundry. Grants have been a mainstay of the academic world since time immemorial, and we're slowly seeing more of them available for software projects. Like with other kinds of grants, you write a list of things you want to do, why you think they're important, how long you think that'll take, and use that to come with a dollar amount, and you go and ask someone for that money. I'm sure the PSF wasn't the first offer open-source grants, but the first that I ran into. A lot of other groups have copied them in the Python ecosystem and offer project-specific or ecosystem-specific grants. Going a bit out from that, the Moss Grants Program, or Mozilla Grants more in general, are another example. Rather than being ecosystem-specific, they're looking for anything that helps with Mozilla's foundational mission of improving the open web. And then one step yet broader, places like Google's open-source programs office, they have a mandate to support basically every project that Google uses. That's basically all of them at the scale of Google. But a big problem with grants, like with VC funding, or any other non-renewable income stream, it's highly dependent on how good your time estimates were. If it turns out that you miss some complex requirements, and it takes you an extra six weeks to complete the project, that might end up coming out of your own project, or the project might fail. We all know that engineers are super great at estimation, right? So we reached the option most people go with. If you like this, how about sending me a few bucks on PayPal? This is so common, I wanna dive into some specifics. So the symbol of solution, straight up tip jar on PayPal, or Venmo, or Bitcoin, if you like that. Really easy to set up, but also very rarely useful. It can be heartening to get a few dollars every now and then, but if you're not okay with working for free anyway, it's probably not going to help on the actually funding your time approach as a maintainer, I'd much rather, and I see a worded thank you email from one of my users than like $5 from one person once. A more stable solution is some kind of recurring donations. This lets you plan out how much income you'll have for the month, you can figure out how much time you're gonna allocate. Patreon basically owns this market right now, but they haven't really reached out to the software world, so it can be a little bit difficult to get people aware that your Patreon exists, and they don't really have support for corporate giving very well, so it can be difficult to get donations from companies. Which leads into a problem that I think both recurring and one time donations share. Basically, it almost always ends up being friends paying each other rather than the companies that are driving economic benefit from my software donating to me. In general, I don't want individual users to ever feel like they owe me for my software. What I want is the companies that are making money off it to get some of those resources back to me so that we can both benefit. But okay, some other donation options. I did actually run a Kickstarter to fund some open source work. It was successful, technically, but it was extremely stressful, and I really wouldn't recommend it. Like I mentioned with grants, I misestimated my timeline by about three months, and Kickstarter's like a grant. The amount of money you have is the amount of money you have. You get it once, and you better hope that that was correct. Some mid-sized projects that are looking to tempt donations from companies rather than people will try to get nonprofit status, so their donations can be tax exempt, at least in the US. But creating a nonprofit from scratch is really hard, and I don't really recommend it, so it's usually a better bet to join an umbrella group, someone that already has nonprofit status and can accept donations on your behalf. Especially when highly the great work of Software Freedom Conservancy, they've helped dozens of projects set up this umbrella donations system. If you're interested in doing that for any of your projects, I highly recommend talking to them. You've got a contact form on their website for it. Outside of the simple logistical issues, a common problem with donations is working out how to divide them if there's more than one maintainer. Sometimes you can side-set this by spending the donations on project-level resources like server hosting or buying t-shirts or whatever. But it can very easily lead to hurt feelings or perverse incentives in the project if you end up paying by the line or by the commit. While money makes the world go round, many donations take the form of discounted or free credits to services. So for example, Rackspace has very graciously hosted part of the CI infrastructure for my chef cookbooks for many years. It's often easier to ask a company to donate whatever their service or their product is than to go and ask them for money. But okay, enough about donations. On to some less common but more interesting funding strategies. Various projects have tried to fund themselves by selling books. Most notably the Lua language did this, but it can be a really difficult line to walk because you still need some free documentation if anyone's gonna use your project. So you have to figure out how to balance those. Oftentimes these days it ends up being if the book is available online for free, but hard copies are available for purchase, but that'll severely impact your revenue. Selling ad space can theoretically be a very lucrative option. Ad sales basically run the entire world these days, right? Setting up Google ads on a website is super easy to get started with but doesn't actually bring in very much revenue. Negotiating your own direct ad deals can bring in a lot more money and give you a lot more control over what ads you show, but it's very time consuming and difficult to get those kinds of things set up and it all might be for naught if your highly technical audience are all using ad blockers anyway. Bug bounties are most common in the infosec world as a tool to try to incentivize people away from selling bug reports and attacks to the bad guys, but they're starting to appear for normal software, other non-security related things like normal features or bug reports, but they're probably still too rare to really qualify as a sustainable funding source. Maybe we'll see this get better in the future. Worker co-ops are a really deep rabbit hole and they all run differently, but we're starting to see some based on open source, Snowdrift and Tiedlift are both getting more interest, but we'll see over time if they are successful, I think right now it's still too early to say. And so we look further out into the future. With science medicine and the arts is a lot of national and even international organizations that provide funding for stuff like basic research. Basic research in science is things that do not really have direct rewards immediately, but they form the basis for future work. That sounds a lot like a lot of infrastructure level open source projects. So maybe one day we can see a national institute of software. There's some government grants available here and there, but it's not really a coordinated system. And if I can dream really, really big, universal basic income would allow people to spend time building things they think are important, even if the market doesn't directly value them. So hopefully by now I've convinced you that sustainable software is important and you've got some ideas brewing for how we can all move in that direction, but what can we do now? As I stressed before, the enemy of open source is most often burnout, not technical failings. Practice empathy both while using and maintaining open source software and it can help a lot. If you make money using open source, which is almost every company these days, be on the lookout for ways you can help or donate to projects that you depend heavily on. And most of all, always be ready for change. Maybe the only maintainer of a critical library for you will have a kid and decide that they don't really care about writing open source anymore. That's okay. Maybe you maintain a library or a tool or something and you get a new job and you decide, man, that thing, it's just not fun anymore. That's okay. And maybe the entire ecosystem will change. Maybe the DAOs on the blockchain will actually take off and that'll be how we write all software forever. That's okay too. I don't believe that last one's going to happen, but that's a different talk. We've all only been writing software like this for a couple of decades. We have a lot to learn about how to do it properly. Thank you very much. There's too many people that I can thank if I had time for their advice and wisdom over the years for how to run sustainable projects. But I'd like to extend an extra special thanks to Nadia Russell and Eric as true luminaries of the space. I highly recommend all of their writings if you're interested in diving deeper. And we have a few minutes for questions depending on when we're supposed to end. When are we supposed to be done? Just so I can keep mine on the clock. By the number 15. Okay. As someone contributing to like a 10 plus year old project that with multiple contributors, I'm kind of feeling mild burnout about like, oh, I don't want to change this. I might break, I could probably break users. I mean, the only thing I can think of is just try to have both, you know, a long-term support version and a branch and a more actively developed branch. Do you have any other advice? Unless someone is paying you, you do not owe them long-term support. I mean, if your project is robust enough and has the maintainer backing to support an LTS release, that's great. But otherwise, just don't. I mean, yes, users will be annoyed, but like, you have to factor in what is going to be best for you as a maintainer, as a group of maintainers. And like, you don't owe your users that much. It is important, but it's not that important. Thank you. Just a footnote on that. It isn't necessarily a binary thing. You know, I mean, Rails, for instance, is pretty, you know, is more aggressive than a lot of people would be about, you know, ditching features that no longer seem like a good idea. But what they do is, you know, what they do is to deprecate and delete on a schedule. Thank you. But, you know, that is a good approach, but users will still be mad. One of the projects that I maintain, we went two years, two and a half years doing no breaking releases because we were very worried that our users were gonna get, and it was terrible. We're never doing it again, ever. The users loved it, but basically all the maintainers said, if anyone ever suggests this again, we're all quitting because there was so much technical debt to clean up at the end of those two and a half years that we were all basically drowning in it. And in the end, I think it wasn't actually better for users. Like, burning out and making all of your maintainers hate their job is not actually in the best interest of users, even if they think in the short term that they would rather not have breaking changes. So it's, I think, a lot about communication and communicating expectations. Any other questions? Awesome, well, thank you very much and enjoy the rest of DevCon.