 Okay, now I said, good morning everyone, the next talk is called Economics of Volunteer Labour and it's held by Archie Schleroy, enjoy. Thanks, so I'm going to tell you the three stories from Debian trying to orient you to a few theories that I have about how economics work in volunteer labour of free software production and I should actually tell you first, this is a short version of a talk I gave two months ago at a US conference called Open Source Bridge and I submitted it to Debian because I thought if I'm going to be talking about Debian I might as well tell Debian what I'm saying about Debian. So that's why I pardoned the very twitery opening because that's common in the US, not at Debian. So yeah actually before I begin there's a note taking exercise later in the talk does anyone have a pen they can lend me? Thanks. Oh great, thank you. Great, okay so there's three, there's going to be three important takeaways I hope that you will leave this talk with. The first is the idea of the difference between collaboration and cooperation. The second is the idea that we're not all altruists when we work on Debian and this is important to understanding how to help us, how to get other people to do things and the third is that you can set direction in the project by changing what's easy rather than just by using force let's say. So the structure of this talk is I will give you a brief introduction to economics maybe and then I'll tell you about three stories in Debian CDN.Debian.net, the Darth of web app packages although if there are people working on web app packaging please accept my apologies but I think this is a useful thing to look into and reproducible builds. So I'll begin by introducing economics in very very quickly so this is the briefest economics one-on-one you've ever seen in general people take actions because they they have some motivation normally in like economics in class that's because you pay them something in Debian people have some motivation and it's not super important to me for this talk what people's motivations are but I know that people have motivations to work on Debian after all we spend our time on this which we could be spending on something else for me my motivation is that I I just think Debian is really cool and I feel cooler if I'm working on it and I'm also there's a when we talk about motivation in free software we often talk about burnout I'm not going to talk a lot about burnout instead of this talk I'm going to talk about the gap in between motivation and burnout sort of normal contribution when people are just effectively working on things successfully so in economics then there's this concept of of demand so if I were to offer to pay you all a dollar and you would give me a pen maybe some if you would if I would increase that price to a hundred dollars possibly more of you would give me a pen there's some graph that looks like this that economists like to show inversely if I wanted to sell a pen and I said okay here's a pen not yours I don't want to just take your things and sell them here's a pen okay perfect thank you right if I were to offer to sell them for a hundred dollars probably very few of you would buy them and if I would decrease that price more of you might buy them and in sort of normal price based economics you have a graph that looks like this the intersection of those two charts is this sort of realistic price in the middle capitalism basically works for pens economic theory probably basically works for planning pens and to the extent that it does is because these are what's called rival goods so if I have a pen and you want to use pens we can't both use the same pen at the same time in that sense they are rival our needs can't be met by the same item and so will each of us will probably pay volunteer labor and free software doesn't work that way so you know if I install the Libra office package you can install the Libra office package there's nothing stopping us from doing that so this brings me to my note-taking exercise actually there is no note-taking exercise sorry I just wanted to demonstrate that people do volunteer to do things the point is that they're really there really is motivation in people to so thanks I can get the fact to you there really is motivation by people to do things for the public good and and so we might think that then there must be no scarcity but there is of course the economics there's this definition of scarcity that people have unlimited wants and resources are limited this is my favorite sentence in economics and because there's no prices because there's no money changing hands it might seem like there is no limit on our resources but there there must be because there's things I want in Debian that don't exist yet so in the end people trade their time away in the other things they could spend their life on to help out in Debian and so I don't need to I don't want to spend the rest of this talk harping on the why people do that but I think it's really essential that whenever you want people to do things in Debian that you make sure that you make sure that whatever their motivations are that you can align what you're asking them to do with those existing motivations so I'll skip this so the first story is CDN.debian.net versus HDP.debian.net so how many of you know about CDN.debian.net probably many of us yeah great how many of us know that CDN.debian.net was superseded by HDP reader cool and how many of you could explain why so I think the story is really interesting of course it's about mirrors there's fdp.debian.org there's people who run mirrors I used to run that mirror there mirrors.acm.ghu.edu I set it up 10 years ago when I was a student at Hopkins which is Johns Hopkins University for two reasons one is I wanted package downloads to be faster and the other is wouldn't it be cool if something I made was in Debian in the mirror list how cool would that be so I made it and eventually now it's in the mirror list so the way that mirrors work in Debian generally speaking leaving aside the new fangled CDN and HDP reader stuff is that every user picks what mirror to use so if you guess during the installer that it takes people about half a minute to figure out what mirror to use and you optimistically decide there are 10 million installs per year then there's about 10 human years that people spend every year picking mirrors and I could sort of cynically say it's like we're killing 10 people every year because we're taking their life away in aggregate 10 of those years so that's that sounds pretty bad and we could some someone here who doesn't like Debian could go tweet and quote me on that and then say how bad Debian is and so the thing that you might want to do then is remove that selector but of course that's not necessarily the best way to do it because then people will spend half a minute every time they install something and if you imagine that they install things 10 times a year that's 100 that's 10 times as much waste as if we never asked them at all and so this the point of this here is that we there are trade-offs between different ways to solve these problems just because something is bad doesn't mean that removing it is good I guess so it would be nice if Debian users didn't have to pick which mirror they were using and five years ago some people put together CDN.debian.net and I love the way CDN.debian.net works it's sort of like a riddle the idea of this service is that you do a DNS lookup and that DNS lookup goes up through your ISP to the CDN.debian.net DNS servers and based on which ISP is asking for the IP address of CDN.debian.net it'll answer with a mirror that is supposedly near that ISP so this is the same trick that all content delivery networks use it was sort of pioneered by Akamai in the 1990s and it's generally speaking very effective it's nice that if you're in Australia for example you don't have to reconfigure your laptop it'll happen this will happen automatically but the downside to the way this works well so in it works not just it when you connect to that IP address you had to tell it the HTTP host you're trying to connect to which would be CDN.debian.net the thing about that it uses HTTP name-based virtual hosts so in order for the CDN.debian.net service to add a new mirror the mirror operator has to add a line in their internet for Apache Conf that their mirrors host also accepts this other host header CDN.debian.net so this seems like it would be a very small ask but in fact I had 10 years ago I started running this mirror here I am today in all of that time I never configured CDN.debian.net support on my mirror and sorry to the nice people who made CDN.debian.net but the reason is the motivation I had for running the mirror so I wanted to run it so that you know I could be cool and have my name basically in the list of Debian mirrors and I wanted faster downloads when I was living in Baltimore which is where the JHU ACM is but I don't live in Baltimore anymore so I have no motivation to continue maintaining that thing it works fine I handed it off to the rest of the computer club they maintain it but they don't even know about CDN.debian.net I could have told them I just didn't because it didn't affect me I wasn't motivated to so the trick of HTTP.debian.net which became HTTP reader is that it does HTTP redirects instead of doing a complicated DNS trick so when you request some package when you do a W get say of HTTP.debian.net slash something it says 301 found go look on this other server maybe mirrors.acm.jhu.edu and so the other servers don't have to know that they're receiving these requests because of HTTP.debian.net and so the maintainers of HTTP.debian.net can just keep adding mirrors without any of the mirror operators having to do any work and the key lesson there is that synchronized action comes at a cost in the end CDN.debian.net went away and HTTP reader is now an official service and it's because it's easy to add mirrors without asking anyone to help but interestingly HTTP.debian.net is very slightly slower than CDN.debian.net in a median case because you always have to do this redirect but this technologically suboptimal outcome is actually sort of a community optimal outcome because it requires the least collaboration sorry yeah the least collaboration among people so David Eves has some terminology for this which is that if you are asking people to do something and they need your then and you both need to approve it before it goes forward that's collaboration and a cooperation is when you can do something and build on top of someone else's work and you don't need to communicate and in Debian where possible if we if we can achieve good outcomes through cooperation not collaboration it's so much easier and cooperation is just so much cheaper that CDN.debian.net lost which is fine we have a nice service now and CDN.debian.net redirects to HTTP.debian.net so no one's harmed or anything so that's the end of story one this distinction of collaboration versus cooperation story two is the question of web apps in Debian so maybe I can get a show of hands how many people here run WordPress either for yourselves or for a friend I know I do okay great keep your hands up and now how many of you are using the Debian package for WordPress oh wow wow okay like maybe 15% that's actually more than I expected but the other 85% of us don't and I think this suggests there's something wrong we're doing with the way web apps get packaged in Debian of course I'm kind of biased because I work on a different web app packaging project but I hope that the suggestions of the problems are useful so I want to tell you a story not about web apps but about Alpine so I maintain this program called Alpine it's a mail reader it looks like this and the reason I maintain it is that in 2006 Alpine was released and it was released as 0.80 beta and I thought a few things I would love it if I were the person maintaining Alpine in Debian and also I want to upgrade to 0.81 when it comes out so I want to make a package on my own machine so that when 0.81 comes out I can update to that very easily and the Debian package format is super easy for that so it made sense for me to make that package for myself and it was only a small marginal cost to share that package with the rest of Debian and I think if we think about the incentives of package maintainers once I made this thing and I wanted to share it I also wanted that to be a good package and so I asked people to review it and now many people use Alpine on their Debian systems I think I've looked at popcorn and 1.74% of Debian systems have Alpine which is kind of amazingly high I don't know what they're all doing but I could use it myself immediately so it was a useful thing to make that package the the interesting thing is if I really cared about Alpine users I would be the Fedora maintainer of Alpine too I would be distributing Windows binaries too but I'm not and it's because I'm not really an altruist who loves Alpine and wants everyone to use Alpine I just wanted it to be easy to upgrade on my laptop and the other and the some computers in the computer club and then from there was a small step to jump up to maintaining it for all of Debian so I think the question for web app packaging is how do we align these incentives so that the way that we make web app packages is very similar to the way that people want to install them on their own machines if we can get that right then I think we'll have many more useful web app packages in Debian and there is there's an economic term for some of this which is positive externalities which is the idea that I'm doing something and it just happens to benefit other people who weren't part of the transaction another great example of positive externalities is when the government makes parks people go to parks they bring their friends to parks they all have a nice time no money changes hands but I'm only going to do this if the value of making that package is greater than the cost of doing it and I hope that we can bridge this gap because there's lots of great open source web apps that aren't packaged there's ethercalc there's H ledger web web based nerdy accounting there's line survey all these great things so that's lesson two that we're not altruists lesson three well story three is about reproducible builds so lunars in the room I hope I get my story right so you all know that the point of binary reproducible builds is that we want a devian package to be verifiable by some external party so that we can then check if the buildings have been compromised we can know and and reproducible builds it seems surprisingly hard because the tool chains for building things just weren't designed this way and so most programs don't compile the same way twice and then our talks a lot more about this earlier but to summarize just one aspect of it if you look at a gzip file you will discover that it has the time stamp of its compression so two years ago at debcon Lunar and I were talking and he was saying wouldn't it be great if debian had binary reproducible builds that was two years ago summer 2013 right around the time the snowden leaks were happening and wouldn't it be great if debian were trustworthy operating system in that way that anyone could verify and unfortunately then our said the problem is too technically complex there's too many things to fix there's no way I'll convince a thousand maintainers to fix their packages and I said it's both a technical problem and a social problem people think the cost is really high to them and so they're not going to do it because the cost is greater than the value for that developer the individual developer so I figured the best thing we could do is make a convincing demo take like some package somewhere make it build reproducibly punch out this time stamp and make a team around it so Lunar then organized a boss two years ago at debcon and I think I don't know 30 people came it was huge I remember yeah and the idea was we needed to research together what things made packages non reproducible and there was became a wiki page that anybody could add to and people did the independently research parts of it cooperation not collaboration and also around that time I sort of vanished from the project got a new job at event bright and luckily other people picked up the work and I kind of haven't done anything for it since but I can at least take credit for smiling at Lunar at the right time and what ended up happening was that there's only a small number of layers where work has to occur in order to get a huge benefit and those layers can be done cooper fixed cooperatively not collaboratively and in the first experiments all it took to get 62% of the archive reproducible was fixing just a small number of packages and these three packages let you reproduce more than half of the archive and what's funny about this is looking at the mailing list posts from the past where people say yeah it's too much work so in 2007 somebody says I see no benefit someone else's I for one think this is technically infeasible but interestingly in 2013 the value maybe increased because we saw documents from the NSA that said things like they were attacking macOS dev tools they meaning the NSA sandy national laboratories so luckily the value is increasing but Lunar managed to decrease the cost also and the whole project of reproducible build is about people to a large extent there's sort of two kind of fixes to reproducible builds problems there's fix the tooling and fix the package and to the extent of fix the tooling works it means that the maintainer gets this cool thing for free so of course the cost is greater than the value because the cost is nothing so the others and you can see that in their in the monthly reports so this is a quote from one of them due to changes in the build dependencies these 32 packages became reproducible we never had to bug the maintainer we just fixed it for them that is efficient work so I want to sort of highlight how efficient the reproducible build project has been and now it's around 80% I think of packages that build reproducibly maybe a few more one thing I got wrong is that I said there would be about maybe 12 or 20 types of issues I think there are 132 kinds of non reproducible aspects found so I got that one technical thing wrong when I smiled at Lunar two years ago but but the big lesson here is that you can set direction in Debian not so much by telling people not just by telling people to do things but by changing what's easy in the project and that's sort of the lesson of Debian policy but it took me a while to understand how powerful that is so that's about the time I have there's three takeaways then consider cooperation over collaboration remember that we're not altruists and you can set direction by changing what's easy so thanks a lot and I'm happy to take any questions yeah a question great I like to rephrase on this altruists argument I am also often frequently addressed as an altruist I think I am but I have also non altruistic motivation to work on Debian right like this right I just think it's so interesting that I don't maintain Alpine of Fedora like why don't I carry even that much about Alpine users it's not easy for you right okay so I actually love the collaboration versus cooperation but that's also why that company is so valuable is because it's time for collaboration yes yes exactly but you can only get so far we're just playing cooperation so yeah it's also exactly how Debian has grown you know the very simple fact that over the years we've had thousands of people all maintaining their individual packages they don't have to collaborate every day they can just go away and do their own pieces it works well exactly yeah I kind of proposed this talk to open source bridge because I wanted to basically say hey you knew hip open source people Debian has something I can teach you so yeah exactly um so I do notice that you smile a lot so that might increase the odds of having an opportunistic smile and I'm also curious if you have any ideas of what might be another easy thing we could change that makes a big impact when you say another are you saying that the first one is smiling I'm not sure I understand because that thing that was mostly a joke because the thing I said with Lunar wasn't just like a smile it was it's a technical problem as well the social problem and you can decrease the perceived cost by demonstrating that it'll be easy and I mean just the the release team already understands this and they they talk about setting smart goals specific measurable actionable R&T timely goals basically that if you're proposing a really something that everyone in Debian needs to do you need to be able to figure out if it's actually been done like successfully been done by the end so when you say one other thing are you saying that reproducible build is the thing or that smiling is the thing or okay I mean I think that if there is one thing it's getting people to talk to each other and find out which parts of their work they could think about differently in order to have them be easier and that's something we could all do peer to peer so you know after this talk find someone in the hall and ask them what are you working on and why is it hard and how maybe how can we make it more more around cooperation and less around collaboration and things like the Pearl teams work on having very standardized processes emphasize cooperation maybe we can do that for people's individual packages to I know that Odeeks was telling me that the print the printing team he likes to say the printing team because it's basically him actually appears to have grown and has a new member and somebody just showed up to the bug tracker and has been triaging printing bugs and maybe there is just a request for help where people can take actions that don't need anybody's approval to merge them and we can just make more of these requests thanks for your attention the talk time is over and see you in a bit for the next talk thanks