 In this next talk, we will talk about philosophy, and this is Peter Stugl, he has many years of experience in software, hardware, security and networking, and now failure, because Peter is going to tell you his story with Lipp USB and about mistakes and things that he learned with that. Give him a warm welcome, please. Hello, my name is Peter, I'm a failed open source project maintainer. Alright, so I will, this is not a technical talk, I will talk a bit about what Lipp USB is, because of that, just mention briefly, I will mention a few things that I've seen in the Lipp USB community and in related communities over the time that I've been involved. I will tell you my story, I will try to tell you my story, I will talk about some mistakes that I made, and I will talk about what I learned from all of this, hopefully. So Lipp USB is a software library for programmers that want to communicate with USB devices of different kinds in an easy way. It's pretty widely used in Linux systems, it's also used in some Mac OS software, and even some Windows software here and there. If you've used a USB printer, a USB scanner, some digital camera, some media player, then it's not unlikely that you've actually been using Lipp USB or the software that you've been using has been using Lipp USB. FreeBusy has implemented Lipp USB on their own, for licensing reasons, as far as I understand, Lipp USB is an LGPL library, which means that it's not quite as, the license is not quite as permissive as the BSD projects, and so they chose to write their own. This is not a technical talk about device drivers, as I said. I gave a technical talk about USB and Lipp USB five years ago. There's the link. Check it out if you're interested in that. So while involved in Lipp USB, I've noticed or observed that even though this is a pretty widely used library and a pretty common project, there are very few people actually working on development here, which is a shame. I wish there were more. There's also, there seems to be this tendency to not work together very much in the extended Lipp USB community, I would say. There are many different Lipp USB alternatives or sort of variants of the Lipp USB concept, and they traditionally haven't done much to work together or they've even started because of disagreements while attempting to work together or after trying to work together. So there's the original Lipp USB 01, that's the very first one, then there was the Lipp USB 1.0, which is where I was the maintainer. Meanwhile, there was the Lipp USB win 32 project, which was completely standalone. It's sort of compatible with the Lipp USB 01. Well, it is compatible, but it also extends Lipp USB 01. It adds new features, and it's made specifically for Windows. But the developers working on Lipp USB win 32, they didn't really make an effort to join the sort of the real or the original Lipp USB project. They just kept to themselves and made their own thing, which caused some confusion for the users because they had to, they all they knew was that, well, this scanner software or digital camera software or whatever it uses, Lipp USB, so why do I need to, what's the difference? Why do I need one or the other? And this of course only got worse when the Lipp USB 1.0 started working differently and started requiring programmers to do different things to use the new features, which Lipp USB win 32 didn't support and never wanted to support and development had stopped a long time ago. Then there's open USB, I'll come back to that in a bit. Then there's Lipp USB X, I'll come back to that in a bit too. And on the side of that, there's Lipp USB K. So the first group of these, they're sort of similar. Open USB is probably the most different. They decided to make some large changes. Lipp USB K is also different. It's got its own line simply because I don't know exactly how many developers there are. I think there aren't really many. There's probably also one or two, but I don't know for sure. So I wanted to mention it separately. Moving on to my story, I joined the project in 2003. I just wanted to use Lipp USB. I was interested in USB technology. It seemed new or it was new and it seemed interesting and useful. I was a mostly lurking subscriber. I read the mailing list. I tried to answer questions when I could. I tried to contribute to the community as you do. Of course, according to what time I had and so on. A few years later, 2005, starting winter 2005 and going all through the year of 2006, Sun, MicroSystems, anyone remember them? A few hands, okay. A few of them joined the mailing list and joined the Lipp USB community and they were working on what it seemed to be a thin client USB support or something. They wanted to have USB over the network. So they were trying to drive. They were putting a lot of effort into the project and trying to set a direction. But the maintainer at the time, Johannes Adfeldt, he didn't really agree with them and a few other people, including myself and some others, we also didn't really agree with them. So what happened then is in January 2007, and I had almost forgotten about this after winter there, the Sun people, they published or started a friendly fork called OpenUSB, that's a separate project. And they did it in a fine way, I think. We had some at times difficult discussions before they started OpenUSB trying to figure out how the next version of Lipp USB was going to work. And they had their vision and other people had another vision. They went off and they did their thing. They got a new name and they went for it. It wasn't successful, but you know, you never know. And I think they did really the right thing. A couple of months later, the then maintainer Johannes Adfeldt offered me to take over the Lipp USB project. He had spent a lot of energy discussing with these OpenUSB people, and he didn't have fun in the project anymore, and he said he would be happy to let me take over or he would be happy to, he mentioned that there might be another few people who could take over as well. And I thought about this, and I was really happy to get this offer, but I didn't feel that I had been in the project long enough. I didn't feel that I knew the code well enough. Even though I agreed with him on many points, I still, it didn't feel like I would be able to do a good job, so I said thank you, but no thank you. But I'll stay in the community and I'll try to contribute, continue to contribute. So then nothing happened because Johannes, he was tired and essentially there was no active maintainer in the project for some time until January 2008 when Daniel Drake, who came along, started working quite a bit on the code. He had already done something similar for some fingerprint software that he wanted to implement or that he had implemented, and what he had, the ideas that he had for USB device communication, they fit well with the project, so he naturally fitted in this position of the new maintainer. Johannes announced it and everybody was happy. I was also super happy to have Daniel in mode. I continued and in summer, later in summer, I set up a bug tracker and some source code hosting for the project. Daniel continued development, so Daniel made the first Libusby 1.0 release after he had started in 2008, and he made several releases, and in 2010, in spring, he released 1.0.8. So that was the last Libusby release that he made or released and two months later, so end of 2009, beginning of 2010, there was a lot of work going on trying out to add Windows support to Libusby 1.0, and out of nowhere, there came this new contributor who was pretty active on the mailing list, and very clearly, he put a lot of time, a lot of effort into Libusby, and I had actually created ticket number one in the new bug tracker, which was to add Windows support, and I always thought that this was an important step for Libusby for several different reasons, you know, get more open-source software working on Windows, but also maybe trick some Windows developers into start using Libusby and then we get the Linux support for free, right? So I was excited that somebody was working on Windows, because I didn't know anything about how Windows USB worked, really, the little I knew was that it was pretty complicated. So anyway, this guy joined and he sent some diffs, he didn't send actual patches the way you would generate them with Git normally and send them to mailing list, but he sent diffs without any commit message and not so structured, and I gave him a lot of feedback about these, and I was expecting that he would send another round of patches, as normally happens in open-source projects, with at least some of these things addressed, where he sort of thought about what feedback I had given, and no, that didn't really happen, so there were new patches, but not much of the feedback had been considered. So this continued through the beginning of 2010, and one of the last things that Daniel did before he moved on was to commit some of the Windows support into Libusby 1.0. Now, I thought that was too early. I didn't think this code was quite there yet, but Daniel had done it, and I didn't really want to object or I didn't want to revert the source code, undo what he had done. I thought that was a bit of respect, showing not so much respect for the previous maintainer. So the solution that I saw was, okay, let's fix this up and let's get the next release out, because it's high time, it's nice to have releases every now and then, but I also felt that it was important not to add too many new bugs, or leave too many bugs, or if we release something with Windows support for the very first time, at least it should work. And that took a long time. The development in this time, I wasn't working so well together with this guy who joined, who did much of the Windows work, endless email threads on the mailing list, and during this time, a lot of people, they just abandoned the project because they couldn't keep up with the volume of email, and I found it hard, too. I don't know what the other guy, how he felt about it, but he kept doing it anyway, and I was still happy that there was a Windows developer, but I also felt that it was very difficult, I really felt how difficult it was to work together. I kept pushing forward, I can be pretty stubborn sometimes, so I kept pushing forward and just trying to work on the code, and trying to get the Windows support into shape, and finally made a release candidate the first one in 2011. Then, a few months later, there was one person on the mailing list, somebody else who wrote this, fundamental, incomprehensible, and it seems unresolvable, disparity between what you see as reasonable, and the right way to contact development, and what the rest of us feel. This was just one of these kinds of emails, there were lots of them, and of course, this didn't help my motivation very much, but not even help my motivation, it was hard, it was hard to hear this, but I kept going, because I felt that we should get this Windows support done and released. A few months later still, April 19th, 2012, there was an email thread started on the LibUSB mailing list announcing a hostile fork with the subject LibUSB is dead, long LibUSBX, and yes, to celebrate, so this is the Windows guy who was also driving the LibUSBX initiation, or yes, fork, he said yes, or wrote to celebrate the two-year anniversary of the last public release of LibUSB, which is the 108 that Daniel made about half a year before he made me maintainer. They announced LibUSBX, and they think this two-year anniversary is proof enough that LibUSB is a dead project, so he's an excellent rhetoric for this person, he's very, very good at writing English, he's an Irishman, and if you look in the LibUSBX, so they started their own mailing list on Sarseforge, and if you look through the archives, you can see that they worked quite a lot on this text before they sent it out to the mailing list. They had multiple iterations going through, and they really, if you compare the versions, you can see how they really tweaked words and sentences here and there to create a very specific effect, or to send a very specific message. It was very well engineered propaganda. They're saying that LibUSB is a dead project in their fork announcement, even though we've been working on it constantly for two years, so there are lots of commits in the repository, public commits, for anyone to see, there wasn't a release. A lot of users, they said, we want a release, we want a release, we want a release, and I wanted a release too, but like I said, I didn't really want to release just anything, I wanted this Windows support to be all right and not to have too many problems, it needed to work all right, and I also wanted to fix as many bugs as possible in the previous version. They also mentioned here, if you're interested in submitting patches to the LibUSBX fork, please be aware that LibUSBX will soon have Garrett and Jenkins operational, so this was in 2012, there never was any Garrett or Jenkins for LibUSBX, by the way, but oh well. Anyway, so on April the 20th, some 20 hours code sprinting later, I put out the 109 release, and it included some 260 commits since the previous version, so there was quite a lot of work that went into this 109 release over these two years since the last one. I think I spent about 2,000 hours roughly over two years, so that's about a half-time job, 20 hours a week on this project, but it wasn't enough, and I didn't do it, I didn't spend it on exactly the right things. There was this people pushing for a release, and I didn't deliver on the release, and I'll get back to that. So in May, just a few weeks later, Red Hat's employee who was also involved in LibUSBX, he had a blog which he was posting in, and he hadn't posted about LibUSBX or LibUSB before, but he posted this very short blog post, just two lines of text, or one and a half line of text about how he had now switched the package for providing the LibUSB functionality in Fedora over to use the fork, LibUSBX, and it took maybe a week, maybe even less just a few days for all the major distributions to follow. They had suddenly also all of them switched to using LibUSBX, and not a single one got in touch with me to sort of ask, okay, what's going on? I guess nobody looked in the repository and saw that all the commits were, or not all the commits, but that many commits were also by or involving me, the existing maintainer of the project, I'm, yes, I don't know. I guess it was enough that Fedora switched. I got to try to get in touch with some of them. I sent the Arc Linux maintainers an email with a direct email, no mailing list, telling them how I felt about this situation and how it really broke my heart that nobody actually got in touch with me, or as it seemed to me, bothered to look into some facts about this whole situation. And by accident, I later learned that, or I got a link, and this email was up on Pastebin. So thanks for that. I don't know. I found that strange. Oh, well, anyway. So there was this fork, and there was still the original project, and we were working alongside. Most of the people, they went to the fork because they saw there was a potential for new and exciting things. But the one guy who had always been working on the Mac OS support, he was sticking with the original project. And he was also very outspoken against this hostile fork. He said he disagreed very strongly with some of the technical decisions they made, and also how they had conducted themselves. And I was super happy to have his support. No bugs. All right. So a little bit later, or sometime later, I don't know exactly because my network connection dropped out when I was trying to research this. Nathan unexpectedly removes me from the Libusby project on Swartzforge. So this was the primary distribution point for the project, and it was where the main list was, and where we published releases. Okay. So he suddenly decides to take over the project, take control over the project. He was working on some changes, and I, again, I felt very strongly that they weren't ready. There were some issues with them. I communicated the issues to him, and he didn't, I'm not sure that he understood exactly what the issues were, and I'm sure that he was frustrated that the development wasn't moving forward. I really am. Still, so the next thing he did was he invited all of the LibusbyX people to join the project. And those two things, they really, really surprised me. I didn't expect that. I did not see that coming. And so they then sort of say that they're Libusby in 2014. The Libusby code, LibusbyX, sorry, code base, that they continue working on their code, and I continue working on my code. And there's this commit in the GitHub Libusby repository, which shows pretty clearly that what they call merging is really just renaming their LibusbyX code base to Libusby. And no, not many people knew that. I spoke with a pretty prominent Linux kernel developer. He didn't notice this either. I mean, he just reads the news like everyone else, right? And in the news it says the project's merged. So mistakes that I made. I made a lot of mistakes. Few of them were more important than the other ones. The major mistake I made, which would have avoided all these problems, was to not release anything, because that is what sort of signals activity in the project to anyone who's not working on the project. You could say that it's only for show, and actually if you do a release and there's nothing really much that has changed, then that's perfectly fine, as long as you sort of send out a heartbeat and make sure that people know that, well, the project is still moving forward, even if there's nothing really happening. Another mistake I made was that I'm Swedish, and I think it's a trait in Scandinavia that we like to seek consensus, and I did that for far too long. I was trying very, very hard to work together with other developers who wanted to do different things than I wanted to do, and who wanted to do things differently than I wanted to do, and I shouldn't have tried to do that for so long. I should have much, much quicker. At some point I finally realized, okay, this is not going to work out. We should go separate ways. But that took me a good year and a half of fighting, as this kernel developer called it, or discussing, as I had the impression that we were doing, but that wasn't how it was perceived outside. The outside, they had a lot of negative negativity come out of these pretty intense discussions, and there's a really good presentation by Donnie Burkholz, where he, among other things, has that one negative experience requires five positive experiences within an open source project, or any voluntary project for a contributor to stay there and to sort of level things out. So for every single negative thing, there are five good that are needed, and that's a lot of positive stuff that is needed to compensate for just a single bad experience. I learned lots of things about how different contributors do, well, what motivates different contributors, and what kind of, that there are different kinds of contributors. I was quite naive when I started maintaining this project, and I was engaged in the project for selfless reasons. I wanted to make a good software library for others to use, for others to benefit from, and for me myself to use as well, but that's not necessarily what drives everyone else, especially if you have corporate contributors. So the red hat guy, for example, he just needed to solve the problem that the paying customer had, and he didn't really care much about the drama, he just needed a working library. All right. Package maintainers in operating system distributions, I learned that they don't maybe don't look so much into the facts. I tried to talk with several of them, they were all super, super busy, and stressed, and overworked, and of course they don't have time to do investigative journalism. I learned about trolls on the internet, I'm not saying that people were trolls, but I was trolled, and I let myself be trolled, and I fed the trolls, and that's not good, because it creates these negative experiences within the project, and that hurts the community, and it hurts me as well. GitHub, that's a funny story, but we're a bit out of time, I'll skip that and go right to what I learned about myself. I want to do things really, really well, and I learned that I, besides the consensus thing, that I tried to do that too much, I learned that I tried to do things too well. I have to work on that. I have to be more open to how other people want to do things, and I have to also not insist that we fix all the problems before we've experienced them. I have to let things go wrong sometimes, and maybe I can, in retrospect, then say I told you so. Thank you.