 Okay, welcome to the second session of today. This one's two years of paid work and contributions to the Debian Long Terms supports project. Rafael will be talking about his experiences, so enjoy. Thank you. So I'm going to present the Debian Long Terms support project. I did a similar presentation last year, so if you already saw it too, there is a first part that is quite common, but I wanted to put some emphasis on what we did to avoid problems related to our usage of money to pay contributors, because this is some of a controversial topic, or it used to be at least, and I want to share my experience here. So, at the start I will go quickly through the slides because I have many of them, and I will slow it down at the end to discuss, to show the new content and the changes since last year. So what is LTS about? It's about providing five years of security support to all Debian releases, thus allowing users to skip a release. So basically, if you're using Debian 7 from start to end, you can then switch directly to Debian 9. What were the challenges? Well, five years of security support is a lot of work. It means multiple releases, maintaining multiple releases at the same time, and our security team of limited results aims to support all architectures, so it's difficult for them to do it. In order to achieve this, we restricted the perimeter. The thing is we only support a subset of architectures, and we excluded some problematic packages. Here's a link, there's a list, and a link with the full list below. The slides should be available on the top page soon. We decided to create a long-term support team which is separate from the security team. Also, there are a few members apart of both teams. That allowed us to have different policies and different infrastructure. We wanted to make it possible for companies to help us, because while companies are the first users of long-term support, once they deploy a server, they want to continue to use it as long as possible, and still keeping it secure, obviously, because many companies were already keeping servers longer than usual, but they were not covered security-wise. So what we did is that we basically announced it and said whoever wants to help us can help us. Here's a list of people that you can pay, but in practice, to help us doing security updates, but in practice, most of the contributors who wanted to be paid joined together and created a common offer that is managed by my own company, Friction. So what do we do at Friction? We collect money from sponsors. We have almost 40 sponsors by now. Then we convert this money in work hours and we dispatch those work hours over all the contributors which are participating. At the end of the month, each contributor is supposed to provide a report of what he did, and based on this report, we pay his invoice. There's a difference in the rate that we use to pay contributors and the rate that sponsors pay that covers the administrative cost and the management cost of paid contributors. A few words about the workflow of the LTS team. Basically, the three-edge work is done together with the security team in the security tracker which is common to both teams. New issues are added to a text file. There are dispatched on source packages and then each team reviews the status of each CVE for each release. And we classify the issue in different cases. So one possibility is that, well, the issue is real, we need to fix it. So we then add an entry to another file which is called dla-needed.txt. Another possibility is that, well, the issue does not apply to whatever we have in this release. So it's not affected, the tag that you put in the file. A third possibility is that, well, the software is no longer supported. We decided that we could not support it so we tag it end of life. Or maybe the issue is not severe enough, not worth fixing. It would take too much time for too little benefit and then it's take no DSA. Well, DSA is for Debian Security Announcement or advisory, but we have another word for the LTS team which is DLA, but we still use the same tag, obviously, because the tracker is common. And the last possibility is that the bug was real but it was already fixed in a former version. So basically we can ignore it. And we, in this case, we put the fixed version as the data. We always try to keep the maintainers in the loop because they can jump in and do the work by themselves. This is a short extract of this Data-CVE list file. So you can see it's a no-dantry for squeeze, no longer for width enough by now, but well, this is not very interesting. So when you decide to, that the issue was fixing, you have to find a patch, you have to backport it if required, and you prepare a new upload with a plus step seven U1 suffix, for example. Well, what's the usual workflow that packages should know? Then you test, update, and you upload it. Testing is not always trivial. It's actually on one of the hardest parts. In the security workflow, because we modify many packages that we're not familiar with, and not all packages have depth eight test suites, and sometimes the setup of the software is rather hard. Takes a lot of time. If you are unsure, you can always seek an external review for the members on the team by sending a depth leaf on the list. But if you're pretty sure, then you can directly upload to with the security. Once the upload has been made, you have to announce it, and we have a small script, again, DLA, which generates textual reports that you can then send with not to whatever mailer you use. In this process, there's some data recorded, which will update the security tracker, indicating that this issue is fixed in this version of the package. Some statistics about the team. I updated those to cover now two years. So the number of uploads that we made over two years is more than 500. And I try to classify those by affiliation, that is, well, which company has sort of sponsored the work, or which group, or team, or also by an individual contributor. So the pet contributors really have done most, as you see, because Friction is the company which collects the sponsorship, so it's actually many sponsors, and no affiliation means, well, package maintainers made the upload by themselves, so they are the second largest contributor, if you want. We have many people carrying, many deviant maintainers carrying about their own packages in LTS, and that's good. And the third largest is the security team itself. We have two or three members of the security team which who are working on the LTS team as well from time to time. Sometimes because the fix is really the same in JSC and WISI, so it's not a lot of work to do both at the same time. And then we have a few other companies. Electricity de France is doing some internal LTS supports even before LTS existed, so they contributed a few. But basically, the pet contributors are nowadays faster than their internal support, so they're not doing much anymore. Creditive is actually maintaining PostgreSQL through Christophe Berg. And then, well, we have a few other sponsors, but most of them are actually no longer active nowadays. Toshiba used to have someone helping us, but they are now becoming a platinum sponsor instead, and so they're no longer any individual working directly. Univansion was because Moritz Milanoff was employed by them, but he switched jobs, so he's no longer doing it. If you look in the contributor list, the eight first are people paid by Frickson. Tish is a member of the security team. Guido is paid as well. Kurt is the maintainer of OpenSSL, is doing the replot of his own package. So clearly, this is dominated by pet contributors. This is a chart trying to show the number of LTS replots down over time. The blue part is the pet contributors. The somewhat dark part here is the maintainers taking care of their own packages. But you can see that over time, somehow the pet contributors are becoming really almost the only workforce, or the largest workforce. The small drop here is just because, well, from February to May, squeeze LTS was no longer supported, and Weezy was still supported by the security team, so the pet contributors actually helped the security team, but it's not keeping track. My statistic, we're not keeping track of this. Here you can see the evolution of the number of sponsored hours. So it has been slowly growing from the start up to now. We are now at 135 hours per month, and we're likely to get another platinum sponsor soon. So we're relatively close of our goal of finding the equivalent of a full-time position. The red line is the number of pet contributors, and you can see that it has, the more hours we add, the more pet contributors came on board. Also give some figures about the various levels of sponsorship that's relatively balanced. So this is the start of the new content, really, because the last slides were updated from last year, but now we're going to speak about something more interesting. So we recently switched from squeeze to Weezy as the long-term support version. We made some changes at this point. Instead of using squeeze LTS, which was a separate repository, we're now using the Weezy security repository, which is Austin on security.debian.org. So that means users do not have to change their source.list file, and they get their security updates like usual. It means some changes for contributors. Well, we have to upload to another repository, and there are some downsides due to not so good integration of security.debian.org with other tools, like package.debian.org, which is not showing them, or not in a timely way. With the switch to Weezy, we also tried to support more packages, mainly virtualization related. So Xen, QMU, and so Firefox, which is a desktop browser. As you know, iZdov, LibAvy, LibVit, all those were not supported in squeeze, and are not supported in Weezy, or at least to some extent. This is all made possible because, well, as I said before, the amount of sponsorship growth over time, so we have the means to do this. We also added two new architectures a bit lately, but we're not supporting ARML and ARMHF. In fact, it came as a request for new sponsors, that joined lately. Bonsor is Plathome, and they're doing ARM board. Their customers wanted to keep using Weezy, so they are now a gold sponsor, I think. Well, FTP masters and buildy maintainers have been receptive to our quest, and in the last day before Weezy, the test started while we added those architectures. We're also trying to work with external partners, because the skill set in the pet contributors is somewhat limited, and there are some package which are really hard to support when they are no longer supported by their upstream communities, and so we reached out to a few companies and individuals. So we have two such cases currently. One is XEN, and we picked Creditive to maintain a sort of upstream XEN branch that we can use in Weezy. And so far, it's Bastion Blanc who has been doing the work. And the other case is Lee Bevy with Diego Burun. I don't have any feedback yet here because he did not do anything yet. Well, this is really an experiment. We don't have a good workflow yet to work with external partners. We're trying things out right now. So how do we try to avoid money-related problems? There are several points that I will detail tomorrow. One is about transparency, both towards the external community, I mean, they are not large, and also in general, I mean between the set of pet contributors. And we have clear rules to join the set of pet contributors. We have rules to allocate hours, and we have rules related to how we communicate. And I made sure that there was a way to complain about what we do so that if people are NLP, they can do, they can tell us, tell us. So external transparency. Friction, that is me. I'm doing monthly reports of how we used the hours that we were sponsored, that we're sponsored. So I document the hours given to each contributors, and I put link to their respective report. And I try some time to do some high-level analysis of what happened in the LTS world and how the funding involved and stuff like that. The report includes a list of sponsors. It's a way to bring them some visibility because the report is posted on Planet Debian. Each pet contributor must do a report of its own and they must document what they did and how many hours they worked. Some do it on their blog, which might be syndicated or not on Planet Debian, and others are doing it with a simple mail to the Debian LTS mailing list. Internal transparency. So the pet contributors have access to a private guide repository where I document which sponsor paid and how much, obviously, and how it's converted into work hours. So we're using Ledger, the common light thingy, and it works with a text file, which looks like this. This is an entry, it's a double entry accounting system. So basically, we record on one side, we have a set of funded hours, and we dispatch them to, well, this will give us two hours in July and a ghost, et cetera. Because when you pay an amount, well, it depends if you pay a yearly amount, we dispatch the hours over the full year, but if you subscribe in a quarterly way, then we obviously dispatch the hours only over the quarter. Every first of the month, I dispatch again all the available hours to individual contributors. This is what it looks like for the entry, for the last June entry. Well, so we have separate accounts for each contributors. Each contributor can set a limit on the number of hours that he wants to be assigned, which explains why some have different amounts. Lido, for example, always, never wants more than eight hours, so as we add more than eight hours per contributor, he has only eight and others have more. And at the end of the month, the contributors must add its own entry, saying, well, okay, I had so many hours, but well, I did them all, and I want to be paid for them, or I could not do them all, in which case, it keeps a few hours for the month after. And he must document here the link where we can find these reports. We have open rules to join, that has been documented on the friction.com website since the start. Basically, you have to be involved in Debian already, as a developer or the maintainer. You must have some prior experience with security updates. That said, it can be on your own package, and it doesn't mean, it doesn't have to be a very long experience, but you must be familiar with the process. You must have good programming skills, because well, you will have to deal with packages that you don't know in programming languages, in multiple programming languages, so the more you know, the better. And if you're really not able to work with programming languages that you don't know, then you're not going to be very useful in this project. You must be able to emit invoice to friction. This is mostly an administrative reason. And you must accept some rules. As I said, you have access to a private git repository with some data about our customers or sponsors. And so you must respect the privacy of this. You must commit to do a public monthly report. You must follow the Debian code of conduct, to which I add you have the obligation to respond to queries from all the Debian contributors. I don't want people, bad people to be non-responsive. I mean, it's not because you are bad that you are not part of a community that you can allow yourself to not respond or to re-ignore other people. And last point, it's something of, you must do your best to do as well or better than the Debian security team is already doing or has already been doing with the release that we maintain. How do we allocate hours? So the basic rules is that we split evenly between all contributors which are registered. But as I said, you can define a maximal number of hours. So then in which case we will have differences. If we have too many active contributors, this never happens so far, then we would organize a rotation so that we do not assign less than eight hours per contributor because while a security date can be involved and you must be able to do it from start to finish. And if we have only three hours per month, it doesn't make much sense. And it would also be an administrative headache for us to handle too many very small invoices. As I showed before, the number of contributors has grown together with the number of paid hours, which is a good thing. This means that, well, each paid contributor only has a limited number of paid hours. I do not want to get into the situation where we would have one paid contributor relying only on LTS for his life or for his income. And it also means that we would have more redundancy. If you, if only one contributor is doing all the work, then if he goes ill, you're in a problematic situation. But here we can smooth things up. And it happened quite often that, well, for various reasons, one of the paid contributors did not manage to do his hours, so we have been able to really dispatch them to others quite easily. So we have also clear communication rules. I think I already covered those, but obviously we want to respect the Deben Code of Conduct. And well, it's one of those cases where the consequences are rather clear and you have something real to lose. Also, yes, we always make sure to involve the current package maintainers when we handle LTS updates. It can be tempting to do everything ourselves immediately because, well, we have paid, we have some time, but it's important to not ignore the usual process and offer the maintainer to do it by himself. Because, well, it means we can spend more time on other neglected package where we would not have the resource and also because, well, it's a more inclusive way to work. And as I said before, you have an obligation when you are a paid contributor to respond to queries from other Deben contributors. I mean, lack of communication is hardly acceptable for volunteers, but for paid contributors, it's clearly not acceptable. So that's why I added this rule. Last point, the friction.com webpage has a Q and A section if you want. And there, I document what you should do if you have a problem. Basically, it's come talk to me and tell me what is not good either in the work of some paid maintainer or in the process, but at least there's a clear point of contact. Because we want to be recognized for the high-quality work that we're doing and we don't want to be badly seen. I mean, when you do paid work, it must be something good. We had no complaints so far, at least no official complaints. I don't know if any have heard of any and feel free to come talk to me. But somewhat preventively, I do complain from time to time to some of the paid contributors. Not in an aggressive manner, in a constructive manner, but when I have the feeling that's one of the contributors not up to par in terms of quality of his work or in terms of commitment. I will say what I have to say. For example, one common problem is that, well, most of the paid contributors are self-employed people and they have customers that call them, they have urgencies, and they take LTS as a side project that, well, maybe I have time, it would be nice if I could. But at the end of the month they say, oh, well, I did nothing. That's bad because really if you request work hours, you take a commitment, okay, this month I will spend 10 or 15 hours because I care about LTS and because I know that there are companies relying on my work, which need timely, secretive days. So it's a bit hard for people, but you have to organize yourself to be able to spend multiple time per month, a few hours because it's not like a project where you can sit down on the morning and have it finished by the evening. Usually, secretive dates take some interaction with others, so you have to be able to spend a little bit of time over multiple days in the months. And actually some bad contributors realized that they were not keeping up to the expectations and they stepped down by themselves after a while. So lessons learned. I believe it's possible to pay debion contributors without disrupting the entire community. I think, well, the experience shows that LTS is now a project which is two years old. We have been paying quite a few people. And the world did not end yet. But care must be taken at many levels. As I said, you must work transparently and in an inclusive way. You must avoid that someone gets locked in a bad position. You must have fair criteria to use the money or at least a fair chance of being bad. And you must be aware that it will have consequences. When there are volunteers working in a field and if you inject bad contributors there, there might be volunteers who will use this opportunity to change their own priorities. Because if they see that someone else is already taking care of this, they will find something. Another neglected area where they can do a difference, for instance. So, that's it. I'm here to answer your question and I invite you also to join my boss this afternoon, two o'clock, about using debion money to fund debion projects. Because I built this presentation as an introduction to the boss, showing that it's possible to use money and are there other interesting ways to use money to fund debion projects in debion? I believe so and I would like to discuss it with all of you. That's it. Thank you. By the way, can the bad contributors stand up so that others can identify them and join? There are a few of them. Okay, thank you guys. If you have questions, you can come see me or there, as well, if you fear me. And if you have questions now, I'm here. Hi, thanks for the presentation. Could you speak to the bus factor of LTS in the sense of if for some reason you had to disappear? Oh, yeah, what would happen to the project? I mean, have you thought about that? Not really, no. I'm still quite young, even if I'm getting older. I don't have any answer for this. There's my wife, which is also involved in the company. So if something happens to me, the company will be managed by her. But I don't know how it would look like afterwards. But probably one of good reasons to integrate it back into Debian. Yeah. But... So regarding new contributors, do you expect people to come towards you or do you ask people if they would be interested? For example, you know people you think would do a good job or how does that usually work? I've done both. At the start, well, it started with a simple mail to Debian LTS. So there were also Debian people interested who joined. But over time, I published it more and more widely in order to get more people. Once applied to Debian and once on Debian jobs. So I fully... It's documented perfectly, so I fully expect people to come to me by themselves. But since I had not enough people coming by themselves, I voluntarily advertised it more broadly. And I made a few requests here and here when I knew of people who were self-employed and it would be a good fit. It happened. I also once asked Antonio specifically for someone of South America because, well, most paid contributors are European right now, but there's no reason for this. And I wanted to get a bit more diversity in terms of time zone because for security dates, it can be interesting and also just general diversity. Okay, so also related to the bus factor question, those rules, did you come up with them on your own or did you seek for input when you wrote them down or how did you? Yeah. It's mostly my own, but while we were three or four of us at the start and we discussed them together, so I've had some experience with funding and was problematic within the community because I was involved in downtown at that time and I did not want to make the same mistakes. So I took some precaution, but yeah, I believe it's mostly my own, but if Guido Agnottia wants to. So, this is also about your rules of conduct. You, luckily so far you haven't had to get rid of someone who didn't want to leave, but do you have already a prescribed transparent rules that you will follow if maybe some contributor keeps underperforming but they won't leave? Do you have a process that you can follow and say, well, this is why we can ask you to leave for these reasons, you can improve by this and if you don't do it with that, something like that, do you have anything like that? Yeah, fortunately I did not have to do this yet, but I believe that if I were in this situation I would probably seek sort of confirmation among other pet contributors because we're really working like a team. I mean, I usually make propositions, but I try really to keep an administrative role, mainly and not fully, even so I'm the decision maker in the administrative way, I try to delegate the decision. And even when we looked for external partners, I did not do that work myself. I asked a few of the pet contributors to look up. And for example, Guido contacted the then community manager who then directed us to two or three companies that we contacted and then we decided together which one to pick. So in other words, it's not something, well I guess I'm trying to see how this would compare to a normal work culture because in a company if someone has to be fired or whatever, you have a process but it's quite internal. You know, it's not transparent necessarily where it's here. Transparency is a goal but at the same time getting rid of someone is a very sensitive thing. So I was wondering if you had any thoughts about how to handle this in a manner that looks transparent enough but kind of isn't necessarily public or embarrassing for anyone? Well, it would not be public. We have internal alias that we use to communicate and if any discussion needs to happen, it will happen on this alias, not public. And since we are only, the relationship between the pet contributor and friction is only, well invoice and I pay invoice so there's no formal contract and there's no fee to be paid if you breach the contract or whatever. So we can decide rather quickly if we have to stop working together without any consequences at least on our side. Technically this is not Debian money and it's not people paid by Debian. So I suspect there might be an impact on when you look for external people to contribute money because they contribute money for a company that happens to pay people that happen to do work on Debian. Have you considered having this be a TO or even having the fundraising part being Debian and Debian being paying contributors or do you think it would have an effect? I mean it's two questions basically. It's A, does it have an impact on fetching money and B, should the money be Debian money actually? Well, it's true, it's a multiple level question but you can find people with both sides of the principle. I mean I got some criticism because well it's outside and we're using the Debian trademark in some way but this is not, well semi-officially endorsed because well when we started this it has been published in a press release that was at the time approved by Luca. So while there's no former contractor telling us you have the right to use the Debian trademark in this way I made a request to the trademark team, said okay, you're only referring to something related to Debian. It's not, it's a user just authorized and not forbidden so it's sort of okay. But it's true that in the age of sponsors we clearly, we clearly put the Debian trademark forward. I mean it's Debian contributors who are paid to work on Debian. So it's really Debian related and most of the sponsors are fine with this but a few of them asked questions anyway and for example, for instance Gandhi was a sponsor the first year and not the second year because well they prefer to give money directly to SPI, Debian or whatever and not through a company. I don't care either way. What I care is that LTS gets done and so if a trusted organization wants to take this over and find with it, if we want to continue to do it externally or so and we have people who believe that it's best down outside with our clear trademark agreement and there are people who prefer, would prefer to do it inside. That would be a topic for this afternoon buff I guess. Okay, we have time for one more question. Just a very short one actually. Are there any non-paid volunteers working on LTS and if not, do you think that's a problem? Are there non-paid contributors women? Yeah, are there non-paid contributors? I mean, also people who do not work on their own packages but work on LTS as a whole but are not paid for it. Are there any such volunteers? And if not, do you think that is a problem? There are a few but at least listed on the Teams page. They were really interested at least at the start but they are no longer so active. I'm not sure if it's a problem. It might be a problem in the sense that we have paid contributors who are not always deeply interested in the generic LTS policies and when you ask questions on the LTS list, so what should we do here in terms of support of this or this? Well, paid contributors do not always have an opinion whereas volunteers tend to have an opinion because they are really interested. So that would be the main downside. The fact that they are not doing secretive data on their own is not so much of a problem but the fact that we have few people, we would say at the management level who are interested in policies of the LTS team here, it's a bit more of a problem. Cool, well, thanks a lot. And thank you for the question.