 Welcome, our next talk will be about the Devian Longtime support team and the speaker is Raphael Herzog. Hello. So yes, today I will speak a bit about Devian Longtime support. I guess most of you already know. Are there some people who have no idea what this is about? Oh, good. I will make my talk in three parts mainly. First, I will present the team how it works. I will also give some facts about how it evolved over the first years. I took some time to collect the statistics. I believe they are rather interesting. I will also speak about the future, but there will be a separate discussion about this, you know, both later this week. And I will show you how to help, because like any other team in Devian, it's open to anyone, and we welcome help. At the end, I will answer your question. So, first part, what is Longtime support about? What was the problem when we created the team? And the choice we made. So, the idea is really simple. We wanted to extend the support period of all Devian releases. So, you know, currently it's basically one year until the next table release comes out. And we wanted to extend this to five years to match at least Ubuntu's offering, which is not our competitor, but for the companies who are making choices, it's one of the important criteria. So, we wanted to do as well. Basically, it's one year after the next table, but since we publish new stable releases every two years, it's roughly three years. A nice side benefit is that users can skip a full release while we don't officially support this upgrade over going from Devian 6 to Devian 8. You can do two disk upgrades at the same time and limiting the downtime to once every five years. By the way, in practice, disk upgrades tend to, on simple server configuration, tends to work rather well even across two releases. So, keeping the distribution secure for five years is a real challenge. It's hard work that not everybody is willing to do. I mean, in Devian, all the work is done by volunteers, and volunteers do the work that you enjoy. Generally, we enjoy working on new features on top of latest release, not really on back-parting patches to code that was written five years ago. The security team has limited resources, so we could not just ask the security team to, hey, so no, do twice the work. But still, they were really interested in the project and wanted to support the idea and really helped to get it bootstrapped. And the security team are slightly larger scope. They support all architectures, which means that you have lots of problems of coordination when security updates do not compile and stuff like that. So what did we do? We restricted the scope by picking the two most popular architectures that most users care about. We have, we have had some demand for ARM architectures, but up to now, we only support AMD 64 and E3866. And we also excluded some package from the security support, either because they are taking too much time. I mean, they have a security issue every two weeks, and well, they're very hard to support. Or, well, or it can also be that upstream is not cooperative enough to be able to support it. This list was basically made by the current security team based on their experience of doing security support. Still, there are, if you look up this list, there are some important restrictions. There's no GDAN, there's no KVM, there's no Rails, there's no browser, there's no IathWiddle and Chromium, so it sucks a bit, but it's a way to get it started. But I think we can do better for Weezy. So basically, there's no virtualization support on the host, only on the West. So yes, the security team helps to bootstrap the LTS team, but it's not the same team. Obviously, there are members of the security team who work also in the LTS team, and they are, well, we work in close cooperation. We have regular contacts, and they watch our mailing lists, et cetera. But the policies are different and the infrastructure is separate, which is a problem that we'll talk to this about later. So we have a dedicated mailing list, devian-lts.devian.org, and a dedicated IAC channel as well. You're welcome to subscribe and to join. So if it's a new team, it means also new people, new members. So where do they come from? The first idea was to get help from people in various companies who are already doing such in-house support. We had contact with Electricity de France, EDF, and we still have, but they were one of the first companies who was pushing for this because they said, well, we're basically doing it already. We can share it with other companies. So the idea was really to pool security support of multiple companies. So we sent a press release asking companies to join, and we had a few response, but I'll come back later to how it evolved. It's not really satisfying. So the other thing we did is that we offered companies to fund the project to bring money and use this money to pay the work of actual devian contributors to do the security updates that we need. So basically we have wiki pages listing all the way companies can help with money. But in practice, most of the wanting to be paid contributors joined together under a single offer managed by Frixion SAEL, which is my own company. So I explain you quickly how this works. So basically, most companies want to... don't want to bother bringing human resources and bring money by long-term support contracts to Frixion. And basically we have a rate. I mean, when you give 85 viewers, you fund one hour of LTS work. This is the current list of sponsors, if you want. Top-level God sponsors, sponsoring more than one day of work per month, etc. So on the other side, we have devian contributors that are doing the work and Frixion is paying them. There is a small difference between the rate to cover administrative costs, because I have to handle all the invoices some customers are using PayPal, which is taking a cut, etc. And while we asked contributors to follow some rules, and among those rules is the requirement to publish a monthly report of the work done on PED time. So they won't get PED until they have published a report. So everybody can know how the money has been spent. So yes, that's the way it works. Currently we have seven devian contributors and about 30 sponsors. So some figures. Who uploaded packages? How did they evolve since July last year, or June last year? How is the funding evolving? I just updated those figures a few days ago, because I used this talk for the first time in Lyon, mini-Debcon in March, but I updated it again. So the number of uploads, it's roughly over one year in one month, since we started in July last year. Over 300 uploads, so it's not so much, but still almost one per day, so it's a significant work. I have given a plate here who paid for the work mostly and who did it on the left. So friction is sponsoring most of... friction sponsors, not friction itself, the sponsor will bring money to friction, doing most of the uploads. Known as a separate category for grouping all maintainers. I mean, there are maintainers who are taking care of their own packages in LTS, in Squeeze right now. Our members of the security team who works also within the LTS team. EDF is Electricity de France. Individuals are debian developers that have listed themselves as members of the LTS team and who did uploads not for their own package, but for packages of other maintainers. Kredativ is a German company that you know, probably. They have a booth here if you want a job. Toshiba Univention catalyst, et cetera. Rather lower figures. On the right, you have people. So the top five people are paid by friction. Raphael is working for EDF. Tish is a member of the security team. Kurt is an Open Estelle maintainer, so he keeps maintaining his own packages. He has a lot of work. Mike is also paid by friction. Christopher is mainly maintaining the debian security support package in Squeeze LTS. And Guien Cong is employed by Toshiba, et cetera, et cetera. Christoph Berg is employed by Kredativ doing PostgreSQL maintenance. So how did it evolve over the year? Again, it's by affiliation. So the big blue part is people, paid contributors. And you don't see it very well, but the part about maintainers is this one. It tends to do better over the month because this year we started to contact maintainers. Every time that we have a new plot coming up, we ask them first, do you want to handle it yourself or do you want to? And so it's slightly increased. But as you see, the contribution of other companies has not really increased over time and rather disappeared. It's unfortunate, but it looks like paid contributors are more proactive than others. So in particular with EDF, they do their work but with some lag and we're faster. So they just reuse what we have done. And I wanted to talk with Raphael. I tried to see how we can do better for this, but how did the sponsorship level evolve? So we have a steady increase, that's rather nice. It's not a huge amount, but it's significant because we fund almost 80 hours per month. It's close to our first goal. We wanted that amount to be able to sustain ourselves. If you look at the way it's the various sponsors, we have a few big ones and possibly one very big. I can't give the name officially yet, so I won't. It will be a big jump in terms of the graph. A few gold and many small sponsors. It's really what I'm after. I don't want to be dependent too much on one big sponsor. I really prefer many sponsors who are doing small donations but donations that are sustainable year after year because we're not here for one year or two. We want to do that over the long term. So you have some figures about how many hours have been funded since the start. By the way, if you have any questions, feel free to interrupt me. I can take them at any time. That's it for the evolution. Now, the future. What do we expect for the future? First, keep doing what we have been up to. Keep supporting the current set of packages. For easy long-term support, we would really like to have more supported packages. Our browser would be nice for desktop deployment and really virtualization support is important for many companies. We should be able to support something here. Also, we want to avoid some pitfalls that we add with squeeze LTS. As you know, LTS users are currently required to add a separate source list entry called squeeze-LTS repository. The security.debian.org squeeze repository is unused. It should be possible for the LTS team to continue using the same repository as the security team when it no longer uses it. This will be the topic of both next week. Even the date is on Tuesday at 18 in the first floor, Amsterdam. What's the problem with supporting the current set of packages? For example, we have MySQL right now in squeeze LTS. It's version 5.1, which is no longer supported by Oracle. We don't even know if it's affected by security issue because Oracle doesn't give info not to release it. This is a problem. We should consider switching to a newer version. Newer versions involve library transitions, which is not really nice in long-term support releases. There is some work to do here if we want to do something serious. We have other packages which are similar. From time to time, we take newer versions. We did that for Wireshark, for example. This is what I said before. The limited scope sucks. We want to be able to support more packages, and we need more support for this. Speaking of Wireshark, we switched to the easy version. It's solved. Yes, exactly. That's what I just said. It used to be a problem. That's the part I did not update since March. Again, what I just said before, so I skip it. Additionally, the problem with the separate repository is that there is a propagation delay to the mirrors that we don't have with security.dependent.org. I will speak a bit of how the team works. Basically, the first step is to aging new security issues. We have a list of CVEs that comes in, and they are added to a text file, data-cv-slash-cv-slash-list. Someone dispatches a thousand source packages, and then someone else reviews, usually in our case, it's someone else, reviews the status in each release. Members of the LTS team review the status in squeeze, and the members of the security team review the status in Weezy and Jesse. And then we decide what we must do with the package. Depending on the analysis, either we say, well, we need to prepare an update, so we add it to data-dla-needed.txt, and someone will have to take care of preparing the updates. We say the issue does not apply, it does not affect the conversion, or we ignore it because the package is unsupported, or because the issue is really minor, not severe enough. Sometimes it can be that this issue is already fixed in Debian due to some maintainer choices. When we do this classification, we contact the maintainers to keep him in the loop and offer him the possibility to help us. Here's what such the text file looks like. So the bold line is the lines that we are adding when we have decided what we are doing with the packages. This is all in the subversion repository on alios. Well, then there's someone who has to prepare the security update. Basically, looking for a patch, it's often useful to be able to look up at Red Hat, or Ubuntu, or upstream. Best case is upstream because there are nice upstream who are providing patches also for older versions. Usually there are already patches available, not always for the good version, so sometimes we have to backport it. So if we prepare a new plot with this plus-deb6ux suffix that we are using now for security updates, stable updates, well, this is a rather known territory for package maintainers. This is the hardest part sometimes, testing the update and making sure that the issue is fixed. When tools have tests to it, it's nice, but sometimes we have to set up it yourself. Sometimes it's too hard, so we tend to add a safety measure to the mailing list and say, okay, I've done my best, but please double-check. It doesn't happen often that we have tests, but some LTS users are willing to test it beforehand, and it would be really nice if you had more of those. And if you don't get any negative feedback, then we'll upload it. Then you have to send an announce. We call that DLA for Debian LTS advisory, and we have a little script, Bing and DLA that generates the template mail, and you have to add a little bit of description and basically send it to Debian LTS announce with a GPG signature of someone in the DD or DM key ring. The process is rather open to all maintainers, so even the right rights to the secure testing repository where we have this data, CVList file is writable by all Debian developers on alias. So really the entrance barrier is rather low for Debian maintenance and Debian developers. And yes, there's a file which is automatically updated when you generate this template and which will update the tracker on the web pages because all this data is nicely browsable on security-tracker.debian.org. And it's all linked from the package tracker. That's it, so if you have a few questions, I'm here to answer you. Have you considered using the old client library and upgrading the server so that you have the same API for clients or the same API for clients and a newer server that actually gets security patches? In my head, yes, but we did not look further yet. I really wanted to use the buff that is upcoming to discuss it because as I said, the amount of funding we have allows us right now to keep up with security update, but we do not have many pair cycles for dealing with such things. But it's slowly coming up. As I said, there is potentially a new platinum sponsor, so we will have a few hours free to look into this, and I think it's probably the best solution. But I don't know if it works in practice. I have not tried it. And since LTS, the end of life of squeeze happens in February, it's not too far away, so it's possibly a nice solution but creating to a new version is harder. I have a question about Frixian, where you have enough contributors that you could do more work, but you don't have the funding, or is it like you would also need more developers? A bit of both, actually. I mean, the web page has always been very clear on the rules that it means any Debian developers who has made secret uploads on its own already can apply and join the program and be funded. Fortunately, I did not have too many because it would have been hard, but it has been steadily growing over the months. Last year there was only Thorsten, Olga and me, but in the meantime we got Ben Etchings, which is helping us on the kernel, and a few more. I'm always looking, trying to make sure that we don't give too much money to a single person. Because, well, it's Debian, there has been dunk-tunk in the past. I've been involved, I know it. But I don't want anyone to have his life depend on LTS work, really. And I don't want the opposite way, also LTS would depend on a single person that can go away. So I try to limit it to, well, maximum 20, 30 hours per month for each person. And right now we're well below that, so it's good. By the way, most Debian developers are currently European, so the fact that everything is Euro-based is fine. But we have one developer who is American and it's no problem to pay in dollars. So the amount will vary with the rates, but it's not a problem. And even more, I'm surprised to have no contributions in countries where the rate of labor is way below. I mean, if we have so-so American contributors or African, I don't want to stigmatize anyone, but if there are people who could live for much more, I mean, if they would be paid 10 hours, they have made them months, they can still join. It's not a problem. It's not a high rate because we want only Europeans. It's because we want to allow everybody and if they can be paid for 10 hours by friction and then spend the rest of the month doing free work on Debian, the better. So, I mean, if you want to join the program, get in touch. There are details on the web pages and the more people found, the better. How do you or other developers motivate yourself to do this kind of LTS work because obviously it's important for companies to have stable releases, but it also, to me at least, doesn't seem very exciting from a technology standpoint. Do you think it's exciting or not exciting? I'm not sure I understood. Not exciting. Okay, I agree. It's not exciting. But a bit of both. I want Debian to succeed and I know that LTS support is important for Debian to succeed, so I want to make sure the project is working well and alive, but clearly the work I'm doing, I'm doing it because, let's say, the security support work, providing security update, back-parting code is for money and because I need to live. But organizing the LTS project is something I do for free because I want it for Debian because Debian needs it. Hi. I have a question. Who handles the public relations, the marketing stuff of the Debian long-term support? Do you do it yourself or do you have some professional help? I do not have any professional help. Basically, the way we could do much better, I mean, in number of sponsors, I mean, there are almost no US-based sponsors and there are lots of US companies that could help. But it's true. I'm fighting between my time, spending free time with Debian, doing LTS with Debian, and I contact a few companies that people suggest me to contact, but I'm not doing much things by myself. So you would benefit a lot from some professional help in the marketing department or the public relations? Yes, but I mean, I don't have money to fund this or not much. I mean, the 10 euro difference is for administrative costs. Maybe a bit for this kind of stuff, but it doesn't like a whole lot for this. Okay, I just wanted to answer the previous question about finding motivation for providing patches for LTS packages. And in my view, it's not very different from providing security patches for stable. Sometimes it's a bit harder because the software is a bit older, but we overcome this situation for Viershark, for example, by updating Viershark because it was quite impossible to backward all the security fixes. And I think it's part of the maintenance. If you maintain a project, you should maintain the security issues in stable and if you care a bit more and if you have a little more time, you can care about the package in LTS as well. I did it for Viershark for free because it's easy for me. It's great. Thank you. I want to point out that it's a good thing that LTS and security is different because we can have different policy. Like we said, I'm putting a new version. It's something we can do when it doesn't break too much. I mean, the common line interface has not changed so badly between both versions, so it's worth possible. It's not possible for all projects, but we're not so strict on new version versions. If you look at what others have been doing, I mean, in Red Hat, they do import new version version quite often, in fact. Even on libraries like NSS and... So, did the like or LTS support then support was an issue with the companies you are working on who are sponsoring Fritzan? I'm not sure I understood. So, the like or... Ah, Brozer, yes. Yeah. Then support in the LTS version. Was that an issue with the companies you work with who are sponsoring your work? Well, yes, obviously. Brozer not so much. I mean, most sponsors are hosters and stuff like that, but they were annoyed by the lack of QEMU or XAN support. They did upgrade their host machine, but not the Quest machine on their other customers. Obviously, that would be the first priority to fix. Brozer is nice, but it's also one of those cases where we just need to back to integrate newer version versions. So, it's a matter of someone doing it. And I was hoping for Raphael J. Serre to do it because he does it for Electricité de France. And I'll catch him and come to him. I know him quite well. He's working in Leonier, me, not far away, but... I mean, sponsors for doing the LTSV work is a signal of the need for LTS versions. But do you have additional statistics on the usage of LTS? Not really. And since LTS is not open on security mirrors, but on all mirrors, we don't have access to all the statistics. But I know from mails, thanks mails and stuff like that that it's really appreciated and used quite a lot, even from end users. I mean, there are users who like the antiquated GNOME stuff. I'm fine with GNOME 3, by the way. Yes, well, it's really widely used and widely appreciated. Thanks for your attention. There will be a break until five. The next talk will only start at five. So, have fun.