 funding free software development is very important and as a developer I would dream to do this during 100% of my time so first time I heard about Raphael Herzog he was crowdfunding his Debian Handbook the translation in English so he's continuing this kind of work with a long-time support Debian funding and release so he will talk about that thank you Raphael thank you hello everybody so I'm here today because while I've been involved in the Debian long-term support project since the start it's been I think five years we started in June 2014 with long-term support of Debian 6 squeeze so we'll go through five years of history quickly and discuss how it went so I guess most of you know it already but the Debian LTS project is about extending the security support of Debian stable release from three years to five years three years is roughly what Debian security team the main security team provides and we're extending it for two years more which allows basically to skip a release if you're running servers for five years always how did this project start really started by a few Debian developers who were doing internal long-term support for a few key packages that they that were important in the companies in the use case and they decided to team up together to share the work and to well make it available to everybody so at that time we made a public call for others to join because we were expecting that other companies were doing the same and they could let their employee work on the project but at the same time we wanted to offer the possibility to contribute with money and not only with employee time so well one option was to have many Debian developers come and say hey I'm here you can hire me to work on this but it's not very practical so I use the fact that I had at my own company to create an offer where many Debian developers joined behind me to collect the money from sponsors and pay Debian developers yeah so that's a bit on the wiki page there are everybody who accepts money to be paid on the LTS but mainly you will see we only find people who are working as part of the team that we created on the friction so friction is really an intermediary in this here's you have a big list of sponsors and the friction is making invoices to them and we collect the money and we dispatch transfer this into work hours and we dispatch those work hours to Debian contributors every month at the end of the month each contributor is expected to provide a report we check the report and publish this on our website to make it transparent so over five years there have been a few changes one of them is that we added more architectures at the start it was only limited to E386 and IMD64 and we added ARML and ARMHF for stretch we will likely add also ARM64 at the start with the limited funding we supported only a subset of the packages the most popular that were used by our sponsors but with time we managed to have enough funding to support almost all packages some of those packages are quite hard to support so for example Xen we work with Creditive who has killed people knowing Xen to maintain the backports that we need so those are not Debian developers it's also Debian developers mainly which are working at Creditive but they are not part of the usual Debian contributors that I manage it's really an invoice to another company as I said at the start the idea was to pull up resource and human resource from many companies but in fact this quickly dried because well the people the Debian contributors that Frichian was paying was paying tend to be more reactive than employees who did that part-time and well they just add nothing left to do when they for the package that they were interested in so mainly nowadays almost all the important contributors to Debian LTS are beds through the Frichian sponsorship also at the start we used to ask politely package maintainer to try to provide updates for us if they had some time we no longer ask it explicitly they can still do it but we no longer ask for it because we have enough resource to handle most updates there are a few exceptions post-greSQL is one of them because the main Debian package is maintained by a Debian developer which is with a Creditive employee so they also maintain the post-greSQL in the LTS suites at the every time that LTS ends which is no longer suppose there are a few company who comes up and say hi why did it does it stop so early I need more usually they don't they come two weeks before and it's too late to organize anything but they made the mistake once and so for the second LTS they came earlier and I also mentioned the possibility earlier in a public way so we created an extended LTS for with the so it works a bit differently because there are far fewer sponsors and so we want to support all their packages but only for a limited set of architecture because they're usually not using ARM device but they don't pay what they want they have to pay what we ask them to do because we want to have enough time to support all their packages and we split that among all the sponsors but some sponsors will want maybe six months more just the time to migrate a few servers and others will want to keep it alive for more so the cost is re-evaluated each quarter so when a sponsor stops to participate the other sponsors have to pay more for their package set so that's how it's work so we see was the first one first distribution where we offer the extended LTS and it will likely last until end of this year so it will have provided the one and a half year more support there is this URL where you can find the repositories the list of package supported and any information related to this project now some interesting data some figures so five years already what I said over the years we paid 24 contributors and there are 14 of them we are active right now and I were paid each month we found 250 hours per month and here you see how it evolved between the start five years before and right now the number of contributors grow together with the number of hours funded it's a deliberate choice I did not want to have anyone well I wanted to have a resilient team so if anyone stops it I'm not in a big trouble but I also did not want anyone to be dependent on the income that friction provided they had to have something else besides LTS to to for the life for so it's always a part-time activity when your LTS contributors you get between 15 and 30 hours per month of work some figures about sponsors we are close to 60 sponsors you can see how they are split among patinum sponsor gold silver bronze etc obviously there are more bronze and silver but if you look closely at the amount of hours that are sponsored it's more well spread so yeah I prefer having lots of small sponsors because it has fewer impact when one stops but if we lose one platinum sponsors it's immediately three days of work per month which are gone away so I prefer many small sponsors and few big but it's quite well spread right now it's a lot of money that is going on through friction to pay the demand contributors the reports only speak about work hours but the rate is fixed so you can do the calculation yourself and I made the calculation for you just to give some impressive figures so we're close to 600,000 euros paid to various Deben LTS contributors over the five years so it's a lot for individuals like us it's not so much for big companies but and the single indeed individual who worked during the whole five years got about 80,000 euros well more interestingly the former Debian project leader was part of the paid LTS contributor and during his two years of leadership he got sponsored by well LTS sponsors and friction and surprisingly nobody complains that I bought or sponsors bought undue influence on the leadership so that's a good thing maybe some people thought about it anyway but nobody complained well at the start we did set up clear rules to try to about problems because we knew that the topic was can lead to problems so there are rules to decide who can join our hours are split among contributors what must be done what can be done on paid time and who decides in case of problems all the expectations are clear at least I hope so so basically anyone who is a Debian contributor developer maintainer who has prior experience with secretive dates can join well obviously need some programming skills because you have to write patches in many different packages you have to have some fiscal well friction is a company I can only pay invoice so you have to have a way to emit invoice obviously and you must accept some basic rules like privacy of sponsored data you have to accept to public to publish your monthly reports so well you must be aware that others know I mean you how much money you you got from me and so friction you have to admit to the Debian of color of conduct and you have to make sure best if or to meet the quality standards of Debian in general and of the day and security team in particular now how do we split hours among contributors it's simple we split seemingly but obviously not all contributors have the same amount of time so you can fix a limit you can say I don't want more than 10 hours or 12 hours and most contributors do set a limit so roundly three or four which accept an infinite amount of work every month we also don't want to give two few hours because you can't really prepare a security update in two hours you need at least four to five hours to do a serious job so we have a minimum amount that we can end out and if we are too many for the amount of hours that we have we would organize our rotation but this never happened so far as time grew as work time available grew I always recruited new persons and it works well so far so what must be done this is just basic work so there's a TVE tree hygiene so handling incoming issue looking whether there are applicable or not obviously prepare security update and publish them there's an obligation to respond to queries to offer the Debian contributors there's nothing more annoying that when you come to ask a question you don't get any answer and this not acceptable from from someone who is paid so there's this obligation but we have also rules of what is acceptable to be done on bedtime even if it's not exactly the core of what we have to do for instance so well you can write the patch if a trim hasn't provided anyone any patch yet you can also prepare a security update for all those release because when you have made the effort to do it for one release it's often helpful to prepare it for unstable if the maintainer is really busy or if the security team kindly ask some help well we do provide it we can also work on the infrastructure which means mainly the security tracker but also on various packages themselves either to enable things which are useful in from the security perspective like their hardening flags or adding auto package tests because well when you will do a security update you want to test the package that you don't break it for all this we don't know all the software in Debian so when there are auto package tests we do run them and so we try to help to add them as well this is the kind of thing that we allow to do when we have work hours and no there are no urgency or no current package to fix and the last rule is we decide in case of problem well friction being the trusted intermediary it's friction who decided and friction is basically me and my wife so if there are issue there is someone who can decide most day-to-day work is no managed by Olga Levson we will dispatch work hours and collect reports and stuff like that but most decisions are taken by consensus among all the paid contributors but in case of issue there's a last resource so lessons learned well it's possible to pay Debian contributors without disrupting the entire community I say this because well for a long time the topic of money in Debian has been taboo because we had a big incident in 2005 I think where Debian leader tried to pay the release manager to get a release quicker out it seemed like a good idea in principle because well release management did use work and it takes a lot of time at release time in a specific point but obviously the money was from Debian well money was targeted to release manager by the Debian leader and nobody else had any say and nobody else was be a few people were being paid at that time in Debian so it didn't go well and in fact many people or a few people tried to find new RC or release critical bugs to actually delay the release that they wanted to be faster by paying people so in the end it didn't work very well and ever since well it's kind of taboo topic in Debian but it's coming again because in the last leadership discussion prior to the election well at least one leader suggested the idea to use money to pay people to do work and well we had some interesting discussion but no big flamer and no disruption well in 10 years the the situation changed many of us are now working in open source and free software as part of our professional life it's not only how be and well so we can go forward possibly still we must be careful and I think those rules that I tried to follow for the Debian LTS are a good starting point everything you do must be done transparently in an inclusive way meaning everybody gets a fair chance to participate well there must be rules always but there must be there must be clearly documented and so that's the expectation are clear obviously you don't want to get anyone locked in a position while being bad because if he can continue to be bad he will prefer this and so he will make choices that may prevent others from taking over that is bad so you have to have rules to avoid this well obviously you must be aware that it will change things possibly for the good but possibly also in bad ways when money comes into play priorities of people will change so and this is where I will hand out a few ideas what could we use money for in Debian actually there are many ideas one is package maintenance strange because it's at the core of what we do and doesn't seem natural to suggest this but actually when you think of it there are many packages that users want but nobody in Debian cares about they are possibly orphaned or maybe there are softwares that are not yet in Debian that users really want but nobody in Debian cares about and actually it's part of my business friction as a web page where I offered to package something into Debian for someone and I do have at least five or six customers like this on the opposite side I'm paying Debian maintainer to maintain treatment it's the software that they use for accounting it's free software but one of the early choice that that made to do all my accounting with free software unfortunately it's not really popular software so at some point the guys I know enough is enough it's too much work for me I don't want to maintain it anymore I said I'm using it I need it so I paid for the date and I will continue to pay but it's a bit a lot of money for just me as a company but again this idea of sharing the cost that many with many me all those is a good idea we can obviously found new infrastructure in Debian we have this project of having a package archive archive for each package so that we can provide multiple versions in parallel but nobody gets to implement it I maintain tracker Debian.org there are dozens of good idea as wishlist bugs that I would like to see implemented and I would not mind if people were being paid to implement them there's a limit because obviously I still want to review and reviewing takes time so if we go in this direction there might be issue to consider that you might also want to compensate the work of the reviewer not only of the person who is doing the work there are also plenty of repetitive and complex tasks like the new review or the review of unblock request during freeze why not possibly using money to get those things down there are often blocking point in the sense that all those Debian developers are waiting for them before being able to continue so it's a good idea to bring money to streamlines those kind of process more recently I also read an article of Moli de Blanc was suggesting to pay leadership role because well being Debian project leader is a part-time job really so only people who have enough income can afford to run because if you well have to work five days a day five days a week to have enough money to live you won't spend two days for Debian so it's that would be a way to have more candidates and be less restrictive well that's it I'm opening a kind of forums or a debate on the where we could use Debian money in Debian now it's up to you now that redhead got bought by IBM and we have LTS established and proven might what do you see maybe some more requests for Debian as a long-term supported Linux distribution and related to this a big company like Google when they want to cooperate when they need a provider of a Debian disk Debian based is so they go to canonical because somehow canonical crossed a threshold to be visible to Google and smaller companies like creative they are not on the screen why do you think is creative not on the screen but canonical is well why I don't know but it's a matter of fact yes that for very big cooperation you you have those rules so if you have a company is in a clear support contract you can work with them and otherwise you can't I don't have any clear solution to this but well it's a problem it's a wrong problem technical wise because there are plenty of companies who have the skills required to maintain Debian and to provide support to Debian but we have no clear standardized offer maybe it's something that Debian project should think about it I mean certifying some partners as being competent enough but I don't know how it would work it's not really interesting as free software contribution so it's not the kind of thing that gets organized but that would be useful still so maybe a way to use money to organize this on another subject how do you reconcile the fact that LTS is paid work that depends on infrastructure that is run by volunteers for instance you put packages to the Debian archive the Debian archive is maintained by a set of volunteers that's very overworked how do you reconcile those two things well I don't I mean I'm obviously the question comes up when we want to change the rules I mean when we wanted to add a new architectures we have to ask building admins and DSA whether they are okay to keep the servers up for two more years and we have to accept those and when we get a know well we do it outside like I did it with extended LTS but well basic LTS is a sort of middle ground that is well accepted I mean everybody is aware that well companies do need five years of support and that it was an important requirement so that's it that said I'm not opposed to share incomes I mean if I if we had to give some money back to help with the support of the infrastructure we would be able to do it and in fact with extended LTS project we do have some spare money that we will use to sponsor Debian this year so yes so a few months ago I was in a conference from VLC who were explaining their economical model so they have an enterprise like Frixion who supports VLC and this didn't make them evil so they they succeed to do good work to maintain VLC to develop a lot of stuff around VLC and also they they also declined offers of publicity of companies who wanted to inject publicity on VLC so they really are not evil people even if they get paid for what they do so is there you know other projects that are in Debian that think about this about sponsoring our work development work and is there I don't know a project of discussing things about how to get paid for everything and in that case you would also pay the the everyone everyone that you rely on so are there projects to talk about it and to maybe not consider it as a taboo anymore yes there are so obviously there are projects like the Linux foundation founding core infrastructure and injecting money into open SSL and this that kind of project but I have seen recently a company called tight tight lift which is trying to generalize the well the way to inject money into all software and not only on the base packages that we all rely on actually I noted to contact them and to start a discussion but I never went that far but it's no time of no I think he was behind this project and we're close to over but I don't mind taking more questions either here or after like you want spell the name of the company you mentioned just tight lift t i d e l i f t okay thank you and in the debian community is there a discussion going on about this kind of stuff so you talked about the incident so after this has it been some other discussions about being paid for developing things in debian from time to time but not really well as I said it was really a taboo so not much discussion this year in the debate election we discussed this a bit it was an important part of the project of martin michael mayer who's a debian old timer like me except that he's a bit of stopped contributing because well he's doing open sources work but not as side project anymore and he wanted to to reconcile debian and his work life in some way yeah we are out of time maybe you can ask questions in the in the hallway yes sure you can go and grab me a mirror up until today tomorrow I guess we can we can thank all of us from this morning thank you very much thank you