 document that we give to upstreams and to tell them how best practices in making their software suitable for Debian. So this is the document. We've got a number of sections about different things and today I thought we'd start with if anyone has any feedback on the document already. Have people read it? Any comments? That's sane, okay. That's essentially that upstreams are not creative with their packaging and especially well one has to be creative if the packaging system is not very smart. But something like Python upstreams, Ruby upstreams, what have you, they should have a reasonably expressive declarative packaging system and it's basically a contract that tells what they're going to do and the less creative they are, the easier it is to make sense of what they're written. So I would say non-creative with regards to packaging, because interpretation happens downstream of them, use their packaging system and accept patches, because they should not be required to know everything. So I think I don't want to require upstreams to make perfect packages because they essentially may deploy in their own system and a bunch of others and Debian can do the work for them of testing their build system in all sort of places and send patches and so it's a service we give to them, not like a nitpicking we do to them. At least I would like it to be perceived that way. And so it's like, yay, we got your software, it's really good. And this is how you make your build system work in S390. Isn't it cool? I mean, and I think that that's kind of the dialogue that we should have with upstreams. There's things that are an issue for us and we could explain why they're an issue for us, like including third party libraries. And there are things that we can do for them and they're very welcome to use our experience in infrastructure. Yeah, I think that's about it. To like stimulate the discussion a bit, I wrote a food for thoughts section. And just by going through the document and looking at stuff that is incomplete or needs work. So these are some of the things. So if anyone has any thoughts on any of these items, can we see who's actually read the document at least once? That counts, yeah. Can we have a volunteer for the first one? I mean, the second one, not the first one. Proofreading it, looking at the grammar and the spelling and also contacting the Debian localization English main list to help with that. Does anyone want to do that? I'm looking at Kathy over there. Basically, just proofreading and clarifying things and collaborating with the people who are doing that on the Debian localization English mailing list. Okay, I can do that. Are we just talking about the wiki? So let's maybe discuss the WAF section. So the FTP masters object to having WAF build system in Debian for various reasons. And someone added an objection to that. I was I'm interested in hearing if anyone has any thoughts on the object objection. It's a random build system that's used by some projects. Are there significant software projects that use WAF that this is a concern of? Or is this mostly just I mean, where is it coming into play from? I believe an example is the Samba project. Yeah, I'm not sure how many other upstreams use it, but it is used. I wonder if it would be better to have positive recommendations for certain build systems instead of saying don't use Scones, don't use WAF, don't use this. If I'm considering WAF, what are the options I'm considering? And should we say Debian would really love if you would use, I don't know, automaker, PyBuild or whatever it is, and recommend that. Yeah, I think that's a general argument that I'm skimming through the guide. And if I were an upstream, I would feel defensive after reading this guide, because it's like you should do this, you should do that. I would rather be happier. I would I would not I would feel less defensive if it were worded like in Debian, we would prefer this over that, because it makes our life easier in that respect. So, yeah, like third party code, please do not include. I would tend to reply, please do not tell me what to do. Maybe I have good reasons for including it. And what are you to judge what I do? But if I read something like other code like libraries tends to be a problem in upstreams for us, because we do security, blah, blah, blah, track dependencies and so on. So if you need to include it, please do it in a way that we can easily disable during build. That will be more like requirements analysis for a project, rather than somebody finger pointing my my work and saying wrong. Or source only tarpaul it says please provide an archive of your source code as blah, blah, blah, that's what our toolchain can currently work with. It could be worded as our toolchain can currently work with this. If you can please provide one. At least it comes from a use a proper use case that that people can reason with and not just we're not telling people our way is the best way you suck if you don't do it. So some of the incomplete sections is, for example, why we have a stable release and why that's a good thing and why we use old versions of software in that stable release. Other distributions like Fedora, they just upload new versions in their releases. So as anyone have thoughts on how to explain this to upstream people, the branching like Debian stable and we don't introduce new upstream versions into that usually. Um, that's like a reasonably contentious thing with many upstreams. We have the equivalent that does remind me there was discussion at the web app off about web apps that have super short security cycles and thinking about a way to put them in the archive in testing and then backports but keep them out of stable. And it might be nice. I mean, this shouldn't be a web apps only thing. It's probably gonna be most common in the case of web apps, but it might be nice for certain upstreams to know that that option is available. And conversely, it might be nice as a project for us to decide this is the thing we think is generally reasonable for certain packages to have if we know that the upstream security cycle is short and if we expect security issues during the lifetime of stable. Okay, I guess I do have input in terms of just a form of argumentation that can make people less defensive and bring them on board into reasoning it out. If in the documentation explaining to upstream that you include what you predict might be some of their concerns. So for example, you suggested tabs. This is why we use stable, etc. These forms, these processes, workflows, whatever. It is a good thing because blah, blah, blah. We accommodate some inconveniences with the following and then list possible concerns you can imagine many upstream people might have and then encourage them some trade-offs between some of those approaches and then describe some use cases. So if you have some super short security, blah, blah, blah, you might consider blah, blah, blah. As anyone have any thoughts about advice about upgrading data and especially and user configuration, obviously the packages can handle upgrades of system-wide data, but the data and configuration in user home directors isn't upgradable in that way. I think that's a general problem with upstreams that comes with a stable release policy or contract that they can have. Some upstreams are aware of this and they will have like stable releases, development releases, migration plans, documentation for it. Some upstreams won't even bother to keep the same API or the same command line interface from one version to the other. So I think that's not a Debian specific issue. It's something I would go through the GitHub people and hit them all over the head with. And we could say that as a distribution that tries to make stability guarantees across releases, we are more likely to do a good packaging of things that have a reasonable development workflow. And then I wish we could point to some external documentation about reasonable development state-of-the-art sort of best practices for releasing free software. Don't know if there are any, but I wish that there are. Because I think it's not just up to us to say that stable releases exist and the API should not be broken across stable releases. Right. So we can point to those essentially and say that that makes our life easier. Because we do take care of a stable life cycle. One thing that they could do in that respect is if they chose to do stable releases, talk with the, like have some interaction with the Debian maintainer so that they could coordinate on what is the version that they intend to maintain for a longer time so that the Debian developer can say, I want to upload the new upstream version until stable is released because I want that version to be in stable. To avoid the fact that upstream finds that there's a development version in Debian stable and they receive bug reports about it for two years when they never subscribed to maintaining that for a long time. So, yeah. Negotiate with the Debian maintainer what version they commit to maintain for a long time. Negotiate with the Debian maintainer saying, I do not intend to support my software for a two-year life cycle. And the Debian maintainer at that point can choose not to have that thing enter Debian stable, which I think would also be good. Or the Debian maintainer can know that it's up to them to maintain it for two years. At least it's clear. Yeah, or the backports only thing, I don't know. But as a Debian stable user then I want to know that the new version I get of that needs to be tested by me because it's likely to break existing things that rely on it. So that breaks the usual Debian stable contract. If you were in yesterday François-Mariette talk, it's like develop stuff with Debian stable and get security updates with minimal hassle. So if a security upgrade means that I need to rewrite my production code, it's not quite the same thing. So I can put in my requirements not to use the stuff that is provided via the other faster channel for stable and rely only on the code that will guarantee that I can do a security upgrade without planning a week of code testing and rewrite. Clarifying question. Are you trying to explain what an upstream developer might need to know about choices they need to make about how they will be involved with long-term support of their package and what they might need to give up to some other maintainer? Is that the general area you were just explaining? Sorry, I have two things going on in my head and I lose bits. More like the... I would like there to be a conscious choice on upstream of their release practices, which could even be I don't want to bother with them. But at least it's constantly chosen and documented and the Debian maintainer can act on it. Yeah, a checklist is a good idea. Do you plan to have stable releases at all? Do you plan to do security support for stable releases? How long do you plan to support a stable release for? What is your current stable release? What is your next plan for stable release? And let their users know even. I think a checklist like that would be pretty useful for upstream. The part that still would be missing from there is that there's different timelines for the different projects that are not always going to overlap. So that seems like something we can't quite put in a checklist. So by that I mean some upstream project has a release cycle that is a certain release cycle that doesn't necessarily coincide with whatever Debian's release cycle is going to be. And there's unknowns usually in both of those and getting those to synchronize is often impossible especially across all the different packages that are in Debian. And that often causes a bit of friction I think with upstream like oh my god you ship that version. Why did you do that? Or why did you jam in that beta version into here or whatever? And I don't know how we can resolve that in a checklist format. But the checklist I think could help there to get the upstream to think about what and plan for what they might want to have in that situation. Maybe this is what you were just saying but it might be worth a package guide for how to talk to us upstream and what to check with upstream in addition to an upstream guide for what to do. I mean I think we can be clear about we've been doing these predictable freeze cycles and things like that. So just being as clear as we can maybe pointing a link to whatever is the current schedule. So at least we can say well the freeze it really should be ready a few months before that if there's any way we can coordinate to get a version that will work for us. Debian could do a lot more there to encourage Debian developers to do that with upstream. Like when the release or freeze date is decided and announced to the project it's often like buried deep in a bits from a release team mail and there isn't anything that ever says like hey by the way you should now go to your upstream and say here's the freeze date. We're not telling ourselves that we should contact upstream and let them know about that. We just think oh god I got to get this resolved before the freeze. And it's fine if upstream saying your release is not my problem because our release is our problem then an upstream can be nice and say cool or they can say well this is my release plan deal with it. Both are okay in my opinion and we shouldn't tell an upstream you're bad because you don't care about us but the thing is as long as the Debian maintainer knows what's upstream plans again that can make choices which I think ultimately ends up with less than maintainable software in stable. It may end up with a smaller stable but it ends up with a stable that's way easier to manage and then yeah the Debian developer can choose to do backports can choose not to have the thing in stable. I see only benefits. To continue that discussion I know one frustrating thing I've heard from upstreams is that all the different distributions have different times of releases so they all have different versions of their their software in them and supporting all those different versions across the different distributions is a major pain in the ass. Some organizations like Puppet have people that can pay to backport security fixes to six different releases across all the different distributions but many upstream projects can't do that and so they just give up on that. So there's another point there of the release cycle between Debian and the upstream project aren't synchronized but they also the versions that end up proliferating amongst all the different distributions can really create a lot of problems for upstream and that that's a tension point I've seen with some of my upstreams before when Debian has this version Red Hat has that version Ubuntu has this version and there's it just becomes this giant mess and a problem for them and people go to them and complain when sometimes it's Debian's fault or people should complain to Debian trying to minimize the upstreams frustration in that aspect it seems like a very hard problem that I don't know how to solve but communication increasing that communication with some of this in the checklist to think about those things and try to figure out how to plan for that would help a lot I think. Are there central places where release teams provide legitimate legitimated official semi-official graphical pictures of timelines like project planning flowcharts that help people look ahead road maps if someone produced that it would be the release team yes. I think one of the problems with that is that traditionally Debian has released when it's ready so there's a freeze and then we release when we get the RC bug to count down as much as the release team thinks is necessary and so it's hard to it's hard to draw a timeline that has those things. So for a user interface for the reader you could have the dates in the past and the future events they are undated still graphically represented so they can see what the stages are to predict not knowing which dates still come. This is an example of software that you could use to do that it does have release information in it and you could use the same software and collect information from other distributions and create a timeline of distribution releases and that sort of thing. This is the Debian timeline events in the Debian world yeah I put that in no a while ago timeline.debian.net yeah my suggestion my suggestion is a new website with Debian's release cycle Fedora's release cycle Gen2 release cycle whatever other distros have a known release cycle the code base is pretty simple it's in Debian yeah just an idea um yeah if anyone wants to work on that I'd be happy to show them how it works so does anyone have uh okay does anyone have the guide has a few sections on Perl and Python and some other things does anyone have experience with Ruby, Node.js, Haskell, PHP, Go, any of the newer languages and pitfalls to avoid in packaging and that sort of stuff in those languages have you tried reaching out to the respective teams about those languages so my experience with many of these I think you're you're knocking your blue pin out or your your red pin out with many of the sort of less mature languages the notion of API or ABI stability hasn't really sunk in yet and that is something that I think Debian tends to actually care a lot about because we care about if your package is useful we care about the things that use it and I think that ends up being significantly problematic for for Debian and for the upstreams I know I've seen it in Ruby Haskell manages to avoid the problem by making everything statically linked and not having any libraries at all so that it's build time dependencies when you build a package it just creates a big bundle and ships that well if the API changes then it doesn't build and then it doesn't and then it doesn't go into the archive right so but then that means if there's a security flaw in one of the in one of the libraries that's being used the only way to fix that security flaw is to actually recompile every single package in the archive that was ever built against it I don't know how many of you have mono installed in your systems but mono strikes me as one of the situations where you can't do a mono update without basically updating all 400 megabytes of mono or whatever on your system which is I think really problematic do you know does anyone know if the security team is aware of the Haskell situation for example and how do they deal with that in stable and so I don't know specifically my impression is that there's not a lot of security auditing that's going on in the Haskell or I think I believe that go is the same in the same situation and I think that there's not a lot of security review in that so with Haskell it shouldn't be much of a problem so that the static linking should be much of a problem because they have a master file of doom knowing all the version of everything and if they need to update a library then they will rebuild all the reverse build dependencies and upload them to Debian I don't know about stable I don't know what they do there but I think here the problem is not quite about the tools but about the communities because I'm I would imagine that we wouldn't have many problems from Haskell on that side because the Debian Haskell people are really good and they have a very good interaction with upstreams we may have problems from Ruby because we both and also the Haskell upstreams are pretty serious about what they do whereas we may have problems like with Ruby or Node.js because upstreams are not let's say mature enough to think about these issues and all the efforts to talk to them have been met with we're not interested so we don't have a good relationship with them because essentially they're not interested in what we do and there's not much we can do about it even if they have good tools they just have a community that does not support people wanting to make commit time-based commitments or like interface commitments so I think the only possible thing in that case is to say let's try again in one year see if the community has grown up a bit but to me yeah it's not about tools it's that's why I don't do any paid work that uses Ruby or Node.js I think some of the reason why that continues to happen in some of those communities is because they they find it hard to not just commit to that time to developing and maintaining a stable API or ABI but the it sometimes is easier to solve that for them by routing around us and developing the tool chains to install and upgrade things without doing it through Debian and creating their own distribution dependency management systems and so you have gems and you have bundler and you have these things and it's just virtual and if you have all these ways of routing around Debian to make it easier and that might be an indicator to us that it might be too difficult what we're doing for them maybe we can reflect on that to make figure out how to make it easier I don't want to give them an excuse a pass saying that it has nothing to do with those other languages and their communities because there's definitely a culture and an issue there but I think saying it's it's their fault not ours is maybe missing that actually for us it seems relatively easy but from an outsider it's quite baroque and difficult to get things into Debian and figure out how to package and do it right in a way that will be accepted and maintained over time plus than the ABI issue oh my god you know so so my impression is that as the outside as these language specific communities mature and develop these version distribution or package distribution systems on their own that are not Debian or Fedora that as they mature as those packaging systems mature and slowly begin to avoid some of the mistakes that we maybe learned to avoid five years ago they become they start to realize why we care about things like API stability and right and so once they have a functioning packaging system and they have enough community norms about hey yeah if you're going to release this thing you might want to think about how long you're going to maintain it for then by that point they're probably already doing the work that we want to do and we just need simple translations between their packaging systems and our packaging systems simple you know ha ha but um but what's what what I find sad is that that means that a they're redoing a bunch of the work that we've already done and relearning those things and and b that by the time they're ready to like make those commitments they already have a heavy investment in their own infrastructure rather than letting us do that infrastructural lifting for them yeah uh agreed on all and um I think we we there's not so those people are building virtual amp and and are building it because I guess they perceive that doing going our way is a lot of work for no benefit um so I think the problem there is that we fail to communicate what are the benefits in our way uh and there's people who understand the benefits in our way uh like the people I work with um but so like if you look at like I I did after force them post uh on my blog a post called that ops uh which uh and the point was exactly to explain why we do things the way we do and what are the practical benefits uh why I would not use why I think that virtual amp is not cool uh why I think that vagrant present company excluded is not cool um because that uh that makes me like lose ours in my workplace that I could use to go and walk on the mountains um so I think that is something we fail to communicate that you know we don't do this because we are uh obsessive compulsive uh maybe but um and and if we communicate if we find a good way to communicate that we also communicate to ourselves so why are we doing this it's also good to have a reminder for ourselves that uh we're not just respecting the policy because we have a policy but it's cool that the policy says this because it means that I can rely on that um I I guess I think of it coming from a different different perspective than that uh rather than like I think I I I like to look at it from why it is that or what what are the reasons behind uh these different languages um needing these virtual amp and and bundler methods of distributing things rather than saying why it's not cool that they do that I think the for the for these sorts of new languages they're developing these things because they need to they have a rapid turnover in development and need to have updates much quicker than they can get if they go through Debian for that process so it makes a lot of sense for a developer to use bundler and gems and whatever other c pan uh pi pi or whatever because when they're developing things they need the latest versions of libraries and they need those quickly and they need to be able to update and develop for those latest versions and Debian is not particularly suited for uh a rapid development cycle in that way because we have potentially long uh blockers like new cues and the entire process of getting something packaged by someone who's not a package or can take quite a bit of time so um and that's what in a way fosters the development of those infrastructures to a point where they're they're wanting to replace Debian when things become stable uh how you move between those two so to me it sounds interesting to think about what what can Debian do to make things for developers of languages uh easier to get into Debian sooner so they get the foot in the door so they're not depending on these uh alternative distribution mechanisms for when we need them to stabilize more because I think that well I don't I don't know what those ways could be but uh things like PPAs or other ways of getting available packages quicker into the archive or making things less policy stringent in certain environments if you know you're going to have a development environment might might be more uh or lower the barriers for that entry a little bit so where where I work uh there's faster development cycles and there's production systems uh and and I guess the same would work for upstream so Debian stable is the production system and uh Debian stable it does not mean that people shouldn't have a development environment um and when something comes out of their development environment that could potentially be ready for production and to be supported in production for reasonably long time it would be nice if it made its way to Debian stable so again uh it's could be good to document that Debian stable is not intended to be the bleeding edge production system the development system it's intended to be when you go to production uh so yay rapid release and everything then something comes out that's worth supporting and then consider getting it into Debian stable um uh can we support development environments in Debian then I'd like to um because there should they should give us all sorts of intermediate staging development environments in a way uh I think that's something that ARCO and infrastructure is not particularly good in handling because when I upload to unstable I upload for stable there's no this version is some fast-moving thing in view of the next stable um so at the moment I don't think there's much we can do about that oh just quickly we also call it unstable which scares away most developers I know because they want to have a functional system to develop on and so they develop on a stable system and then they're mad because everything's out of date and they can't update things and so just the word the language like calling something unstable may may discourage people from actually using it as a development environment so like the problem is that there's two things that are unstable there's the packages you're using to develop and there's like the kernel xorg you know your disk encryption system your mail client all that other just stuff and as a developer I don't really want my kernel to be changing all the time I'd like that not to I'd like that to stay if I get my hardware working I don't replace my laptop and to just get security updates I don't need a new kernel basically for the entire life of that hardware but I do need new versions of like you know Python the core interpreter that's that's sort and one thing I think is kind of worth thinking about is which is completely unintentional uh is homebrew the mac people homebrew the mac packaging thing uh they provide development environments and I see quite a number of developers who are doing all their local development on max because they like the hardware or whatever it is and they do production on Linux and homebrew has this implicit property that is providing packages that aren't part of mac os the mac os base system is there I'm not running pre-release versions of mac os I'm running homebrew which is giving me whatever I want and a better way to separate you know a stable desktop environment plus an unstable development environment and tools for that I don't know if the right answer is do all your work inside s truth which is what I've kind of been leaning towards but is a power user tactic uh something like that would the use case of hp be useful for describing some of these issues they work off of testing and they create their own future from there so now they're wondering how to get their work back into stable um so they I missed some of I missed some of the actually speed discussions if you go into more detail about this uh but are they building a local development environment off of testing like this desktop I will use the wrong vocabulary if I do that just triplet point about some time ago that uh coordinating with upstream can make it easier to pick a single long-term stable version across several distributions to make life easier for upstream I just had one last comment that reminded me of which is how do other distros do upstream guides do they do upstream guides should we work with fedora on this and say you know this isn't just the crazies in debbie and this is the general consensus of the people doing packaging I think fedora debbie and might be enough for that and conversely how does deb dry interact with that so I just wanted to um describe a bit of recent experience with that kind of coordination which I think was very fruitful um what's that right time you don't get the mic my understanding of hp's situation is they take snapshots of testing they develop an entire alternate um distribution out of it called h linux now they want to switch and work on and work with debbie and instead of just h linux they want to have debbie but how to coordinate their work environment which completely reproduces so much of what goes between testing and stable as far as I know you know you go from testing to various stages up to another a new stable how they do that all through their their own internal systems which are huge how do we help them integrate if they have made this very high level commitment to work with debbie and how does how do the two cycles get get integrated in a way that's productive for debbie and yeah so about the use of testing I specifically asked them why they use testing and not stable and they said that uh they made their choice because their developers asked them to use testing so I just um I wanted to just follow up on your remark about the coordination with other distros um so uh in terms of packaging tcp crypt which we had at boff about earlier this week I actually uh was lucky enough to meet with paul wouders who works on fedora um and I mentioned that I was going to be putting tcp crypt into debbie and he said oh we should put that in fedora I'll go do that tonight and um so so it's not in fedora yet because paul and I both took a look at it and said you know we reviewed it and we said but there are a few changes and we were I mean it was it was useful because there was face-to-face interaction also with upstream but we basically went together to upstream and said hey we have these concerns you know you're not dealing with the churu properly like how are you going to use this separate user account you know there's a set of things that will make it easier to put into into distros if you do these things and both the fact that it came from both fedora and debbie and I think made upstream say set up and take notice and say oh got it like we're happy to work with you it may just be that we had reasonable upstreams but that kind of coordination and and face-to-face communication which is good but that kind of coordination is excellent if you can if you can do it and and make sure that you're aware of who the other packages are upstream mailing lists are a really good place to do that identify yourself on the upstream mailing list as the debbie and packager for the software and see if you can you know you could even ask for other packages on that mailing list and then you can contact them out of band if you feel like you need convincing he hates me representing him but he then he can disagree with me and tell me and tell you where I'm wrong my understanding from hearing Ubuntu and red hat and other folks describing vagrant's role this is not from vagrant is they have said that he was important for facilitating the cross-distribution integration of the package so that it was the same across how you did that well one of the ways was by attempting as a debbie and developer to make it work on fedora which then they went and did it the right way because I did it all wrong but we had tons of users just asking for hey when is this going to be available on fedora so kind of just like okay I'm going to step out of my comfort zone I'm going to try this fedora thing and and I did that with centOS and other stuff but I'm mostly just encouraging people to do it trying to be available and I don't know nothing pretty noteworthy just actually communicating with people and encouraging communication and really discouraging distro wars I think we should just finish up I just wanted to try to I think we need to try to encourage upstreams to actually think about the errors given off by their builds and I know this seems like really a really basic thing that but if you can like patches that clean up warnings in the build process are things that should be applied if they make sense and that you know the cleaner you can get your build to be the the better our tool chains like breakage in our tool chains will be more visible if the if the build is clean in the first place is it in the document no you're saying I think I may have missed it because I got in late I don't know if it was brought up but I really want to make sure we encourage that so I think we should close here it's after the time but um this boff actually uh an upstream developer just came into the ISE channel and asked about it and I put him at gobby and he added a whole bunch of stuff starting here feedback from an upstream developer and there's a lot of things so it's on my to-do list to like look through it all yeah a lot of it is organization some of it is other stuff so I'm really happy about that and yeah I'm going to go through it sometime when I have time yeah we often get people on mentors asking for help writing upstream software anyway let's close