 Hello I'm Guillem Giver, I maintain the packets currently and I'm going to talk about the past present and future of the packets and how some defining traits that were Decided long time ago have affected the package itself and The actual devian project and the ecosystem so I'll cover the past the regions of the package What I like to call the package ethos the regions devian started in 93 and after Well, some some of these I Was not around so I've gathered from either Coach Comments or mailing list pause so I guess I mean I've been in the project for a long time, but I don't consider myself a Long timer, I mean I don't consider I was not there in the region So if some of the of the old all timers are here might want to correct and just feel free So the first change look in the package is from 94. So one year after The first announcement, I'm not sure how much time It passed from the first implementation from the announcement the first implementation and Then the first appearance of the package source was in 96 So there was actually a long period of time where people maintainers were actually handling source packages Manually and there there was nothing. I mean there was just tar and and and extracting and doing doing things manually So this I will show later how this also affected How some decisions were taken about about the package? The package ethos is what I like to think about the pillars the design pillars that were Decided at these really early stages of the project and For me this has defined Many things that that are key to the deviant project and our ecosystem and many things that define us and that have changed have made it really Difficult or easy depending on how you look look at it So the first is that or or one of the foundational pillars was that We should only require standard unique tools to manage or basic formats. So the binary formats they are Using our and tar and they don't require anything else and even then they use standard The standard formats of those they they don't use unknown extensions or extensions specific to deviant and This is good because at the time it allowed to handle those formats from other systems So linux and all the free like the free software operating systems were not widespread Well, there was bsd, but but it was Good to be able to handle deviant staff from from external projects So it also helps with recovery. So if you have got the system and then you have to unpack binary package, it's really good that you just Require some standard tools that are available everywhere They are not the most efficient way to to store those and we actually for some right now have a hard limitation Due to our the R container. So The good things that gives us using standard Formats also This allows us to to move forward The other one is the metadata most of our meta meta data is stored in text files And most of it is stored in depth 8 to 2 format and that was also the beginning of of The creation of the package and such of the project and it has set a standard for everything else Most of our interchange formats in deviant not all of them, but most are in depth 8 to 2 right now There's increasing use of yaml for some from ftp masters, but this is this is kind of the or specific format for for it interchanging data and It is good because you can edit it with a text editor. I mean it's easy really easy to get into and It requires I mean the fact that we are we have standardized on a single format it makes It requires only a single parser for everything It allows because for some of the package stores its internal databases of as a text file So it allows for Workgrounds so you could you can go into the package database and that was one of the principles design principles the beginning That no binary database was to be used so that Cis admin could just go there and and like mangle whatever and and just fix Easily what was what was on the database? But at the same time it invites people to to mess with the data. So the package Has this vector of of users? I mean it's not just the data is coming from the from the dev packages that might be bogus It's also that the Cis admins that might be messing with with it all its internal details like all the deep package database is text file based and The this format is not expressive enough. I mean you have to pre-define what each field means because it doesn't have Semantics so you cannot express Lists you can't express many data formats you have to have someplace which specifies What value is what and and and how to interpret it? and Then the the last pillar is the the very thin plumbing The package as I showed before the package at the beginning didn't even have a program to handle sources So the the plumbing in the package is really very thin and that's one of the reasons why we have so many helpers we have many helpers and Different implementations. We also have like tooling on top depth scripts and and lots of additional tools and helpers and get Whatever built package So we have lots of an environment that's built on top of the package because the the tooling is really really even if it does complex things like the package source, but the tooling is really very very basic and It is good in a way because it allows it has allowed for exploration so the fact that we have depth helper or CDBS itself it is It is possible because the layer was really thin and The rpm world for some of they don't have this kind of thing because it's everything is provided for you by the package manager It also gives a maintainers freedom to use whatever they want The whatever BCS they want to use whatever workflow they want to use it gives freedom to work as as As more the more a most comfortable way that you you you you can But it has a price. So the prices that Because the tooling doesn't enforce policies or doesn't enforce ways to the things we don't have an archive wide Way of doing things. So we don't have unified BCS Everyone uses even if they use it they use it in a different way. They use it Like the package upstream or not or the deviant directory They use different different helpers and then you have to cope with that if you're interested in the distribution white problems then you are screwed and the present I'm going to talk about the package how it is interacting with the rest of the ecosystem and What are the problems or the the problems that are faced when maintaining the package? One of the biggest issues with the package is that it has a very vast Interface surface so the package has The formats themselves the binary packages and the sources and the stuff that you upload to archive and Those are by themselves are actually interfaces the format is defined so that you can actually create them by the standard Unix tools so the package has to support not only creating and and parsing them, but also The format themselves are part of this interface then also the The files contained in the control area of the of the packages Those are also part of the interface all the control triggers All the maintenance creeps how they are called And those most of those the interface itself. It's not even version So the the difference for someone with the with the binary packages and source packages is that you have a version where you Can specify what what the format looks like but for stuff in the in the Control area the the metadata is just there So we it's really easy if we extend it it just on the global namespace Then there's the runtime behavior guarantees the package currently Follows what's supposed to be in policy, but even then sometimes it has implementation details and when those implementation details change that may affect packages even if they are not compliant So the actual implementation details are also part of the interface There was recently a fix in the package which changed The this the sequel breaker Cycle breaker ordering and then because package was ordered before another one then it broke things that were not supposed to break so Then also the command line interface all the tools that the package exposes All those are used by not only people They are used also by all packages or most packages in devian maintainer scripts And then also external programs that they use them Then there's also the pearl modules that have been There's a mix of public and private modules, but even then we have packages that use the private modules So yeah, and then there's the leap the package see interface which supposed to be private But we also have well, it's semi Public, but it's unstable, but we have packages which use it because it's better than than having an unstable shared library I Try to take care of those users So that they don't suffer like much, but still and then we have the internal interfaces as I've mentioned before which is for example the database and This is part of the exposed interface even if it's supposed to be a private interface If I will consider changing the database things will break Things are actually right now depending on the database being available and being a text text file and There's lots of things that depend on this One of the issues with the package is that it touches very disparate areas in the project It has to deal with everything around I mean what I mentioned before that this is the glue of the of the ecosystem the package is there to glue Everything together. It's gluing up streams with Deviant stuff It's like making things fit together and of course one of the benefits of glue is that it Binds things together, but at the same time it makes them hard to move So the package deals with build systems dynamic static linking cross building. It has to like interact with all teams that are involved in this it also has to deal with new ports and architectures It has to deal with whenever there's a new feature required in the archive or for the system then it has to get involved It has to deal with binary source packages and everything that the relies real relies on those and with policy stuff even though I tried to make To keep the package outside of policy as I Really don't want the package to enforce policy because it should be a neutral package but sometimes it has to take care of that anyway and other stuff so Uses in devian it's it's as I was mentioning before using in devian is really big because we have like it's used by all the packages and Britain in like any system build system any language has to interface with all all those it also is used by maintenance packaging stuff and then the very actual end users is admins and and Deviant and sorry the package itself reaches far beyond devian and derivatives something that I think Sometimes people do not realize in in devian. I mean Of course the package is its primary user is devian itself, but the package is being used by the not only derivatives by Other system like think or even people that are using the package on HP ukes or I X or whatever Even on like iPhones or So in the end devian is just another downstream It's probably the most important one is where all the development is happening But it just at least from my point of view. It's just one downstream So in that sense Generality and portability are really important and that's one one of the things that I've been trying to improve over time recently I got access to the GCC compilation farm so I can port to more systems and The but that's something that was mentioned before that's something that I get the impression that some people in devian Do not realize and even though it's the most important project or most important user. That's still something to take into account and Maintaining deep baggage. Sometimes it's difficult because as it is in a central Position in the project and it's interfacing with with everything and many parts of the project are using it then It's it's easy to get to get pulled into into lots of trouble One of the issues is the decision-boundaries you have to be Aware of when something can be decided in the package itself or when it belongs Let's it's just a pure the implementation detail that you can change freely or it it's kind of owned owned by the package itself and It's your responsibility Or if it's a cross cross team Responsibility so you have to involve other teams because they might get They have to decide as well or even if it's a deviant white or distribution white decision sometimes Things are affect so many people in deviant that it's it cannot be decided by just the the package maintainers or like few teams around and Changes as I was also mentioning before can and will break things even if the changes are correct I mean because the the surface interface surface is so so big like everyone expects Things to work as is so many times you can predict them beforehand and you can you can Fortell that something will break and you can just look for the breakage and plan and and create a transition sometimes Well, you fix some inocles back and then it just breaks some stuff somewhere Just matter of fixing if there are no many packages and sometimes it's a matter of damage control you fix the breakage In the places you have seen but you also have to fix the package to cope with the breakage either to stop evering out and and warning and you have to backpedal and create an actual transition So you have to but but the problems that you realize it afterwards And sometimes you just have to revert because there's no other way around it I mean if you start breaking like hundreds or thousands of packages, there's just even if your changes is correct I mean, there's there's no other way so for transition planning usually what what It is on is that there's if you plan ahead you announce it somewhere like this is going to happen and then You add a LinkedIn check or another distribution white checker and Then you might also want to do mass rebuilds of the distribution to verify that nothing gets affected by the change and Even the packets might be able to emit warnings so that people Upgrade to the new thing sometimes because it's deprecated or because You are going to remove it But in any case changes are traumatic. So no, there's not I mean In any way you look at them like most changes in dev in our traumatic the test suit has ameliorated this and Right now we have a quite good coverage not not too good. I'd like to improve it I like to to improve the coverage, but the packets now right now has unit testing and functional testing for the sea and pearl parts But still it could be it could be better and Temporary solutions have a high cost and sometimes people do not realize this and propose things and and they just want to get things moving And I can understand this and I know it's frustrating to to to want to do things and because the package is on the on the on the key position and my blog your work, but if You introduce temporary solutions many times that that temporary solution will become an interface And that interface will stick forever and the package will have to either maintain it or we have to handle a transition to get out of it or You might get you may introduce changes that Involves work for lots of people and lots of people will switch to that to that change and when the new Better or correct solution is implemented then you have to revert and implement the new thing So for me, it's also matter of responsibility I mean if if you are introducing things that you know beforehand are temporary you are imposing work on like lots of people in the project and Most of the time is taken by design and thinking about the problem corner cases testing It feels sometimes that there are patches around and it's just a matter of applying things But this is not the case. I mean you have to really consider deeply what's going on what will happen What if these affects other teams if it the it affects all our use cases? So just because there's a patch it doesn't mean that you can bling blindly be applied And this is probably the project where I'm most conservative with I mean I when I Devian was frozen. I was running almost 400 packages from Experimental I like to try new things but for this I don't know Maybe just the project itself, but it just sets you into this conservative mode As an example, I just wanted to present this source version soups bar case this was a soups bar that had confusing semantics and Around 2007 it got replaced by the binary colon source and source No, sorry binary colon version and source column version and at the time as an experiment There was a Lintian warning of course, but as an experiment I wanted to see how long it will take For these to transition to the new ones without active actively pushing for it. So I just did the change in the package, but I didn't do anything else and This is the what happened So at the beginning due to the most of those of those packages Got converted due to Lintian and because it was affecting bean and mues, but the rest Has been like lots of years that people have been converting slowly But there's this long long tail and right now in their guy I think there's 30 40 packages still using this obsolete obsolete soups bar. So I think this is Really good example of how things work in devian. I mean if you don't push for your changes things will stall They might just go like asymptotically to Infinite I mean just keep going So what as I was saying before it's really easy to get in the hot spot Many people the demand conflicting features So one team wants something and another team wants something else and they just don't match and you have to try to either Convince both of them or try to find a common solution People demand things that are this this solution specific and right now with the vendor stuff It's actually possible for for the package to to move that aside and make it devian specific, but still These sometimes are not not good People demand workarounds and hacks be implemented in the package itself people demand insane things and I've heard people saying that the package is both too slow and too fast on its development I've not yet heard the same from the same person, but You you never know and Many times you sadly you sadly have to say no and things just This is not the place or the solution is not good and I Think this this is actually not as bad as it might have Been in the past, but many times you've explained the proper reasons for why you think this is not good people understand or at least if you hear and understand the problem and and Acknowledge that that this is a problem that should be fixed most of the time people understand I mean they are there understandable and Because the package is the interface at least in devian. It is just a focal point of Conflict so and the future I'll just talk a bit about Things that I like to see implemented and and open or the biggest things that that are Currently like people most people complain about so I think the biggest deficiencies right now are No file metadata tracking the package doesn't have any knowledge about stuff that has been unpacked from from a binary package So once it hasn't packed a package it loses track of everything and this this is actually blocking many things that Need fixing and because we don't know deep packages doesn't have this knowledge then It kind of do for some of the typical cases the sim link to directory switches It doesn't know if this change has been done by the Cis admin or by the previous maintainer of the package. I'd like to fix that Eventually I've learned also my lesson that I should never promise dates. So I Just like this will be fixed I'd like soon, but yeah No confiled content tracking. So I think that this is one of the biggest also Features that we are lacking compared to to other systems that the package doesn't doesn't know what was the previous contents of the Confile so it cannot do a three-way merge of of confiles and then There's no built-in package signatures there's been lots of people requesting this and We have Packages supporting this case, but they are not part of the core Deepak is distribution and I recently adopted Depsic verify and I like to integrate this better into the package itself and Then lack of more declarative interfaces. I think that's probably one of the biggest Also issues with packaging that's making things way more difficult So we have lots of things that could be just placed in a file and specify like Alternatives or diversions or many things in the packaging that could be declared instead of programmed in maintenance script and then open problems One of the biggest issues because of the pillars mentioned at the beginning is that because the plumbing is so thin It implies that you have to learn lots of layers to learn packaging So you have to know the package you have to know also the policy how to apply it because you have to write your rules from scratch Or you have to use a helper and then you have to choose which helper to use Which style inside this helper you have to write your your metadata there's lots of information to use and then you have to choose which Program to build stuff in top of the package build package and I mean it's really it piles up And it gets really difficult. I think this is one of the biggest problems We face come first on compared with rpm world rpm or even gen 2 or most of the other distributions It's really central. Of course, it has the other drawbacks that they have really strong Policies and and they they don't have the freedom that we have but I think making packaging easier by by Not making the the plumbing More thick, but at least making it easier for people to use to get into I Don't know how to fix it. I mean some of this stuff It just probably to move things lower, but I don't have a solution But it's things that I think are important for the project then the other recurring thing currently is the source distribution thing How do we distribute it? Do we use Digi? Do you do we use the new source for formats? Do we switch completely off? source packages and use proper like BCS, I don't know I think this is a thing that will keep coming up because we have not properly solved And everything that is going on is good as I've mentioned before it's it's this experimentation that's allowed but I don't think we have a clear answer and or an answer that convinces the whole project and then how to deal with applications and like App markets and all this stuff and containers. I don't know just I think this is something that we might have to Cover perhaps, but this is also one of these big issues that that lots of people seem concerned about and With the current model, we are not covering. Maybe we we should not I don't know but just One of those big big items and yeah, that's That's it questions If you have questions if you'd like to make it around to the microphone to the left of the auditorium We can take your questions there. Thank you. So hi again. Thanks for maintaining the package one issue I stumble into often is that Packages do a lot of work around For the files and user shared doc and they do sim linking stuff and all ugly hacks and linking files and linking directories I think you had a plan for that of turning those files into a deep package meta data and Do reference counting or that is that still a plan you? Envision or do you have different ideas how this could be solved? That's something I like to do But it's one of those things that depend on the project to accept it So there's there's many many things that I'd like to improve But if the project doesn't buy it and then it's it's not something I will be doing Unil unilaterally so but that's something it's in the My head and I probably will try to push for it at some point But I think it should be in it will improve part of this and it will give us at least for the Machine readable stuff. It will remove space. I mean get rid of duplication and I think will be good But again, it's something that that this freedom. I mean this thing that we we've got with the system just So there's no technical problem with that, but I don't think I don't think so It it will require a bit of implementation in from the package side, but all the required Interfaces are already there. So it's a matter of the project mostly agreeing to it. Yeah from the deficiencies you Lined out is there anything that you that is already being worked on or anything that you think may be Ready for stretch these release. I will not promise anything, but I do have I do have code for the confile Database stuff and for the metadata also started working and I forgot one which is that I like to implement to make the package have its own internal tar for building not for extracting because it already has its own internal tar implementation, but on a Building tar implementation will allow us to get rid of I mean to build packages without being rude because you could have a manifest Which specifies which all users or owners of the files are there So that will allow us to probably also optimize and only call one rule Like not binary build and binary and also not have to use fake root and stuff like that. So