 to the frame. Good afternoon. Welcome to the post lunch sessions. I'll speak now is Jocelyn Mouette and he will be presenting on packaging shared libraries. Thank you. So I will try to tell a bit of what I know on packaging shared libraries and this is quite a large topic so it will be a bit shortened. So if you want to package a shared library for Debian, what you could think is you just run the configure, make, make install and build for your LibFu library, a LibFu 3 package containing a shared library adversion stream and the LibFu dev package containing what's needed for development meaning the symbolic link for the shared version, the static library, the headers and development tools like LibTool or PKJ config and that's all. You've done, you've got a package. But we are not too good too, so we are going a bit further. So for the beginning we'll ask what is a shared library after all. ELF, the ELF format which is a binary format for Linux includes the definition of shared objects which are code that can be loaded at runtime anytime. You know the DL open code for example and shared libraries are only specific cases of shared objects that are automatically loaded at start of time. A library is defined by its application programming interface which is used by the programmer which is what is in the source code, the definition of functions, how to use them, what are the semantics. So when the API of a library breaks in a new version it means sources need to be changed. The API application binary interface is a lower level interface which depends on the operating system. When it breaks it means that only binaries need a rebuild because this is an interface between the binaries and the library. And the ABI is defined by the SO name which is a canonical name that defines for a library a specific ABI. So when we have a package in our library that I described earlier, this is not enough. You have them to make it evolve when upstream provides new versions. So the general rule is that the packaging system which is in the BND package has to know about what happens especially about interfaces because it provides inter-package dependencies that mean interfaces. So if you have a new version that is just a bug face without the BI changes for example, there is no change in the packaging. You can go ahead. But if the new version includes for example new features that means new symbols, new symbols meaning new functions added to the interface or new variables accessible, you have to inform the SHLabs system of the package. So it's just a code you add the dash v version, the dash e parameter to the h make SHLabs. The third case that symbols have been removed or the meaning is changed. This is what we call the ABI breakage. So in this case the SO name has to be changed. In turn the library package which is built upon the SO name has to be changed as well. And the last case is when the API changes. This means you have to change the sources. So the best course of action is to change the development package name itself so that package that used to build against lib for 3dev have to be changed and to reflect the change in the build dependency. So what is a library transition? When the API changes you have a library transition and it means all the package using the library has to be rebuilt. And the testing process, the release process means they all have to go to testing together. So the first thing to do before uploading such a package is to ask the release team. Because if two libraries are entering together and they are untungled then they have to go to testing together and it complicates the transition. So there are several courses of action in DBM. If the change has sufficiently small consequences you can just upload right to an unstable and you have the release team to trigger binary non-maintain uploads for all the packages that are using the library. And if it's more complicated a more sensible course of action is to upload to experimental and try to see what's happening in experimental. The worst case is when you have to keep several sources in unstable at once and it can even go to stable. See for example GNU TLS or PNG for such examples. We'll go on with a small case study of a more complicated library than the lib foo that I presented earlier. It's the case of G Conf. G Conf is more than a library because it includes, well let's see how we have packaged it. In lib G Conf 2.4 we have the shared library which is called lib G Conf 2.4.2.so.4. And in the G Conf 2. common we have put the configuration and data, for example in telecinization data, and have a development library documentation and support binaries. But what should we do with the demo? There is a demo. It is spawned by programs using the library. It is also linking to the library. So if we put it in the G Conf 2.4.2 with the support binaries we end up with a circular dependency because that would have to make the library package depend on the binary package. And if we put it in the library package it goes against policy which requires library package to have only files that are versioned to be able to install several versions of the same library at once. So the solution is to modify the sources upstream to relocate the demo in a place that is versioned. And then when this is the case you can include it in the library package which introduce a general rule about circular dependencies that have, I repeat, to be all fixed because most of them are really critical bugs. And if it cannot be fixed it generally means there is no need to have two packages. If those two packages have to depend on each other then there is only need for one package. Then upstream often likes to make the DBN developers life more complicated and here is how they are doing. The first mistake is the non-position independent code because when you are loading the code dynamically as a shared object it is not loaded at a given address that you can know by advance. So the code itself must not depend on the position in the in the memory where it is loaded. So there is a compiler flag to do that and some developers omit this flag and for assembly code this is still not enough because well assembly code is not necessary position independent so you may have to ask for an assembly specialist if your package does that. Lin Shan can detect non-position independent code run Lin Shan. Another very frequent mistake is changing the ABI without changing the su name and this is generating a great mess because well once the library is released upstream doesn't know what to do because there is already in the wild a release that is broken with the change ABI so generality know well we will say with this version so what happens in if the if the change is small enough you can be you can use package conflicts can conflict with a package that are broken by this ABI change or you can if this is of a large scale you have to rename the library package without changing the su name which is this is not of course a good course of action but sometimes you have no choice so you rename the package and a library transition starts and some upstream developers are even worse because they do this for every single upstream release so an example is a gf5 which is a scientific library and there is a su name which is always the same whereas function definitions are always changing and another example is a Mozilla suite at every new major version they change everything and it's all entangled in a complicated framework that was not designed for Unix but for Windows so in these cases you can handle the su name specifically in Debian instead of as often recommended use only the static library for example yes it has a drawback it breaks binary compatibility with other distributions shipping the same library that's what you will be told but the answer is what's compatibility compatibility between distributions for Mozilla which is built in C++ and when different compilers have incompatible ABI's oh this is just there is no compatibility in this case so you can break it happily and you it's better if you break it on such examples for example Libxbecome.so from Mozilla was added a su name with a D at the end so that nobody ever thought of adding a D at the end of the su name so it keeps Debian specific for HDF5 I used the lib-release option which adds to the su name the version you pass it so you are sure to change the su name at every new upstream release another common mistake is to export all symbols that are defined in the C code because every single function and variable that you write in your library if it is not declared static it is exported in the end but you don't necessarily want this function to be part of the public application programming interface so the drawback is well you don't put it in the .h file so you will say that's not a problem nobody will use them that's just that's just wrong people use them as long as there is a loophole there will be application developers who will define the function themselves and then use it from the shared library and of course it is entirely broken and you have to detect such brokenness so here lib-tool can help a lot for example it has a nice it has nice features like dashboard that symbols dash regex you give it a regex of accepted symbols to export so if you have a well-defined namespace it's quite easy or if you have a list of symbols to export you can use dash export dash symbols you can also use directly the new LD version script feature that I will explain later and finally small issue that arises with plugins is that well when you load a library or any kind of shared object all symbols arrive in a global symbol tile table and there is no way to distinguish where they come from so if there are namespace conflicts because they can be caused by private symbols for example in this case you end up severely broken because you will end up calling a function that has the same name but which is in completely different software and it just breaks for plugins for plugins this is critical because the plug-in code can be added to any application with which wasn't written with the plug-in in mind for example if you use a GTK plus team engine or an input module it can be added to any GTK or no application in the wild and not all of them have been tested with all plugins so there is a simple option provided by the new dynamic linker against it it's dash the dash B symbolic so you pass it with dash L when you're going it with a compiler so this should be used for all plugins all team engines and things of the like so we are now abording a more complicated problem let's consider this package is a picture viewer written using GTK plus well you can imagine lots of such small applications it used GTK for the user interface and it used a leaping to read PNG files and however GTK already depends on the PNG to display its small nice big maps for the buttons for example now suppose the leaping developers change their RBI so the surname of the PNG changes and well well you say I rebuild the package against that new leaping e3 library but there is a time when when leap GTK is still linked against the former version and what happens when the application starts both leaping e versions are loaded at the same time and there is no way then to tell symbols from both of them and you crash so the solution is the symbol versioning so you have to use a version script which is used which is passed to LD with such an option and you get look like sort of something as simple as that this means you give version PNG 12 0 to any symbol hence the star any symbol in that library and you give a different version to the different version name to the different version of the library and as a result in one library you end up with a symbol same symbols have different names when you're using a library when you're using this library in the binary you see that the binary retains the version of the symbol additionally with its name so if now in the global symbol table you have two symbols with the same name and different version now you can distinguish between the two and you can link with the two versions of the library of course as a long term is another good solution but it allows for smooth transitions so to benefit of all of version symbols all package have to be rebuilt so what happened for the PNG which was one of the largest transition of the like this package with version symbols were uploaded right after the woody release and then all package depending on the PNG having rebuilt before such was released so in the such release we still have the two library versions but with all binaries having using this version symbols and after the such release we can remove the former version the older version and rebuilt all applications that used to link with that version with a new version and everything goes smoothly not a single bag happened another drawback that changes cannot be easily reverted because once a package has been built to require a version for a symbol you cannot tell to not require it anymore so when when you are going back you will get a warning but if you are changing the version name this gets worse because it won't even work at all if the version that is name isn't found isn't in the library so that happens if another distribution starts using a different symbol versioning scheme and that will also happen if upstream starts using a different versioning scheme that's what happened with my SQL the DBN maintainer forwarded patch with symbol versioning it was accepted and then the my SQL developers they said oh yeah this version is nice but we can we'll give another version which will look nicer and they broke the DBN package not knowingly of course but it happened to go even farther we can use the same version script to version the symbols and restrict the list of exported symbols with the global keyword you tell about the symbols that have to be exported and with a local keyword those that don't have to and you end up the clean library that upgrades smoothly finally I will talk about the inter library dependency hell in DBN let's see what happens when you're using lib tool to build library package which is the case for most library package now of course this is the case because it's very nice too it's integrated with auto com for not to make and it makes library building very simple and together with your library it adds a dot la file that generally we add in development package and it contains dependency information between libraries about all libraries you depend upon so sorry at at link time so that static linking works that la files are recourse entirely to found to find all dependencies that you have to link this is not necessary for shared libraries because we already have dependency system in the ELF model and it is already recourse at loading time so when you are in a static environment this is harmless to have extra dependencies listed but when we are in DBN it is a fast-moving distribution this is really harmful because if you have full which depends in turn on the bar which depend on the bath what happens when you link with lip tool you are adding a direct dependency from leap food to lip baths and it gets listed in the depends of your package now libates libaths has a migration you have now lip as one guess what happens you are now linking to two versions of the same library without any need for it and you have to rebuild the new the lip food package against the new lip baths and of course it's added complexity the DBN lip tool is fixed to not exhibit this behavior the dependencies are used only for static linking and when you are linking dynamically they are not recourse so we strongly recommend many people strongly recommend using the DBN lip tool so you have to rerun the old the lip tool it's process on your package of course it is annoying because you have to do this at each new upstream release and even worse it is generally not sufficient to get rid of all and the indirect dependencies and even better proposition is starting to be implemented recently is it is to entirely remove that la files first of all if you do this start with the leaf libraries don't do like the X people who removed the dot la files from X render on X cursor with hundreds of libraries depending on them and failing to build because they couldn't find the dot la files and if you start with leaf libraries and then go down you don't have any problems and it's recommended to only do this when the library use PKG config which provides another means to do static linking with all necessary information speaking of PKG config it is a tool that provides and that provides metadata concerning libraries and which allows to access them in a configure script provide a C flag to LD flags and dependencies which are recourse and of course we are seeing the dependencies is bad but there are there have been recent changes and they are accepted upstream so you can fix your dot PC files by using the requires dot privates and lips dot private fields for all dependencies that don't need to be linked in for shared building now I had some nice music but it doesn't start what happens when everyone everything else fails when there is no more hope when despite all of your efforts you still have lots of indirect dependencies then the super new linker comes to the rescue because it has a metallic option his name dash dash as dash needed and when you pass it this option all following libraries are only linked in if the binary uses actually symbols from them which means it will detect it will detect reliably whether you really need those dependencies and if they are added by something else like lip tool like PC config like this stupid upstream it would just be skipped so you just pass it like this unfortunately it recently stopped working for libraries because of lip tool which reorders the arguments and turns this option into a dummy one happily we have a patch that works so it's it has been submitted for inclusion into the dbm lip tool so hopefully really to losing your package should also make this option work anyway it still works for binary only package and I highly recommend using it also for binary package because it will remove dependencies if really to losing the package is not enough you can use it to remove a needed dependencies and simplify the release teams work as a conclusion there are no magic recypes for packaging shared libraries as you have seen there are there are a lot of lots of problems and there are no solutions that work more or the less when you are facing these problems however it shows also that release management is impossible if libraries are not packaged correctly so you have to deal with the release team to ask them for help if there is something wrong or to ask other people that also maintain child shared libraries for advice if you need it and even more importantly you have to forward patches upstream you have to help upstream understand your patches apply them correctly so that you you are not forced to use up to maintain lots of patches against upstream which is of course not a good course of action as well when upstream doesn't understand anything well just kick them and go away and you stay with it with loads of patches like for example Mozilla are there any questions I've read the advice before that the as needed option for LD should only be used as a last resort are there disadvantages to using that option sorry are there disadvantages why you would not want to use the as needed flag to LD there have been some cases on some architectures where it broke because it is not very widely supported by the binutils upstream also some people consider it as a last resort measure because it's kind of a hack but I tend to consider it as something you can also add on top of the working thing just like we use DH fix perms to fix permissions we can also use a hack like this to fix dependencies and there are also cases where you actually want to linking some unused dependencies but they are quite rare cases hello I am Junichi Uekawa and I maintain the library packaging guide hello and what kind of content would you consider to be inappropriate in the current library packaging guide that I maintain it's even a bit outdated I guess but there are any point would you consider to be wrong oh well there are many things that are good in the library packaging guide but in fact we could discuss it for for a long time for example I and of course it needs to be updated to talk about such things such improvements against library package but there are some things like always using by default the leapful 3-dev I don't think that's needed for all libraries we could we could talk about such things for quite a long time in fact okay I guess I will talk private that was a very long discussion you talked about what the symbol versioning does to the elf header but you didn't mention what the b-symbolic flag does does that actually change the elf header or I don't know what exactly that's to the elf header but when you use this the dash b symbolic option the functions from the from the plug-in in this case cannot be overridden by existing functions in the global symbol table as the as current LD works the first function to be used is the function that is in the program rather than the one coming from the plug-in or the team engine but yes if this changes it won't be enough because it will go on the other side the plug-in function of a writing of the programs function well if there are no more questions I think I will let the room for the next talk and I will thank you for your attention