 Welcome everyone to the workshop on Multi-Arch. Multi-Arch is a new and tricky subject. And we have Steve Langasek doing a nice workshop for us. So here you go. Well, it's not really all that tricky. It just completely flies in the face of everything we thought we knew about how we put packages together. It's not all that complicated really when you get right down to it. So I was talking with Wookie beforehand. He tricked me into doing this workshop. And so I was asking what he was looking to get out of it. And basically kind of the conclusion we came to is, we have this Wookie page on the debbie.org Wookie, which I'm more or less just going to be reading through and walking people through after which we'll take some questions and I'll show a few examples and whatnot. But, right, so starting with the basics. And actually I should find somewhere else that actually mentions this, which I don't think is in here. The first thing to talk about is what is multi-arch and what does it mean to convert a package for multi-arch? Increase the font in theory. Where does that setting live? Better? Good enough? More? So what does it mean to convert a package for multi-arch? So the basic idea of multi-arch, which if you were around for the last couple of hours when we were talking about how we're using it in cross-building, you already know all this. But in any case, multi-arch is the idea of installing packages for more than one architecture on a single system and structuring those packages so that they can sanely coexist together on the file system where everything that needs those files agrees on where they should live and is not confused by things suddenly moving them about underneath you to try to shoehorn things onto the file system. Instead we say, no, each architecture has its own directory. That's always where its libraries should live and everything should look for them there. And as long as the entire system agrees, it works great. And it allows you to mix and match libraries and packages of any set of architectures. So this is not simply about x8664 plus i386 or anything like that. It's much broader than that. People are using it for cross-compilation environments. They're using it for emulation environments where any binary you might want to run on any system, it should be possible to at least satisfy the library dependencies and install it on your system, such that provided you have some way to execute it, that the packaging system doesn't get in your way. That's effectively what multi-arch is about. So you get to reuse the existing packages from the archive of any architecture and you can install them with the support and integration of the package manager. And now as far as what it means to convert your package to multi-arch, I'm afraid I'm a little bit reaching here. I'm going to drill down a little bit here because I want to get a reference up on the page and I'm sorry I haven't actually prepared any slides at all for this. One sec. Right, so here's the key bit from the actual multi-arch package management specification, which is what does the multi-arch field mean? And this seems to be a tricky bit for a lot of people because we didn't user test our naming scheme before we deployed it. Yeah, we should have had some focus groups involved in figuring out what the right names were. So to try to clarify this in everybody's mind, because it has been a big issue for understanding what exactly it means. So multi-arch colon same means the package that's tagged multi-arch same is a multi-arch package. It's aware of this multi-arch system and it satisfies the dependencies of a package of the same architecture. Multi-arch colon foreign means it's a multi-arch aware package that satisfies the dependencies of packages of foreign architectures as well as those of its own architecture, which is kind of the reverse of how a lot of people try to think about this. So if there's one thing you take away from here, that's kind of the big bug number one in terms of how this has been doing. Okay, I was asked to give an example and I'll go ahead and do that. So your typical library package let's say libpng. You create a package that's multi-arch for libpng and it has as its contents. You've got your doc directory and then you've got files that are in lib triplet libpng.so.such and such. So it's a library package. It's clearly been configured to do the multi-arch thing in that its files live somewhere that you can have a separate set of them for each architecture. Now the question is, now you've got this package, what kind of dependencies, what kind of things depend on this? Well, the answer is things that are AMD64 binaries that will link against this library or will deal open this library. So anything that depends on libpng has to get the version of libpng for the same architecture. So that's what multi-arch colon same means. The converse would be something like... Larger? Is that good? Sorry? Okay. So looking at things that are available on the system here that are marked multi-arch for and let me find a good example that we can look at. R-sync? Was that on the list? Okay. So, R-sync. Yeah, it's been marked multi-arch foreign. And why would that be? So let's look at what the R-sync package contains. You've got user-shared doc, some files and etc. And the main interface that R-sync provides to its reverse dependencies is an executable. And an exec boundary is architecture neutral. So the kinds of things that depend on R-sync depend on it because they want to call the R-sync program. And it doesn't matter what architecture the R-sync package is or what architecture the calling package is. It just matters that there's something you can call. So that's a multi-arch foreign relationship where the thing calling doesn't care what architecture you give it on the system. So this is important, for instance, if I think this one in particular has been marked as multi-arch foreign on account of the Linux kernel source packages in Ubuntu, at least, build depend on R-sync. If you want to cross-build the kernel, you don't care about getting the ARM-HF version of R-sync. You care about getting whatever version of R-sync is most convenient that you can run it. So that's why you have multi-arch colon foreign. So you can express that and so that the package manager can work out do I have a version of R-sync available? Yes, good, let's go. Instead of trying to cross-install the other version of R-sync for a different architecture which then tries to remove the one you already have installed and then might try to remove other things that are depending on that as well. So that's multi-arch colon foreign is to say here's a thing that is architecture agnostic. It has an architecture. So it's not architecture colon all which where there are no binary files in it and you install the exact same package on all architectures. It has an architecture as part of its nature but the interfaces it provides are architecture independent and because of that we don't care for dependency satisfaction which architecture we have installed. We have a preference for which one we do install if there isn't one installed. We will always try to pick the most sensible one. The package manager will prefer whatever matches the native architecture if it can but for satisfying dependency resolutions if I go in and do R-sync, I have to get it installed, R-sync colon I386 this should work. It might not, let's see. No, so it has some dependencies whereas R-sync has actually been declared multi-arch colon foreign it has some dependencies on other things that are not and so it can't actually be installed in this case but the annotations are correct on R-sync itself it's just additional packages have to be converted for multi-arch before it's useful in the archive. That is one nice thing about the way multi-arch has been specified is that you don't have to do the conversions in any particular order so the fact that this package, the fact that R-sync has been marked multi-arch colon foreign is not a bug. It doesn't have to wait for base files to be also marked multi-arch colon foreign or whatever in order to be co-installable and actually I don't know if that's the correct solution for base files I'll just say it doesn't have to wait for multi-archification of base files before you can mark R-sync as multi-arch colon foreign. You can do these in any order, shove them in the archive it's massively parallelizable that's what we've done in Debian and in Ubuntu as far as converting libraries over it's write the patch, push the patch to the BTS when the maintainer gets around to uploading it and we've got a critical mass of them then suddenly things just pop and start working and we kind of have what's happened over the past year where IA32 Libs, I, let's see, I think we did a multi-arch library sprint for Debian back in December and basically pushed all the patches of the BTS and then I went away and did other things for six months and then I'd see popping up in June the discussion about how IA32 Libs is ready to go away and I'm like, oh, okay, that was easy. So, right, even though in this case I can't install the foreign version of R-sync in place of the native one, nevertheless, if I were to try to do a cross-build of something that uses R-sync, like if I tried to cross-build the kernel package and it needed R-sync, this would be sufficient to make it do the right thing or if I have other things that depend on R-sync, I don't know if there's any good choices in here I could try to install the foreign version of case splice. Let's see what that does. Oh, that didn't work. Dervish? RMHF is in the configuration as a foreign architecture on this system so I don't know why exactly, I don't know, maybe I keep finding packages that are just architecture all and that's why it's failing on me because they don't actually exist in app's view of the world as being foreign packages. So anyway, that's not a great example with respect to that but in theory if you had something that was an I-386 only piece of software that depended on R-sync, the R-sync package is all ready to go where you can install your I-386 binary on any system and the native version of R-sync will satisfy the dependency which is usually the way we want to look at what multi-arrange colon foreign is for. More often than not, in the vast majority of cases, a multi-arrange colon foreign package is the one package on your system you will never install the foreign version of because it's the one that's satisfying the foreign dependencies of the other packages that are interesting to cross-install and you generally want only the native version and that's how you say this one is the only one I need and you don't have to worry about trying to get the other version of it on the system. So a multi-arrange colon same package is one that's co-installable because all of the files are split into different directories and they install alongside each other on the file system whereas a multi-arrange colon foreign package is not co-installable. You can only ever have one of those installed at a time but multi-arrange colon foreign says whichever one you have installed does the job and so that's in a nutshell that's not a very small nutshell it's kind of a coconut shell. That's what a multi-arch field means and then we also have this multi-arch colon allowed thing which by and large I'm just going to ignore here on account of the fact that the Debian Archive actually doesn't let you do anything useful with it yet today although it's specified. We've got some changes that have to be made on the build these before you can do anything really interesting with that and so effectively today multi-arch colon allowed is treated the same as if you didn't have the multi-arch field at all because the packages that would make use of this have to change the syntax of their dependencies and that's where things get hung up because you have to have additional support in the package manager. So how do you actually go about converting a package for multi-arch? Well, most of the time the interesting case where we would look at this is for a library and so that's why we have some documentation that's been written up about how we do this for libraries if it's not a library that you're working with if your package is an executable the process for converting it for multi-arch is looking at it saying do I provide any interfaces that are architecture specific or can anything call my program or is anything can call my program? Well, then you ask does it have any reverse dependencies and is it actually interesting for something of a foreign architecture to call it and if the answer to that is yes you just put in the multi-arch colon foreign field in the binary package stanza of that package. For instance, here's one example from PAM the PAM source package which is a helper package which actually was split out specifically to accommodate multi-arch which I can get into in a little bit perhaps but the binary package it's a dash bin package implying that the interfaces it provides are executables and it's tagged multi-arch colon foreign and this little bit here is actually the only thing that that package requires in order to do its job in the multi-arch world the more complicated part of things is when you're looking at libraries and that's why we have this wiki page which I hardly encourage people to bookmark or whatever people do in place of bookmarks now that browsers have made those obsolete wiki.demy.org multi-arch slash implementation this page gives you what should be everything you need to know about actually converting a library package for multi-arch and the easiest way to do that by far is to use a dev helper dh1 with compact level 9 because that will basically do all the work for you and the only thing you need to do is let me find it this page is longer than I remembered maybe it's because the font is so much bigger so this is what it takes to convert a package to multi-arch using dh1 you have to bump your build dependency for the higher compact level there's this particular pre-dependency that you have to add which is relevant specifically for arcane reasons of us only having partial support for multi-arch in the squeeze version of libc due to some late changes in how we decided we were actually specifying multi-arch on some architectures so this is basically there to enforce that you're moving an upgrade and your libraries are moving all over the place you make sure that ld.so can find them before you start moving them because otherwise things like apt which use libraries might notice and have a problem or your shell or whatever it might be so yeah that's the other thing you do have to add this additional field of pre-dependence which becomes less important after wheezy releases then it resolves to a dependency that will always be satisfied by the wheezy versions of the packages so that's an upgrade handling thing there you say the debian slash compact should be 9 and provided you're using a build system that has standard ways of specifying paths then dev helper can do all the work for you and figure out ah we're compact level 9 let's put the libraries in the multi-arch directory for you currently I think the only build system that we have a reliable way to do that for is auto tools um I'm just gonna scroll here quickly through and yeah I think that's the only one that we have um now if somebody knows of a standard way to do this with for instance CMake it would be great for somebody to update the documentation we actually had a it just works so it just works so the troubling thing about that I think the CMake I think that implies that CMake upstream has perhaps encoded some platform specific knowledge and making some platform specific guesses about where to install things or that we have debian specific patches to do that I think right so it's relying it's using the CMake when you use CMake the build system logic is encoded in the CMake package which is installed as a build dependency and it has the patches that say um the reason we specifically didn't didn't cause that to turn on for debhelper um and have debhelper automatically put things in without you bumping the combat level was that uh it breaks first of all you have to add that pre dependency field which is a manual operation and secondly all of your .install files break because suddenly all the files not only have moved around so even if you're using globbing your globbing is going to fail so in fact I think in this wiki page we specifically say something something about why it's not automatic anyway detailed rationale is given in the wiki page but so basically I looked at that and said no we shouldn't have this suddenly happen out from underneath the maintainer and just have it happen the next time the package is built because there are these other things you are always going to have to touch so at least for auto conf what we do is with combat level 9 it passes the dash dash libdir option which says this is the library directory for the architecture you're building for and of course it has debhelper is great about encapsulating all of that and you can override it as needed and actually I will show you a package example where there are some deviations from the default and what that looks like in debhelper and it looks like anything else you would do with debhelper one with dh1 with small overrides or whatever so you've bumped debhelper you've told it combat level 9 that's why you have to do this next step of your .install files if it says use your lib currently it's going to have to be changed and if you are doing anything special other than just listing the files if you have to indicate a target of something to debhelper whether that's a target of a simlink or a target of a directory creation or a target of install you have to figure out how to substitute the actual architecture name in because obviously you can't use a glob as a target of a simlink or a target for installation so the way this is done is the recommended way is to use dhexec which is with combat level 9 .install file becomes a script that gets run and you can build depend on dhexec and dhexec knows how to pull out various common substitutions automatically for you so with no additional effort you do something like this and this is actually it doesn't show well here because of the font size but this is actually a single line where you're saying take the files that upstream installed into userlib directory such and such and this is the target directory which has the architecture name in it so that's how that's useful and as it says you don't even have to do anything special in devian rules because dhexec knows how to extract that information from the environment for you as needed so once you're taking care of your install files the next thing you do is make sure that anything that you're telling the build system about through devian rules and scrub userlib and replace it with the architecture string in practice most builds will automatically get this this variable from the environment already without any additional effort however policy does say that you're not promised to have that set because if you call devian slash rules directly as a script you don't get this in the environment which means that that's why 7 applies if you had to actually set that environment variable there or reference that environment variable you should actually make sure you've set it somehow by doing this a little bit there so latest d-package about three versions ago has an include architecture.mc which sets all those variables so maybe we should just say put this line in I hate make file includes as an interface right because they're mysterious and you wonder what the hell it did yes because there's no encapsulation boundary about anything it does and it's very easy to break them so I know that's buxy's position that Raphael says yes here's a make file snippet just include this I abhor that and yeah but yes you're free to do so they're your packages so yeah once you've gotten to step 7 there and done all that I mean step 8 is you just set the multi-arch colon same field and that's about it actually step 9 is kind of interesting we talked a little bit about multi-arch for and before there's a strange little question of backwards compatibility where because the behavior of the package manager to not go haywire with existing packages that are in the archive a multi-arch colon all package even though it says it right in the name that it's colon all if you actually want to use that as a dependency of a foreign package that you're installing you have to explicitly say that it's multi-arch colon foreign the reason for this specifically has to do with certain packages which are multi-arch colon all meta packages or defaults packages in fact is probably a good term for them so suppose you are depending on the package python which happens to be an architecture colon all package however what that actually is is it's an architecture all package which is masking a dependency on the current the current architecture any package that you actually are depending on so if you are up here and you're depending on python because that's what we say is you don't over specify your version that you want you just say I depend on python but the python package doesn't just provide the interpreter user bin python it also provides some library interfaces and if you're doing something with the python package and you're depending on it and you actually want the library interfaces it's not sufficient to just say yeah any architecture all package satisfies the dependencies because you can kind of get yourself messed up on the system on an upgrade where if the right set of packages happen to not be available to you at the right time or you do something extraordinarily convoluted with an app to get install manual install and not all the right packages have the multi-arch bit set you can wind up trying to swap out your python for the wrong one or say yes this python satisfies dependencies but it doesn't so because of that issue the sad fact is that 80% of the architecture all packages in the archive would need to be tagged multi-arch colon foreign because their interfaces are architecture independent but we have no way to tell apart the ones that are the ones that aren't so in order to not break things on upgrade and change the semantics in a way that's incompatible with the packages we already have you have to do this extra step so for instance if you have a libfoo-common package it's architecture all you still need to market as multi-arch colon foreign so that the package manager can look at that and say yeah okay you've looked at it you've made sure it's actually safe and we will use that to satisfy the dependency of each runtime version of the library that depends on it so I mean and this wiki page covers other scenarios actually I'll scroll up to the index of them there's instructions for how to do cdbs with auto tools and dev helper step one converted to dh1 classic dev helper not using dh1 or cdbs with auto tools with no helper at all cmake apparently doesn't need a section because it's all automatic anyway I don't know it's a wiki page please help extend it if you have other build systems that have sort of standard ways to do this sort of thing I'd love to have more documentation a lot of the people that we're working on multi-arch we're most familiar with auto tools as a build system and some of the that allowed a certain amount of bias to creep in in the implementation plan in the sense that we kind of forgot there were things that weren't auto tools that don't do as good of a job of doing build time detection of environments and then we discover that oh yeah we actually have a whole lot of other build systems we have to go out there and patch to find the libraries that we moved because we thought it would use the link the compiler's path to actually figure out where the libraries are and it wasn't doing that so things that one of the consequences of converting a library to multi-arch that was surprising to us at the time was that suddenly you have things like the PHP build system which walk the file system looking for the library you specified you could have just asked the compiler it was there and it would have told you but so be it and it PHP not to pick on PHP specifically it's just one example I think we had some issues with the Python build system as well and I those are two examples that come to mind but there were a number of others where we find ourselves playing a little bit of whack-a-mole where every time we moved the library we found a few more reverse dependencies that would stop building because they couldn't find the library now so that is one thing to be aware of so it does mean that if there are any members of the release team listening we should probably be careful about doing further library multi-arch conversions in Weezy without making sure that things still build afterwards and that they build and that they don't miss build so it's something to be be careful about okay so real examples of what things look like here I happen to maintain a couple of library packages myself free type is one of them and here's the here's the Debian rules for that we need to go up a couple sizes still so this is the Debian rules for free type you see that we are explicitly setting Debian host multi-arch as well as Debian host arch which I don't recall why that is there but I guess we will see as we scroll down other stuff that's not related here to multi-arch we're doing some different C flags and what not using using the great D package build flags interface there some stuff about symbol checking crazy stuff we don't actually like to look at because it's reminds us of bad times and then here we have the DH1 global rule here which just calls DH with all arguments let's see exactly why we do have Debian host multi-arch being set aha so here's an interesting sort of thing that's going on here it's being referenced in DH auto install we have an override rule for DH auto install so everything else is actually pretty much done it's work for us but we happen to have a special case here where we are stripping the dependency libs field out of libfreetype.la which probably can go away now because the .la transition is fairly far along actually in this case I could have used a glob instead and it would have worked just as well but that's the only reason that this particular one cares about multi-arch yeah that's the last occurrence in the file so the general bits are really just this right here because we're using Debian Compat level 9 that pretty much takes care of itself the control file what do we look at for the binary packages well the libfreetype 6 runtime library it has that pre-dependence we mentioned that gets expanded I will show you not on this machine I won't that's what that expands out to it's a debhelper variable that gets expanded to multi-arch dash support which is a twisty little transitional package built by glibc that ensures that you've got the right version of glibc on before you upgrade so that's the pre-dependency and multi-arch colon same means that you can have more than one version installed at time which I think in fact I do yeah there we go there's an AMD64 version and i386 version doesn't the fact that the pre-dependence version means that we have to keep it forever so it's not versioned in order to hide all the details of what versions are actually required it's I'll show you what the multi-arch support package actually looks like right the AMD64 version of it depends on this version of libc6 which you'll see this is actually a rather old version of libc6 which is satisfied from squeezy and squeezy squeezy and before now if I do it if I look at the i386 one however you'll see that it depends on a much newer libc because that was we it was much later when we finalized what version we finalized the architecture path we were using on i386 there was some back and forth discussion and we wound up changing our minds much later than we would have hoped basically because it was changed after the squeeze release this upgrade support is in place so the multi-arch support package is going to be around for a while we don't need to get rid of it however once Weezy releases we can get rid of the pre-dependence we can change dev helper to not add the pre-dependence in unstable because it's a no op it's just adding nodes to the package dependency graph that don't really do anything different they're already satisfied on any system so yeah we can phase that out the actual pre-dependence misc pre-dependence feature of dev helper now that it's there I think it's actually been used in a couple of different scenarios as well so if you've added the field to your source package you might as well keep it but down the line I think we'll see it falling out of the binary packages sorry that's the runtime library package nothing too fancy there it's just multi-arch same means you can have more than one of it what else do we have we have the lib free type 6-dev package which is not currently marked multi-arch colon same because of a little thing called free type-config which we discussed last hour free type 2-demos they're demo they're just programs you can run that are examples I have not bothered to mark this package multi-arch colon foreign why? because there are no reverse dependencies in the archive and marking it multi-arch colon foreign only matters if it has reverse dependencies so although I could install a foreign architecture version of this package there's no reason to market multi-arch colon foreign because nothing depends on it so it would be useless information so actually here why don't I show you that I can in fact install now this one should work you should be able to install free type 2-demos i3a6 because the library it's running out there and grabbing it and you see it's even though I'm on an AMD64 system but the package that's been installed here is the i3a6 version of the package and it installs just fine it doesn't have to be marked multi-arch colon foreign to do that your leaf packages that are actually for the average user the most interesting thing you would want to install foreign architectures of so for instance Skype which only has an i3a6 executable anywhere in the world that package the multi-arch system is designed such that those existing packages that are out there that Debian doesn't necessarily control have any influence over the binary package control file those should install just fine you only have to annotate the dependencies the packages being depended on are what have to be annotated to make this work so free type 2-demos supposing that were a non-free binary package being distributed by a third party it had to have no modifications done to install in a multi-arch scenario and have all of its dependencies satisfied free type happens to be a library it's a fairly core library so it has a udev that's used by the installer we don't do multi-arch in the installer because why would you ever want to have multi-arch in your installer if you need more than one ABI in your installer I think something's wrong um at least I hope I'm right about that um but uh so in fact most of the udev packages still install directly to userlib and slashlib instead of using the multi-arch directories they don't have to but it's just that some of the tooling around the installer is rather domain specific and there's lots of special cases of libraries and so it's just simpler to put them in the the userlib path I do know in the early days of multi-arch it did actually cause a problem for the installer I think mklibs might have had a problem with them not being in the right place or something so udevs generally you don't do anything with those at all um let's look at the .install file there you go you've got that extra glob there saying grab me all of the .so run time libraries in sub directories of lib because they've all moved down a level um so like there's userlib on my system which is rather full um but userlib actually let's loop this way there's as much stuff in the sub directory of userlib for the architecture as there is in userlib at this point um there's quite a lot that's been moved over to that sub directory um I think actually I should probably stop there as far as rambling on up here and start taking questions on account of the fact that we are running low on time so yeah five minutes so what questions can I answer for people about multi-arch who in here is a library package maintainer okay put your hands down if you have already converted them to multi-arch okay so you should have questions the four of you I was waiting for somebody to ask an easy question um the library package um that I'm thinking of is auth bind is what? it's an LD preload hack yes um so uh if you install it you probably want let us say both the amb64 and i386 LD preloads but you want the binary the sort of little wrapper binary and there's a set you would help a program so I'll have to split it into two packages but how do I make sure that all the .so's get installed good question um multi-arch has failed to solve that one because it was not an issue we identified earlier in the process but I think you were part of the mailing list discussion on where that question was being asked it's something we should solve because it doesn't affect just not LD preload hacks but like for instance PAM modules, NSS modules any sort of plugins that you're installing you generally want to try to install those for as many architectures as you can and that you're using on the system at the same time what bodge would you recommend in the meantime? I don't have a recommendation or I don't know I think there were some ideas in that thread I'd have to go back and look at what was suggested there but I do remember some people came up with some kind of hacks that might have worked but the right answer in the long term is that we need to have the package manager have some smarts about this and it might require additional annotation for the declaring that hey this is a package that I want my brothers to come with me when I install it so that's going to require additional information because you know you can't tell just by looking at the fact that it's multi-arch colon same that that's the case it's not so much a question but it's more a result of all the changes that go through with multi-arch we don't explicitly support building packages from Weezy on squeeze we have ways of doing back ports and things like that but if you're doing a back port from this kind of situation you have to take the changes to the install files out before it's going to be able to build on squeeze do you have to take out the multi-arch lines in demo control as well or will they just be ignored by a squeeze I think they're just ignored aren't they I think you'll get a warning from D-Package that it's unused but it'll be thrown away and it's probably a good practice to take them out anyway if you're doing a back port you definitely have to make some source changes how many or few of those you choose to do I don't know I'll repeat the question so the question is will it break the multi-arch upgrade if the fields are there and I believe the answer is no so first of all I don't think the squeeze D-Package puts it into the binary file the binary control file I think it spits out a warning and says this is an unknown field so I'm ignoring it second of all if it were there the package manager in squeeze I don't know why I keep saying squeeze in squeeze that'll be the next release name how about that I'll talk to Neil about it afterwards in squeeze doesn't understand that field at all and so just ignores it so it doesn't have any effect on calculating the dependencies and you're using the squeeze version of apt when you're doing the upgrade the upgrade is done with the previous version of apt if you're doing it like a normal person so yeah and even if it were in the new version it's not like if you're doing an upgrade you don't have any packages of a foreign architecture currently installed nor do you have any architectures configured in D-Package so there's no possible way that as part of the upgrade it's going to miscalculate anything and pick packages from the wrong architecture because that requires an explicit action by the user to enable multi-arch and just maybe adding to the question of backports the debhelper backport even if you keep the compatibility has the multi-arch bits taken out yes and that's necessary because if you did do a backport using a debhelper backport that enabled the multi-arch bits and built it on squeeze it would be uninstallable in i386 because the glibc is not new enough in squeeze any other questions? what happens if you're installing packages that puts things in etc and say you have the armhf version and the i386 version presumably they have to be in lockstep versions so the cases that you have config files that vary by architecture that don't vary by architecture so the config file is the same for both arm and i386 in the package don't change anything because the d-package implementation of multi-arch says that if the file is the same in both packages you share it on the file system it ref counts it and that's it and what happens if they're different? if they're different then you would need to somehow change the package to look at an architecture specific path for it so I think there are some examples of this with maybe they've all gone away maybe they've even moved to slash user share I do remember there used to be at least some oh I think maybe it's pango is that no anyway there have been examples where a library has a config file which requires architecture specific information and the solution has been to encode the architecture name in the directory that it's using and just split the hierarchy within etc there's no multi-arch has not attempted to specify to that level of granularity what the correct thing is so it's you know make it up as you go just make sure it's unique and that they can co-install and I'm being told time is up so thanks everyone for coming hope it was helpful to you