 So, I think it's time. Thank you for being here for what I think is almost the last talk of the conference. I've been told I have to explain that joke for the stream, so that joke is actually a French mistake that originated in a murder case where the person who was murdered actually brought with her own blood pointing to the assassin. It was Omar Mathieu and the promise that is that it has a French mistake that she would never have made. So it pointed to a person who went to prison and apparently later in the history of that case he was freed hence the title that at least I found funny. So this talk is about why I tried to kill the LSP, so I'm going to see some details about that. But first, who am I? Regarding LSP, I'm maintaining the LSP package since 2012. Since 2010 I'm the Deben printing maintainer, so founder and actually the more or less alone package maintainer. And since March 2015 I'm a member of the Deben technical committee and the current chair. But let's go to the beefy part of that talk. So we're going to talk a little about the Linux standard base or familiarly called the LSP. But what is it really? So I thought it would be interesting to go look what the Linux foundation actually said about the LSP. So the goal of the LSP is to develop and promote a set of open standards that will increase compatibility among Linux distributions and enable software applications to run on any compliant system. Even in binary form we'll come back to that. In addition, the LSP will help coordinate efforts to recruit software vendors to port and write products for Linux operating systems. It's actually quite old projects. The first version of that standard was published in June 2001 and the last version more recently two years ago in June 2015. It's actually a wide set of standards. I didn't go and print despite my vast interest in printing. I didn't go and print the LSP standards because it's actually a big, big book. But the important thing there is to highlight what actually the goal is in terms of further Linux foundation. So the goal of the LSP is to enable software applications to run on any compliant system even in binary form. And that's actually the interesting part about the LSP. This is just a graph from the LSP documentation or the LSP Wikipedia page I think that explains that we have everything on the Linux kernel side that they can break this thing internally but they guarantee stability on the API side and we kind of have the same for the software and that was the previous thing they didn't want to have and then they could have OceanBuilder, Siemens, Enix, Katya5, Maya, etc. that could be compiled against LSP and run on any Linux OS Alpha, Bravo or Charlie. The LSP itself is five different aspects actually. It has the five system hierarchy standard, things about the system initialization in which Debian has an opinion there. The LSP release, the binary compatibility itself and the Red Hat package format, we'll come back to that. So I'll go through these five aspects so that you understand the coverage of the LSP. So the FHS describes what we want to see in all LSP compliant Linux distributions. So these are the ones I want to read them because you know them. But where LSP especially pick is that it describes quite in detail exactly what we want to see and where. So we have of course Home and Root and Mount and SRV and slash OPT. But then in slash USR we also have all these plus, so I think I've met all of these but then you have things like slash var slash lib slash hw clock. I didn't know that existed. So it describes for each of these directories what they are for, what the rights are expected on these directories, if they are mandatory or optional on a Linux system. And of that in Debian we actually amend it because there are parts of the FHS we don't necessarily agree with or we have special use cases. For example MuleDiarch, we have different exceptions to the FHS. So Debian is not FHS compliant. Debian is Debian policy compliant. And regarding the versions, we currently have the version 2.3 with some amendments. If you fancy that, please go read the section 911 of Debian policy. It's actually probably an A4 page of exceptions. And there is an open bug to actually migrate to the 3.0 version, bug 787816 if you're interested. So let's follow the files. The files is an area where it's quite easy to agree. Let's move to init systems. So there is a whole chapter about init system. In particular it's chapter 22. And it describes the cron jobs how they work, the init script actions, the comments conventions for init scripts, etc. etc. And that's exactly what you would see for example in EDC init d cups. And it's these provides cups and it requires syslog and remote fs to start. It should probably start the network, the avahi, the demon, the slapd, etc. etc. So that's what we had in cc init. And that's the comments that were actually read by the in-serve to actually try to do parallel booting. That is not LSP compliant but it's quite faster so we do that instead. But it also describes a set of functions that you can use in an init script. The goal of that is that you enable, therefore you enable someone who wants to write an init script for a proprietary system to actually know what interfaces they can use. So you have startDemon, killProg, pdfProg and various log messages. That's for the init functions. If you want to go read the LSP please do it. There's also LSP release, that's actually a binary that you can run. And if you run it, it will tell you what LSP modules you have, what is the operating system you're running in, what is this reader id, what release, what codename. And it's widely used in virus scripts to actually detect if that shell script is running on Red Hat or Debian or which version. Actually nowadays you should really depend on functionalities and not version numbers but that's still what everyone does. But LSP release is just one of the binaries that you need on an LSP compliant system. Actually it says it goes through a very, very long list of binaries that are mandatory on an LSP system. Batch, PC, I didn't know of call for example. But yeah, I didn't go through all the list. So actually it defines 62 mandatory commands. And then we have the ABI, so that's the more complex stuff. Binary compatibility. So it goes in details in multiple base libraries. So you have VIMC, LibM, LibD thread, LibGCC, etc. You also have utility libraries, LibZ, LibN curses. And you would think, oh, that's enough. But it goes on and on. So you have the X network security services. You have LibX11, you have LibIce, etc., etc. So actually there are quite a lot of libraries that are mandated. And I took one specific example so that you understand why it's quite boring to read the standards, but also in which detail it goes. So if we take LibZ, for example, it says in chapter 15.2 it says this is the name of the library. It has that surname, and it has these various interfaces. Adler32, Compress, etc. So if we go see what Compress does, it says, so here comes the function synopsis. So it starts with a bytef destination, a ulongf destination lens, etc., etc. It has a description and some constraints on what the output does, what the input does, and what exactly the function will do. And it has this verbose descriptions for all the functions in all the libraries that are documented. And if you see the numbers, it's actually 25,000 constants that are described in the LSB standard. It's 36,000 binary interfaces that are described in various details, etc., etc. So it's quite a lot. And apparently there was hope I found this article the other day, New Slash from IBM in 272, the problem solved. Linux has taken the next evolutionary step beyond the source code compatibility of Unix instead of porting or rebuilding source code from one Linux release or distribution to another. And this has achieved binary compatibility between them all. So we could have just stopped in 2002, right? But of course there are other things that are slightly more confusing for people like us, is that the LSB mandates that you have an RPM binary that works on your machine. So it does not require the implementation to use RPM as the package manager. It only specifies the format of the package file that implementations must have some method of installing and forming packages. I could go in much more details about what the LSB is but you're probably more interested in what we have done actually in Debian. So in Debian, we've had multiple packages to actually try to go in the direction of having LSB support. The first one was Alien. Who has used Alien recently? I'm surprised. So it was introduced in 1997 just in time for BO and it converts RPM packages to dead packages and I think it also converts dead packages back to RPM packages. And its point is that it also will try to do things with its dependencies and depend on the LSB core, correct version, or LSB, this correct version. But we also introduced the LSB package in 2002 in time for Moody. Version 3. So in July 2002, for Moody, we had LSB110 at the time and there was a separate binary package for LSB reuse. And there was just one LSB binary package. This package provides an implementation of version 111, 110 of the Linux simple base for Debian on this only architecture with the Linux kernel. But I have highlighted that part I think it was in the news file in the changelog. It's one of the first compromises that Debian has done about the LSB and we'll see more of that later. So LSB11 assumes a 24 kernel, but Debian ships a 22 kernel so we're not compatible. But we still have some sort of layer to actually try to do it but we're not. In search after a very long freeze, June 2005, we had LSB20 and LSB release was reintegrated in the LSB source package and we had LSB, LSB base, core graphics, and CXX. And it was the first appearance of that longer message that says that this intent, the intent of this package is to provide a best current practice way of installing LSB packages on Debian. Its reasons does not imply that I or the Debian project believe that Debian fully complies with the LSB and it should not be construed as a statement that Debian is LSB compliant. We'll come back to that later as well. And as a slight diversion, I'll also take a little time to talk about the LSB base package. Who has the LSB base package installed in the machine? Well, you should all raise your hands, you will see why. So this package only includes some unit functions that are mandated by the LSB. So a little shell library may be used by other packages initially initialization scripts for console logging and other purposes. And it came along with, it came along well with CCD unit in September 2005. CCD units started to use the LSB base functions and therefore dependent on LSB base. And in February 2006, it actually modified the init script templates that everyone was copy and pasting to create new init scripts to actually use the init functions. So if you take a look at the popcorn graph of LSB base, you see that at some point there is a change and that kind of means that since 2009 everyone has LSB base installed. As soon as you have one reasonably complex init script and or CCD units installed, you have LSB base. Which makes it a quite scary package to maintain, I can say. So going on through the releases in Devin Edge, for zero in April 2007, we introduced new packages, LSB desktop and LSB Qt4. And in 2009 we had LSB 32. And at that point in time kind of started the long and slow ossification of that package because it was quite actively maintained in Florence before, from 2002 to 2011 after Stretch Freeze. But he had other things in life and couldn't continue to maintain it. Squeeze, yes. Squeeze was frozen in 2013. Oh, yes. And there's a mistake on that slide as well. That's what you get when you die in history. So when Squeeze released in February 2011 we were still... So the only support we had was LSB 32 despite the fact that three years before we had LSB 4 that was released. So there was basically three years in which nothing happened on the LSB front for Devin. And as you know we have a famous distribution who was already maintaining a different pile of code to try to achieve LSB compatibility on their own. So Ubuntu, since 2004, had people working on an LSB compatibility layer. So in 2009 when we had version 3.2 they had already the 4.0 LSB compatibility layer for Karmic. And then came Hope. Or I just adapted the package. So in 2012 I saw that the LSB package had some bugs so I started to do some enemy use and compare the package for Wheezy. Whee... Many mistakes. And one of the... Part of my job was to backport some of the Ubuntu modifications. And of course we had different opinions of how pile of proc should go find its little bits here and there to find what the pile of process should be. And I don't think we ever managed to actually merge that properly because it's quite hard to test all the little corner cases. But we backported quite a lot of the Ubuntu modifications in particular things like the LSB invalid MTA to make sure that you don't actually have an MTA on your machine but you can pretend that you do. And as I said before, as everyone has an LSB base it's kind of scary to upload a new shell init functions thing that you have tested on your neural machine. And I'm not a super shell script man so I was kind of... It's kind of scary to like deeper that and hope that not all the people come to my home and try to find me. But it's a very good learning experience and being able to make a change on so many machines at once is interesting. So part of the work I did was to migrate to LSB41 I also added tests for the Python code because it was not tested. And the left info blocks. Does anyone know what I'm referring to by left info blocks? Yes. Actually it's a thing I'm quite proud of because it was a very visible change in Weezy so that's what we had on other systems. So you notice the green OK boxes on the right. And we figured it's not OK on the right it's much more fun on the left. So the reason was people were starting to have big screens. So we had a boot log that had like 20 cm between your message and the error that was kind of confusing so we just said probably it's better on the left. It's not a big change but it's also funny to see that appear on various people's systems. But there's also the time for the first compromises as we saw the other one before about the kernel. Actually Q3 was not released and went in Weezy but it's still mandatory for LSB41 which was the LSB release at the time. So that was the news entry that was explaining that we had an explicit and they've been specific derogation from the LSB41 specification and if you wanted to have an LSB package that needed Q3 you needed to go find it in Snapshot. But if you look at the dates it's 2012 and Q3 was deprecated since 2005 and that's a quite long time in the history of the software. I mean, seriously. And then came disappointment. From the distribution point of view LSB is interesting because it's a single go-to code for a system initialization. It was before SystemD of course. Because you had one place where you could put logging functions and pdf proc things and all the things that everyone has to re-implement on their own in one single place. So at one point in the package history we started to add more functions there and tried to convince the various maintainers of init scripts to tell them look don't do that complicated thing on your own just use that code. It's tested, everyone uses it. So what LSB plays in init functions are interesting for that from the data point of view. We're also shipping something that everyone expects on any Linux system. That is LSB release and it doesn't necessarily make sense to remove it. And we have some packages to serve proprietary somewhere. LSB core, LSB desktop for example there are various packages out there that just expect that to exist. But then came a new init system. You init system, gee that's... I'll fix that. So system became and it's not at all mentioned in LSB. LSB assumes that you have sysv init. But if you have sysv init present but it doesn't run sysv init then the LSB kind of conflicts with that. So you're supposed to keep sysv init around or keep it running while Devin was starting to move towards system. And for real, I mean I searched and I really tried to find LSB compliant packages. The only ones that are of reasonable use by people that use Devin or even to is Google Earth and some very old Epson printer drivers. And I'm yet to see any other. And the interesting question there is currently the distribution of proprietary software happens in different ways. If you work in the web you see that you can get things piped to the bash a lot. That's scary. But trust me a lot of people do that. And actually when people want to ship software what they do is they do that package that kind of runs on an old version of Ubuntu and or an old version of Devin and they do an RPM that will mostly work here and there. And there are some containers. No check certificate. Ah, yes. Actually the no check certificate is disappearing slowly and slowly but it was in the documentation all the time. It's HTTP. Oh, you're right. That's because example.com doesn't have HTTPS. So the problem is that we've kind of done half the job and we're mostly there but the FHS has exception in the Devin policy that Devin is not LSP certified. So what's the point of all that? Talking about the certification there is a distribution checker and apparently it's possible to run it, it runs test for 10 hours and will test all the binaries and all the things. But the real question is to be asked there is for what purpose would we do that? And have other distribution done that? No one does, why would we? So I checked and apparently there are 3 LSP40 certified distributions, Oracle 6 HAL6 and Ubuntu 9. That's 8 years ago. And only Red Hat Enterprise version 7 is LSP41 certified and Ubuntu stopped being certified and Oracle has not tried to get certified either. So when it came to me I figured rather than just taking LSP away I do it the Devin way and try to discuss that. So I had a first buff at Deppcon 14 important and from the room what I got was no one cares just remove it. So and then it took a long time for me to actually write my thoughts to Devin Devin, Devin LSP and the answers were if I got them no one cares. So I figured we should pull the plug on that. And here's actually my thoughts that were taken also by LWN nothing is still valid the crux of the issue is I think whether the whole game is worth the work I'm yet to hear about software distribution happening through LSP packages back then when the LSP certified applications list was online I tried to check it this morning, it was down there were only eight applications by six different companies ever certified for LSP and only one of these was certified for a version of LSP that was older than four. So the three points there is that Devin doesn't verify LSP compatibility and has never done so and a part of the argument there is that it's fine to say we have a state of distribution that is compliant but if we have to do a security upgrade and it breaks some ABI somewhere for a very good reason then we would instantly become non-compliant so we would have lost all the work because we would have valued security over LSP compliance so it's kind of pointless if we cannot guarantee that over the course of any security problem we would stay compliant furthermore only Rana has doing it and whenever we talked about that at Devin.Devil, at DevCon on Devin LSP, whatever other channels on the LWM comments no one except expects Devin to actually do it another problem is that it's not like we are big team maintaining an LSP compatibility layer across the archive, I was the only one and I'm not interested in LSP because I didn't have RC bugs so I figured I should still leave the door open if there is interest because that's how Devin works rather than just opening it I could just hand out the package to someone saying look I can help you do the licensing effort so I wrote to the Devin.Devil and Devin LSP saying we could be certified I'm welcoming patches but I would not lead the effort I'm gonna adopt it, I'll help you get up to speed with that and you can do it and I waited three months and I received zero responses it's way too short, yeah right so I took my chainsaw out and I removed all the packages we had besides the ones that I could not remove LSP base will need a long transition period before we can remove it and LSP release kind of makes sense and embed it in plenty of scripts all around the place so it doesn't make sense to remove it so the answer from the Linux foundation or the answer from the LSP side was oops was nothing so that's the end of the day but unfortunately we had to somehow plug the plug back because of the things I mentioned before it's kind of nice if we can install it on the stretch stable release and the option drivers are proprietary we don't necessarily want to have a sort of script that takes them and un-mangles and re-mangles then to make them installable so we figured if there are depths out there that depend on the LSP core then we could do something so I held above last year in Cape Town about that and so for stretch the idea is that I reintroduced a new package that is LSP compact with a version that is not linked to the LSP version on purpose and the idea was that we provide LSP core in LSP but we just add the libjpeg or this dependencies that are enough for the depth packages that we know so there was a news entry I think that said if it doesn't work for you and you have a package that depends on LSP then we add dependencies so that was like we fall back to the minimal layer we can find and there was a question are there others that needed that and again no answer so yeah it's not the best technical solution because obviously the best technical solution is that Google says oh cool we don't have to depend on LSP we can actually depend on the correct version of libjpeg or the correct version of libjl or something LSP software so it's not our problem so there was and then I emailed the announce for this on the list and I was I didn't want to say the LSP is dead immediately so I said we're not throwing all of the LSP overboard we're still firmly standing behind the FHS and our LSP script mostly conformed to LSP and I was expecting some answers to that but I was I wasn't really surprised but just support basically people saying yeah it's fine so yeah I mixed my slides so the answer from the LSP was so the questions is also what happens next so in Buster I had a image of a dog here anyway it was a cute dog so the idea is that we hit the final maze nails in the coffin for the LSP I just made that LSP based on essential so if you need it just depend on it and it will be installed but it's not installed by default and I just removed LSP combat I'm not very good at that but maybe I should communicate slightly more about it rather than just removing packages and have a chain claw entry there's the idea of having communication in the release notes maybe I think there was something in the stretch release notes about it will be deprecated soon but maybe we should communicate more about that maybe we should also talk to Google and Epson saying look your packages not going to work on the next table release so please fix your things and maybe we should communicate to the world or make sure LWM just notices and does an article for us I don't know so back to the introduction actually that was in the talk description easily moving LSP making LSP obsolete or is that the final nails in Debian's coffin so my opinion there is that the LSP is long dead despite good intentions it just didn't survive reality that is for applications as we said before free software applications are just not chipped using LSP anyway you would use a PPA if Ubuntu has that package you can use snaps, slap pack app image and all the recent things to have applications distributed or you can still use the WGIT thing without the certificate checking also system is now reality and I think it has a mention in the LSP 5 but it's not yet the way the modern way of doing in-it systems containers I mean if you want to have Googlers probably the best way to do it is like find the docker recipe for that, launch it and it just works 5 minutes of downloading things from the internet and it just works and the point of having an LSP core that depends on a very old version of LeapJPEG and that you have to take Q3 from snapshots is not really practical then on the side of distributions there is the point that the stimulus is what is in production so we are not building Debian to fit whatever LSP thinks is a good Linux operating system the LSP is trying to encode in a document what our experience has shown is a good Linux system so it comes to the point that the standardization process for the LSP has been way too slow it would have been ideal if they had multiple updates in a stable reuse cycle so we could have considered like if we had a vivid conversation with them saying we will have that in the next stable release so if you could update the standard then we could be all on the same line but it was quite slow binary compatibility we all had fun about that the standard talk is a chimera it's just we know how to build software and make it run and we know it's kind of hard to make that run everywhere but there are surprising things that happen apparently nowadays you can run Ubuntu via binaries on windows so it's not impossible and there is also the point that we might have foreseen initially is that corporate back distros have a direct interest to keep their clients captive that means that canonical has an interest to have people actually use PPAs and use things that work on Ubuntu because that makes it more popular RedHats or Fedora have an interest to make sure that people can easily create RPMs because that makes their distribution more popular and no one is really backing the idea of the LSP with how good it could work and innovation was not happening on the LSP side, it was happening in these distributions and PPAs are a good innovation for that the other side of the coin is that we don't need the LSP Devin does not need the LSP our role, the debilith role in the Floss ecosystem is established now maybe it was not at the time of LSP 1.0 but nowadays people have heard much more about Devin than they have heard about LSP applications Ubuntu with its PPAs have also vastly democratized the Devin packaging many more people actually do Devin packaging, they might not do it as we would like them to do it but it happens and we see binary packages on PPAs and then you have all the automated build-ins and you can just install that it mostly works and this fantasy of having one single LSP world on top of all the distributions is just not how it works nowadays so you would do either both in a deb or end an RPM package or you would do either of those but it's a reality now that you have to do one deb and one RPM at the very least if you want to shift to users so some clothing thoughts I think I'm quite on time I'm surprised maybe I'll talk too quickly so giving up on the LSP I think is a way to push everyone to package for Devin directly so for free software it's obvious we have to bring people to Devin, they package their thing and we ship that as a stable release but that's our view of the world so it might not fit all application shipping purposes we need to be supportive of the modern ways that are worked on to ship software to our users and LSP is not that nowadays we have to face it LSP is not that modern way it kind of have never been we could have been if there were a much more closer relationship but it has not been it has always been lagging behind what we were doing on the innovative way to ship software but there is still this concern we are not bringing the Devin magic to fast ecosystems to the NPM ecosystem to the Python ecosystem we just depend on other external ways to ship applications to our users and we just live with that so as a provocative final thought I think it's important to remind that Devin is not about TPKJ or APT it just happens to be our best implementation of our dream vision now but I just tried to formulate my thoughts then I think it's important to remind that Devin is about crafting the best technical arrangement to ship the free software that satisfies our criteria as widely as possible if that's currently implemented by TPKJ and APT it's fine but we need to have the mental agility to think about what are the best ways to actually provide Devin free software guideline approved software to our users in the most efficient way and that's all I had to say thank you for your attention Hey DDA awesome talk, great story good description of what's happened you have compelling arguments I think you're doing exactly the right thing really good stuff I do have direct experience of the LSB process myself when we were bootstrapping RHF three cycles ago I got involved in a discussion as to whether or not we should add ARM as an architecture to the LSB because some white people might need it so I got involved in some of the regular conference calls we started working through the process and that was when I realised that LSB was dead it was basically doomed to there were a few reasons and you've already picked up on several of them Red Hat were paying lip service and nothing more they didn't want to make it possible for people to run proprietary apps on other systems they wanted people to run one is all one red hat there were never enough people involved in the LSB to be able to keep going so it's not just that they don't care they're not talking to us just like you're a single maintainer I saw there were essentially three maybe four people involved and they had a vast amount of work to do so when I was looking at it they were talking about QT4 and then QT5 with this vast amount of work to do knowing full well that they'd never get there it's a kind of shame it seemed like a really nice idea at the time when they started in 2002 it's dead let's say thank you Hi Didier I concur with Steve I too have a history of sitting in on some different skulls and I can only say I don't know what his time frame was I think it was a bit after me but it certainly looked the same there and a lot of what slowed them down was the fact that they were also trying to go to ISO and synchronize with them and the ISO has its own massive bureaucracy to deal with and it was a handful of people at LSB trying to cope with that but the main point I wanted to emphasize here is that I've seen a lot written on LWN in particular about how to kind of to seeing the praises of D-Package and to a lesser extent at it seems like a lot of these groups that seek to circumvent the TDM of going through a distributor as for instance OpenSSL famously complained about they end up tackling the packaging problem because by God they're going to do it right this time and they rapidly run into the same problems that RPM and D-Package ran into years ago and found solutions for so perhaps there was an interesting talk the other day on containers and flat packs and stuff like that but perhaps what we can best do to interface with these communities is not so much find ways to repack what they're doing but to help communicate the knowledge and hard one wisdom we've obtained by attacking these difficult problems just to point out one of the most notorious is that the package dependency graph is directed and cyclic we all those of us who study CS in our undergraduate years spend a lot of time with acyclic graphs but cycles are a real pain in the butt to deal with and that was one of the big problems I believe Jason Gunthorpe had to solve with apt that's just one example of the kind of real painful experience that we've learned from and solved and that maybe we can help these other communities and in the in the meantime perhaps a good relationship with them will lead us to a closer technical working relationship so that things are completely gross for people wanting to run npm or pip on a devian machine yeah thank you very much what is the status of snap packages in devian I don't know I think from what I've heard I think you can find snapd packages to install on devian and then run snaps but I'm not sure it works in main now but if anyone from the room has more information there oh welcome my understanding is they can sort of work and like most of the bits are in devian but the sandboxing is like absent because it relies on getting a farmer upstream I could be wrong about this this is secondhand snap is not the one I work on but that is my understanding okay thanks Manny I had two questions for you so thanks for this nice talk the historical side of the story was quite interesting once you get your slides fixed I didn't know about the distribution checker so for now we rely on the auto packages tests on hundreds of users to test the distribution etc did you test it and if yes did it help you find bugs I've looked at it and probably spent 2-3 hours figuring out if I needed a VM or in what way I could just not explode my laptop with that because by definition the distribution checker will do also the things across your machine and then I stopped so I didn't invest more than that but I'm looking forward for anyone to do that so as you said, LSB is there you're not willing to invest more time in it so I won't be losing your time with LSB so I had another question I know you're the TC chair you're not eligible yet but will you try to make yourself nominate to next DPL directions I think we're out of time I'm sorry to answer that difference to that question I'm not sure how is the best to address that I think in conversation we should all have to find technical ways to bring the VPN good things across the space fair question Simon something I think we can potentially learn from the LSB is one of the things they do in the distribution checker is checking the behavior of various functions and I've like seen some of the other side of this from like the NAMM community for instance because GTK is one of the libraries they test and some of the tests for GTK it turns out the LSB checks that there is particular behavior and the GTK developers look at it and go that's not what we meant, that's the problem we should fix that but so if we're testing for particular behavior we should like talk to our upstreams and check that the behavior we're asserting is actually what they wanted yes I would welcome more automated testing good thank you everyone for your attention and have a nice trip back because soon to end