 Thank you. So shall we get started? Can you hear me? Is the microphone working? Maybe if it isn't. Yeah, good. Cool. So I want this to be the sort of boff where I say as little as possible. Because this is my first depoconf and I've not been doing Java packaging very long either. And I say the more everyone else will get out of it. And I think. So I put together really one slide other than the title slide of things people suggested we might want to talk about. So I also, I put those things on a gobby document on gobby Debian org. So maybe if someone here could sort of scribble down what people say on that gobby document then we'll have a record afterwards of kind of what was said. So kind of those were the things that people sort of wanted to talk about. The reason why I proposed this boff was that I foolishly tried to package a Java application. And I actually went like, well I was about to start doing this. Russ, and he said, yeah, I tried to do that and it was just too much like hard work. And my experience also was that it was too much like hard work. So I have, you know, I ended up with ten libraries to package. And then the, so I got all of those and packaged them and then I never quite finished the app itself because, you know, so I got so far behind everything else. And I think it struck me that there must be a better answer to this because, you know, one of the things on that list is a problem with manpower. And if everyone who tries to come to think, well, I've got this Java app I use, it would be handy if it was in Debian. They start off on this process and lose the will to live halfway down it. Then we have this kind of vicious cycle where each new potential contributor gets stuck, loses heart. So their application never makes it into Debian, which is a problem. They probably go off and do other things and we have fewer, few of those dependent libraries packaged up. So the next person who comes along, there aren't so many libraries, so they have to package more libraries. These sort of self-reinforcing problem. And I think that Maven actually makes this worse. I've come to loathe Maven more than I loathe almost anything else. And I think as a distribution, it presents a number of problems for us. Ignoring the fact that this default mode of operation is not terribly secure, you know, it'll download any old garbage over HTTP and compile it and install it on a system for you. I think it encourages upstream authors to be kind of slapdash in their approach to dependencies, the libraries they depend on. If your expectation is either everyone will just install the binary jar, which is a common model, or they'll just run Maven and it'll download everything. You don't care about using sort of popular libraries in the way that someone writing a C application might. There's a lack of clear sort of versioning, you know, you declare a version dependency or some other library, or maybe you wanted a particular API that's provided by a particular version, or maybe you just picked the latest one when you happen to write your application. And then when we come along, we don't want 50 different versions of some Java library, so we often end up trying to have to pretty much guess where the API has changed in the versions of a Java library. So that makes the whole process harder as well. Yeah, so I think this sort of whole approach that Maven encourages is really unhelpful for us as a distribution. And I wonder if there's anything we can do to try and ameliorate that. And Andrew, is Andrew here? Hi. Andrew's Google Summer of Code project I think might be a starting point for that because do you want to say what you did? There's a microphone. Tell us what you did. Save me talking. Hello. Is this on? From my project, I was creating a Maven plugin to actually build a Java application and iterally build all of its dependencies and dump it all into a Maven repository. I mean, it struck me that this sort of... because we have MH make, which is pretty good, but it's not perfect. And I wondered if we could... some modification of that approach, that sort of automatic finding all the dependencies and building them, we could use to make the job of someone who comes along with their new Java application they want to get into Debian and makes the job of sorting out all of the libraries and getting them into distributable form easier. And that might solve some of our problems both in terms of manpower and make people's lives easier in terms of getting Java applications into Debian. Someone say something. Disagree with me. Excellent. Matthew, I'm so glad that you brought this up. Could you please explain to the group how you propose that Debian support Maven 3 coexistence with Maven 2? No. I don't have an answer to that question. So, as I said, I'm not sure I'm the right person to be coordinating this. I sort of did it because I thought it would be a useful discussion to have, but I'm not an experienced user of Java. I don't really develop Java at all. It's just I had a need to use a Java application for my work. And the Shibboleth team already has everything except the Java app in Debian. And it would be useful if we could get the Java app in as well. So that's sort of where I come to this from. So if you ask me kind of detailed questions about different versions of Maven, I'm not the person to answer them. Just a minute. All right. So I'm Tony Maxwell for those of you who don't know me. And I've been on the Java team for a while. And I have to, I also have to claim certain ignorance of some of the inner workings. And we have the Maven repo helper, Maven Debian helper. And it's a little bit of black magic for me even having worked with them for quite a while to get those to work. And I think we're at the point where to support Maven 3, where we really need to back up and retool and reevaluate this. So we're talking about a couple. There's a few issues, right? As one, and I think, so my answer is we already have to support, we have to support Maven 2 and Maven 3 build systems if we're going to continue to distribute Java apps within Debian. Yeah. And coming to this conference actually, I didn't want to put it up there for topics of discussion, but some days I feel like we should distribute JDK and we should distribute maybe Tomcat, maybe if we're willing to do the security support and then helpers to allow folks to basically run their Java apps on top of Debian because this cascading effect of every library that anyone has ever used is arduous. It takes a simple, what's, there's some open CA, some nice CA tool and it had 45 reverse dependencies and we don't know how many of those also had reverse dependencies. So the interesting thing is of course the problem isn't really numerically all that terribly different from the world of C-based applications. The distinction is that most of the libraries that matter to people in the world of C applications grew up through the history of Unix and lots of people over a lot of time have thought about what's the right way to package libraries. We transitioned to shared libraries, we figured out shared library versions. Just a lot of infrastructure that's grown up with the application community and so new people coming in who want to write a C app sort of have this body of well engineered infrastructure that's grown up over time. The problem that I've seen is a lot of folks that are writing Java apps while the resulting apps are really cool and the one that Tom and I sort of got entangled in his open rocket and a really cool rocket design and simulation program and a reverse dependency list just about killed us. Then of course Keith and I our rocket ground station software is all written in Java these days so this is why I'm hearing why I care about this but my observation has been that just in the world of people writing Java apps there's nowhere near the sort of focus on distribution, distributability, packageability and all of that because the model really is here's the URL to download the latest JAR just run it and I in fact have given up and for open rocket I'm now delivering an installer that knows where to go get the latest JAR and download it and you know wrap it so that it can be invoked and gee you know it works and my life's gone back to being less painful but it feels like it feels like treason somehow this isn't how this is supposed to work. It's actually I'm sorry to hog the mic but it's actually more complicated than this because we have upstreams that have this post open source attitude that they will just shove bits on to GitHub that build with Maven and they're happy when they create a POM XML that will build their thing and you're actually lucky to have reasonable version numbers and releases and if you have version numbers and releases there really isn't much consistency in Java developers using semantic versioning to actually represent ABI changes absolutely so this makes Tony's problem even worse because not only do you have the large graph of reverse depends but each project has a reverse depends on different versions of libraries and there's almost no path to resolve that except hand to hand combat every time. The way that I resolve this in Java packaging is actually to have a SED script that runs over my source code and renumbers the library every time I make an ABI change and so I literally edit the name of the library and append a number an ABI version number and ship multiple libraries I mean I looked all through the documentation is there really no ABI versioning in Java at all? Who could build a library system like that in the year 2000? I'll just comment I gave a talk on this in 2012 and it's this, you know so Gosling's dream has come true right at one to run it anywhere right? So the point is basically once it builds that's one of the issues I think with struggles with Java as a language once you get it to build you know that's it that's what Maven is there to solve is hey let me be able to reproducibly build this at some snapshot in time and there is no conception of let me have just one copy of this library on my system and move it forward I'm sorry I don't have time to move forward and let me upgrade that I'm supposed to wrap my arms around the whole system and build it all together well I think it's more of a container view where basically hey this app runs with this set of libraries and that's good I just wanted to comment that at my company we definitely tried to fight this battle some and we had given up and basically just each application ships its entire set of dependencies and the premium that it used to be runtime requirements are also not that bad you can make an argument for security issues I'm not sure where we stand on that but I see a lot of value in the package the JDK and package Tomcat and do nothing else approach to things because it's a lot of efforts that when people get down and use these things in anger and alternative approaches just to get their job done it's kind of a council of despair though isn't it how do we solve the job of packaging problem we don't the security thing that someone alluded to at the moment is an issue so I looked at this application I was trying to package and see on some SSL library that some PhD student had written back in the day and he was he's going to get it into security commons on this gov and yet it has forgotten and here it is doing SSL and something that is a security relevant application this sort of maybe encourages you to just say at one point I'll just carry on using it till the cows come home and if someone finds the security bug in it well it herbs one kind of related minor thing we might want to consider is I believe past few weeks Sonotype has now enabled support for HTTPS for downloads from the main repository so it might be handy for our tools for people to switch to use that as a default because it's not the upstream default yet I don't think for Leiningen or Maven or whatever I expect for Leiningen it probably will be soon if it's not already but I don't know about Maven that's it Rob brought up a really good point which is Leiningen is basically the tool built on top of Maven for closure applications and I happen to be a closure developer and I really enjoy closure and so I started talking with some people about well what should we do for Jesse well what we really ought to do is get Leiningen into Jesse and we started down the trail of doing that and then Emmanuel brought up the Java 8 transition which you have in your slide which is a big issue we tripped over the Maven 3 thing which is another issue and then the reverse depends for closure to get Leiningen to build but I think that this problem isn't just Java it's also JVM languages so this comes up with closure it comes up with JRuby potentially Scala and other things as well and I would guess that many of those Java languages are probably using Maven or in the case of closure their own build tool which is sort of a layer on top and I don't have any answers but it certainly brings up a good point okay I certainly don't want to one comment and then I'll I don't want to hog the mic I guess my look I mean I realize this we could move in a lot of places with this talk I mean I don't want I would like feedback and I think Debian Java is the best place to provide it certainly there are folks who aren't here but feedback on how we continue to make Java within Debian what's useful sounds you know sounds like supporting JVM run times development tools possibly I mean I think applications gets pretty hairy pretty quickly writing up policies for how to package Java applications just telling us how to do it right now the instructions are the instructions are make sure all of your library the current Java instructions are in Debian and then build your application on the top of the Java libraries in Debian and given the ABI issues that's clearly just not a supportable plan Open Rocket is an example here we have you know BDEL tried to get all the libraries in Debian and the ABI instability across versions was a disaster I've tried to do the same thing and ship you know one library with multiple applications and every time I rev the ABI I just I can't do that having a policy down that says the Java distribution policy is you package all the libraries except for this tiny little set of ones which are guaranteed to not change an ABI that we will supply package them in this way put them in these directories and then your application will work and just abandon the notion of having common shared Java libraries in the distribution That's quite a radical suggestion I think it might be the right answer because currently our other Java policies like the C policy if there's a library we should have more version on the system because as you pointed out we solved all these problems in C years ago and it's clearly the right way to do it but maybe that's a fight we can't have with the Java world because they don't give them monkeys We don't have we don't have shared libraries in Java What does this mean? But does this mean are we going to insist on building all the jars from source or not? I think we have the whole point of free software is that you distribute the preferred form for modification I assume you wonder if you're starting to talk about I just wanted to make sure that's still what we were talking about In what sense if you're not building from source in what sense do you have the preferred form for modification I think, sorry, do you speak? Right, so I'm a newbie when I first started to learn Java I used Linux to do it and I got in the habit of adding dependencies right with my package and I really thought that was the way to go because reading about Maven was just too complex for me to want to I didn't have the time to do it I didn't have the knowledge to set up Maven and stuff like that and the wiki page was too complex for me even to touch upon so that was my issue I just wanted to comment I do think we do always want to build from source no question asked but I think there's something to this idea of if you have these set of libraries that belong to your app to a lot of other apps maybe at some point they'll be promoted but maybe there's room for us to distribute some jars provided they build from source that are part of your class path that aren't part of the system's class path so I'd welcome discussion on the list about possibly it's not really relaxing Java policy but it's changing Java policy to support this for me personally I think it's untenable right now to continue trying to support this because invariably you're well you're robbing someone one of the issues we run into the Java team is we're pushing new we don't have visibility of everything so we're trying to we need upstream version X of package foo that needs version Y of bar push that and it turns around if you don't build all the reverse depends you know this isn't so our next summer of code project I would love a tool that just said alright I'm proposing uploading this build every single reverse depends because invariably then and then there's a we have to quickly we have fail to build from source and then we end up spending a lot of time just trying to essentially now we're reporting those ABI changes within the team and that's the team is not staffed for it a problem that will arise from this approach at the moment you know if we find a security problem in a particular Java library well it's easy we fix that Java library and you know where you go so if we're going to start shipping entire apps we're going to have to have some sort of metadata in the app you know in the dev file or somewhere that describes the versions of all the libraries that are contained within that app both and partly so this problem and a security issue we can fix that but also then that will give us the tools to say well actually you know we're shipping 20 apps all of which contain this same library maybe at that point we should start thinking about taking that library into its own package. How do we plan to deal with the issue of the different packages depending on different versions of the same dependency? Do we allow them to be co-installed? The problem I see is that if you allow them to be co-installed you easily end up with 5 million copies of the same library because some program won't be whatever be changed or in the next years be changed to use the new version that would be easier to take away but yeah So is the answer to that we need to push back on upstreams more to make them produce higher quality code? Do we have that power? We can try but I think I think this is kind of the crux of the problem is it's unrealistic to expect we can change the Java habits that are there but it is really frustrating hold on a second I'm torn because in the one hand I see this idea of well let's just ship open JDK and then go to town the thing that scares me among other things like security is that this is kind of breaking our relationship with our users. It means that you can't just say up get source and get something that you expect I'm wondering if I know this sounds kind of unrealistic but I'm wondering if there's a way that we could you know with a policy change like insisting that every Java library that we use has some sort of auto package test thing in it that we basically have some magical tool that will like examine the ABI of Java libraries download every version and automatically determine when the ABI changes and then you know help us do this basically automatically so humans don't have to be involved that would be cool so I just I keep reminding myself that the fundamental problem is a clash of world views and that the objectives of what appears to be sort of the center of the behavioral model and the Java community is not the set of objectives that we have had traditionally in the Debian project with respect to how things get packaged how they get distributed you know the notion of never shipping more than one copy of a given piece of source code built on the system as a shared library or something that world view I mean you know is Keith's point out the situation with the support for ABI versioning you know you just look and there are things that just aren't there and the reason they aren't there isn't because those people are stupid it's because they have a different set of objectives and that wasn't necessary to meet the objectives they had is that good for us? No to your question is this something we should be pushing back and upstream about? Hell yes because you know I think that the model that we have evolved of sort of how we care for source code and how we deal with security issues and all of these things that we use in Debian is immensely powerful do I think we actually have the power to you know drive change in the Java community? No I really don't and so where does that leave us? It leaves us having this kind of a discussion off it's a we think we know what they ought to do to make the world a better place and we have absolutely no idea how to actually affect that change so where does that leave us? Well as you use the word magic in there some magic tool I mean it would be nice to have some magic tool but I don't know how we make that come into existence there is a tool called CLEAR a C-L-I-R that will do some of this other packaging problems that I've come across one of the other areas where we have a similar problem is shipping multiple versions of GCC targeting different architectures I built a cross compiler environment for ARM that's in the archive the way that we resolve that in Debian today is we actually have the build process have a build dependency on a source package would it be appropriate for us to allow in the archive innumerable different variants of the same source package for different libraries and have perhaps a Debian specific numbering of the ABI's of each of those packages so that when you build your package you're saying my package needs this library and I need this source version of that library and as part of building your package you incorporate that source version of your library and build a JAR from that and make it a very lightweight process of the new source version of a particular library and it's like well I know it's not and probably not make the ABI any sort of monotonic progression in time just say yeah I downloaded this new program it used this random version of this library it's not already in the archive what's in the archive doesn't work for me I'm just going to upload a new source version of this package and give it another number that would have a couple of effects it would allow you to share the same source library across several package that you maintain if it's the same version it would also allow us to identify rapidly locations in the archive where we have security problems and we can upload security fixes and it would be a tremendous burden on the Java team to actually go through and say oh we have 47 versions of libSSL used by different Java programs I have to go and do the same security fix to all 47 versions it would be a mechanical process at least because you would the code would presumably be effectively the same everywhere but it would be a larger burden but at least it would allow me to upload the sources that I needed to construct my package to make sure that we had sources that corresponded to the binaries in the archive yeah I do you have a question you were waving your hand out there I'm just going to comment quickly I'm carrying the mic over here that's a little bit of a maven that's kind of getting closer to the maven model we'd have our own repository of lots of versions possibly the same library but I like that it's interesting yeah I mean I was thinking that I don't I don't know that we have enough leverage to be able to you know push we have a limited amount of leverage but I think it's very good for us we may have being a reasonable decision to be have some stronger influence over time and there probably are plenty of upstreams that follow and there's a number of other versioning processes that are just fine and so for those we might not have too much trouble handling things the normal way I suspect that given the way the JVM works there may be tooling ways we can help but it sounds to me like maybe over time what we need is some sort of hybrid approach possibly where you know there's the obvious way we want things done and we try to encourage people to do it that way but there's a lot of packages that upstreams that aren't following that model we still have a way to address it and in particular be able to handle the security issues carefully that kind of thing all right if anyone else I'm going to bring up a couple other topics before you do just to try and sort of summarize that a bit to make sure I've understood what everyone said and I think we're going to have to go back and talk about this on Devi and Java because we're essentially proposing a policy change and we've sort of two alternatives I think have been suggested there's the alternative where we ship OpenJDK that's about it there's the alternative where we carry on with our current approach which I think is clearly unsustainable and there's an approach where we we move towards the point where if you want to ship an app in Debian it's okay for that app to be built using a pile of libraries rather than having the sort of have one copy of this library on the system instead you know either we have some sort of mechanism with different source packages in the archive that you incorporate in your build or something like that where you have an app that definitely builds from source but it comes as one giant jar rather than you know with its own class path my feeling is that latter approach is better for our users it means for common applications people will be able to use it on Debian and I think it will make our lives tractable in a way that our current approach doesn't just a quick that was just a straw man saying yeah no shot I think we needed some sort of combination between those two because there was one case where you have just a pure Java library and it then can just incorporate everything it magically finds something somewhere else but there was also a case of library just happening or programs just happening to have some features in Java and in my case they just incorporate the code from Apache Commons and just build it into the jar and be done with that so what that package does now is to use the normal packaging way and if we get to the source code solution you need to invent some mechanisms to get the source code into that program so I think it's not a good way to change the complete way but have a mixture between so I think shall we leave the dependency health question for now because we've used most of our time in the word there are some other issues up there so Tony do you want to say something I think you are going to I just wanted to bring we just kind of spitballed on the list to figure out what will we talk about during this I think the dependency health question is more I'm glad that we spent a lot of time on that I just wanted to bring up so Java 8 transition I can't give a good overview of that other than the main struggle there seems to be that wouldn't it be great if we had JDK that built across all of our architecture and just just one or two or even just three would be great actually microphone sorry just in the spirit of getting to all the architectures it would be a lot easier if we only had to support one version of open JDK absolutely no argument there I think when we brought up the Java 8 transition I mean I don't my gut feeling is for Jesse we I don't suspect it's going to happen maybe if folks really wanted to happen and everyone gets really super busy we have two months right it doesn't look tractable here unless somebody is suddenly going to have a burst of enthusiasm on that front but let's talk about it on the list it's what remains to be done I just wanted to know what remains to be done for the Java 8 transition I haven't been following it closely the main problem as far as I followed it is that some random package does not be a twist to ADK 8 maybe because there was some javadoc issues or some code issues or whatever so I think Emmanuel did file a pile of bugs but I'm not sure whether it's all bugs existing part of the answer to your question can be found on the . Emmanuel created a bug to track javadoc transition things some of them are kind of benign there's one little Java library a leaf library that needs to be updated but there are significantly more complicated cases like j-ruby I've taken a couple of steps at updating and packaging j-ruby but that's tricky because there's native bits, there's it used to and I've been working with upstream because I know upstream to get some of the binary blobs out of their thing but still we have to, I mean it's a really tricky package to do the problem that they have gems which is a whole other thing and so there are some really tricky things for example to get this done to do it right what we really want to do is we want to get an updated version of j-ruby that builds against JDK8 and then that will solve that particular problem but that's an example of how the transition is going to be tricky so to be continued javadoc transition talk so with Tomcat I just want to list this one explicitly because we've gotten to the point with Tomcat it's a little bit of a deviation from the Debian model in terms of supporting stable where we're no longer back porting so we no longer back porting everything to say Tomcat 6 6035 instead in fact we're actually proposing uploading 6041 to old stable and I'm talking to Holger about that and so we reached this point I just kind of wanted to bring it up and let folks know we reached this point the security team needs these updates and we're currently the way we're staffed with manpower we it seems less dangerous to go to the new version but that feels odd to me having been that hasn't been the Debian way in the past and so I just wanted to bring that up as a topic and maybe hear comments and ideas for that and one other thing because I've been hogging the mic so much but in terms of getting more contributors those in the room and those listening to the stream we make it sound so bleak and all I think it's a blast it's fun, these are interesting problems these are some real kind of gnarly engineering problems plenty of meat to sink your teeth into maybe Andrew wants to comment on that so if you're interested Debian Java at list.debian.org please feel free, we welcome all types of contributions there's healthy, there's a lot of sponsoring going on there and I'll let anyone else who would like to speak I mean this business of not backporting security problems it makes me unhappy as well I speak from a position of massive ignorance because I don't use Tomcat at all but from every other package I've maintained whether it's security problems we always backport them but I guess we have to do something for our users and in the end if that means moving to a newer version it's better than not fixing the bugs at all there is precedent for it for a while years ago I used to maintain the bind packaging domain name service stuff there were at least two times where some issue upstream resulted in a new release and not from which it would have not been easy to backport a specific security fix and the discussion and resulting decision sort of came down to were you did you have a higher degree of confidence in the quality of solution you'd be delivering to our end users of trying to generate a Debian specific backporting patch which might or might not get a lot of testing and attention or accepting the fact that there was a large intense very technically astute community working to ensure that a new upstream version really did all the right things and was it better for our users to just put that in these are never easy discussions and I would not begin to suggest that our stable release managers will ever be happy having to have that conversation but there certainly have been times in the past where it was clearly the right answer to just accept a new version and put it into a stable update or put it into a backport depends how compatible it is but are we going to break everyone's with Barton you know if you break everyone's name service they'll be quite unhappy we've got five minutes so I think any more comments I'm just hoping that the gobby document now has all of this sort of stuff in it and maybe we can get at least get any executive summary up there and then send that to the Debian Java list so the people who haven't been here maybe we've talked about some quite radical policy changes we should try and get a summary of this discussion on the list so everyone else can particularly Emanuel who's not here does an awful lot of work and I'd want to keep him on board Absolutely, I agree. I was just going to touch every single point up there because I'm Tidey. All of the corrupty packages I actually had an email exchange with Emanuel regarding this and so the take right now is as long as there's not an RC bug we're just going to continue to bring things forward but that we will draw belief packages with extreme prejudice if they end up taking too much time it's one of these things I got real active with the team in 2010 and that was when Torsten Werner was giving his talk about the dependency hell and there was I think leading up probably 2008 to 2010 was maybe the heyday of Debian Java where a huge number of packages were added and there was a lot of support perhaps maybe even some day job corporate support I'm not sure but one of the issues we have right now with the team is there are a lot of packages that the active members on the team haven't actually worked on bugs get filed that's great another way to help that's awesome thank you to all of our bugs submitters and a lot of times they come with patches too I appreciate that but you get a patch and then you realize I don't even know how this build system where I mean it's about enough half the packages are CDBS, half of them are dead helper half of them are maven, half of them are ant and then half of them are super homegrown so and maybe well I just want to kind of share that as part of this buff is that at least in private email we kind of agreed as long as we're not going to proactively look for packages to cull but I think the count now is about 780 packages that are Java team maintained and that's for the size, it's hard to judge the size of the team but that's touching a fair number of packages every week I think for folks and also quick, well we're right at the thing thank you very much Matthew for organizing this and getting us together well thank you all for meaning I didn't have to stand here until talk all afternoon any more last thoughts anyone I'm interested in the reproducible build effort and one of the things that is on the list of things that aren't reproducible are time stamps in jars and time stamps in javadoc generated documentation would there be any real objection if we made these or if we attempted to make these deterministic based on like the stamp of the change log no I don't think so just engineering time yeah I think we're out of time now so thank you all for coming I think it's been quite useful we'll have a discussion on Debbie and Java about some of these ideas in due course