 Java Park, we are going to have now Matthew Johnson and Niles Hiker, we're gonna talk about packaging with Java Helper. You must probably know Matthew because he gave the previous topic for lunch and presented. So he has a long-term contributor to the Debian Java team and has contributed all these great tools we use. And Niles is also working on that and also not for his work on the Eclipse packaging. So, without further ado. So, when I started packaging Java software, I noticed there was quite a lot of work required and not from any tools to support it. So I wrote Java Helper. Java Helper gives you tools to help you with the packaging workflow all the way from processing upstream releases, which, with a lot of Java software, is more tricky than one would like, right through to installing and building the resulting dev. So, I'm gonna go over a bunch of the tasks that have specialist Java requirements and how the tools in Java Helper can help you do that. And then, also a quick summary of how to use Java Helper from CDBS and DH7. So, firstly, upstream. The problem with a lot of Java upstreams is they give you their source as Java files or as zip files, which obviously we can't put in the archive. A lot of the time they will have bundled the jars that they're from the third-party libraries they depend on. They may even include all of the built-class files and the Java doc that they want to ship as well in the source is all in a single bundle. And if you have to repack this, then you're gonna have to do it every single time there's a new upstream release. So, in order to help with it, we have JHRepack. So, JHRepack is something you can give to Uscan, so you can put it at the end of your watch file list and if you do that, Uscan will invoke it for you every time it gets a new upstream release. Or you can obviously call it by hand with the same syntax. So, what JHRepack will do is, the first thing it will do is it will take whatever archive format upstream has and turn it into a TGZ. It will also go through and remove any of the jars and class files that are embedded in it. And if there's a Java doc tree, it will remove that whole tree and try and clean up all of the directories. So, with any luck, this is sufficient for you to get a DFSG free toggle, which you can actually use as a source file for your package. Now, once you've got your upstream toggle, there may be some things wrong with the build system. So, the first problem is if you've stripped out all of the third-party libraries, the build system is still going to expect you to have them in the tree and not in user-shared Java or wherever else that you're pulling the packaged dependencies from. The build system might not actually work. This is sadly a lot more common than you would like. Oh, it might not even exist. There's a lot of Java upstreams where they ship you some Java files and they say compile them. And they don't give you anything more of a build system than that. And then the build system also might not produce the Java doc, which as I mentioned in the previous talk, we would like all libraries to provide Java docs for the people using it. So, there are a couple of tools here which help with these problems. The first is JHlinkJas. This basically looks through your list of build dependencies and finds all of the jars that are contained within them and simlinks them into the directory where upstreams build system is expecting to find them. This may give you some extra jars that you unnecessarily require links in there, but hopefully the upstreams build system will ignore any of the ones it's not using. And you can do this in the normal DevPelper style by putting a file in Debian directory, just mentioning where you'd like them linked into. And if you do that, one of the things that Java helper has is a clean command which will automatically remove all of the ones there if it's mentioned in Debian link jars. If you don't have a working build system at all, and it may be the case that it fails the same build system test rather than just doesn't work, perhaps actually what you wanna do is just throw it out completely. So jhbuild is designed to, as a simple point and click, I've got a bunch of Java files here, please compile them all and put them in a jar. But for a large number of packages, this is actually all you need from a build system. So there's a selection of ways to call this. Again, there's Debian slash Java build and if you have this file you just list the jar file you want to be produced and where all of your source files are. And it will find all of the Java files and compile them for you. One of the things you will need to do is to set the Java home and the class path because we require all Java packages to build with a known good VM all of the time. So you have to declare which one that is and it can't figure out your class path. Having done that though, it does do a number of useful things for you. It will put the class path in the manifest of the jar and it will also build the Java doc for you. All of the jars which you require to build. Do feel free to chip in with questions at any time I believe there is a mic around. So, that's build systems. If you find that you actually have a reasonably complicated package, this may not be suitable for you. It is designed for the common simple case and in that case you may end up writing an Antwerp Maven build file which you keep in the Debian directory in order to build the package. But hopefully not too many of them. Now if you do have an upstream build system that works at least to some extent, it may miss a few things which actually in Debian we would like. I was talking in the previous talk about whether we should require library packages to declare their class path in the manifest entries. And that is generally a good idea because it means that your R depends don't have to know about your dependencies. However, most Java upstreams are of the opinion that if they release some code, it's in a jar, they've told you what jars you need and perhaps maybe in upstreams world they embed the jars into all of their third party libraries, they don't need to have a manifest entry. So a lot of Java upstreams are missing them. So in order to make this easier, there are a couple of tools for doing that. The Debian specific manifest entries I will refer to later. So the simplest one is JH class path where you basically just have a Debian class path file where you list the path to a jar and the list of jars that it depends on and it will update the class path entry for you. In some more complicated case, maybe you need to set the main class because this is going to be executed with Java dash jar. You can use Debian manifest where you basically embed a manifest entry for each jar that you want. Again, you can just invoke these without any arguments and then if there is a Debian manifest or class path file, it will pick those up. So it's perfect for using in the normal Debian helpful style. So once you've got past building upstream, Java policy gives you a number of requirements for how you install things into your package. In order to avoid code duplication to support all of these, there are a couple of useful Java helper tools as well. So in addition, we may want to change some of these requirements future in policy and if there's a tool that does it for you, then we only have to rebuild packages, we don't have to make source changes. So the first one of these is JH install lips. The policy currently requires that if you have a Java library then at least the top level jar you expect people to link against, you need to put in user share Java and it has to be of a particular format. So package name-version.jar with a sim link from package name.jar. If you just simply list some jar files in Debian slash Jlibs, then they will all get put into the correct location and the version number appended and the sim link set up. Not only that, it will attempt to strip out from the jar file name, things like upstream versions and so on. So it will do quite a good job of guessing that upstream's built this jar file name but we should install it like this. If it will use the package version and strip off DFSG suffixes and things, if you find that you've got a case which is more complicated than the automatic detection works you can give it a regex or help it along if it doesn't can't work things out by itself. The second one here is jh install javadoc and this has three forms. The simplest one, you tell it where your javadoc has been built and it will install it to use shared doc, your package name slash api and it will also generate doc base entry and install that as well. So you get it registered with doc base automatically. If you want, and actually this is one of the more common cases that you want the api to exist under the part user share doc and the, was the doc in a separate package. So live through Java rather than live through Java doc then you can give it the path that you actually want to install it to. And finally, if you've used jh build in order to generate your javadoc, if you say internal it will go off and find the javadoc that it built for you earlier and install that in a normal fashion. So now that will get you as far as giving you a binary package but your binary package at runtime may need some help. Every Java program needs a wrapper script. The kernel can't execute Java code natively. And in some cases you may want to load libjvm.so in your own application, not so much in a Java application but in something which can embed Java inside itself. Now the problem with this is that this is shipped in an architecture specific directory which doesn't use the same architecture names as Debian. So there are a few tools here which provided to help you with that. Oh, I don't have a separate slide for that. So jarapper is a bin format miskelper for jar files. If you set your jar file to be executable and depend on jarapper, then you can invoke it exactly like you would any other program and the kernel will go and run jarapper with your jar as an argument. And jarapper calls java-jar. And in order for that to work, you have to have a, all of your class path entries in a manifest and you need to have a main class attribute to tell it which class is the or entry point to your program. And as I showed earlier, that you can specify some arguments to JH build and it will set those up for you or you can use JH manifest and it will fix the manifest entry of your jar file for you. If you're in the unfortunate situation, which a lot of people are, that you need to pass extra arguments to the JVM or you need to use a specific JVM, there are also a dev-in-specific manifest entries for those which it will unpack and then use that in order to invoke the JVM on your jar file. Go and have a look at the documentation if you want to know what those are. It's documented in the jara-java-helper docs, yeah. Now, one of the other tools I have is JH exec. Now, this makes it quite easy to set up an executable jar file. It will, package build time, scan all of the places it expects a binary to be. If one of those is either a jar or a sim link to a jar, it will set that executable for you and then automatically in your build tree. And then as we'll see later, this will automatically generate a dependency on jar wrapper for you. So, if you install your jar as normal and use DH link to put a sim link into user bin, everything else will happen for you. Java files.show is something which is shipped. There's two versions of it. One's shipped in java-helper and one is shipped with jar wrapper. If at runtime you need to know, get, for example, the path to libjvm.so, you give it a java-home and it will look at your devian architecture and work out the path you need for that. There's also a version which is a make file snippet so that you can include this at build time if you needed it at build time, which is shipped with java-helper as well. So, now, dependence lines. After having put all of your libraries in build depends and put all of them in your class path and put all of them in your manifest entry, putting the list in the menu depends line is also a little tedious. Fortunately, we can do something about that. So, there's also a few other things you need in your depends lines like some alternate depends on different jarris and virtual packages. And if you've got an executable jar, you also need jar wrapper in your dependencies. So, we have jh-depends. The advantage of jh-depends is this is a single place where all of this happens for you. So, you don't have to write the code and if we change any of this in policy, it'll all happen automatically. So, jh-depends will do a number of things for you. First thing it does is it looks at all of your jar files. If those jar files have a class path entry, it takes the jars listed in there and works out which packages contain them and then it puts those into your depends. In fact, it populates a variable called javacode on depends which you can then put in your control file. The second thing it will do is if you have an executable jar or you've called it to tell it that you need to depend on a runtime, it will automatically figure out which virtual package you need to depend on based on your class file versions. And it will also, you can also specify which VM you want to be as default. CS. So, the next thing I'm gonna hand over to Niles who's written the support for Eclipse and Eclipse and features in Java Helper. Yeah. As said, I have built some tools for building Eclipse packages, not Eclipse itself, but packages like hopefully Eclipse CDT and EMF and RSS, which are already in the archive. They use these helpers. They're based on the PDE-based build from upstream which basically looks at the manifest files of the plugin it's building and a number of other files and auto-generates an end-base build on that. Which is also very great, but the problem is that the clean target doesn't clean up these auto-generated build files which causes a problem in terms of our requirements for cleaning. The solution to that is today is to simply copy the whole thing into a new year, build it there and then remove that directory when we're done building. This is also the thing we do for Eclipse itself when we build. And for that, we have JHA setup environment where you simply tell it which files it should copy into a build there, which it will create for you. There is a deep helpful file for that, which is not listed here, but you can make it help a like file for that. Secondly, Eclipse is based on a lot of non-Eclipse plugins and they have a special syntax for being a second special requirement to be included in the build. Namely, Eclipse plugins are automatically figured out from their features that are included. But when you need to build a link against something like some of the Apache graph files which are actually pretty common in Eclipse, they have to be specified as orbit dependencies, which is actually a very fancy name for just external dependencies. That being said, there are some special requirements for that. First of all, it needs to have the USGEI metadata, which most of them most often don't share with by default. We are in the luck that most Eclipse core plugins, they take the upstream plugins themselves and compile them with this so we can just download the official OSG bundles, OSGI bundles and read the manifest from there and just copy it to our build and include it. In some cases, however, they embed the actual orbit dependency directly into the Eclipse bundle and just had this require bundle class path that tells Eclipse to load the jar file from within another jar file. And that obviously doesn't work too well. Again, we have to manually solve this by adding the USGEI metadata to the external dependency and then replace the bundle class path with a require bundle instead. But other than that, it's mostly done. So, other than that, by default, if it just from orbit, what we do instead here in Daven is we set up a orbit directory during build time where we just put symlinks to their actual dependencies. And we have a help or two to that because it's relativious and the names doesn't fit. So, JH to generate orbit deal will be given a list of jar files. It will read all it needs to know from these jar files if they have the OSGI metadata and we'll just create the actual symlinks and names. It will even go further and generate a part of a dependency list for in a substitution mark called the orbit call and dependence you can use for your package. It will always be set if you use this tool, even if there are no orbit dependencies. So you can always just add it. Next thing is actually building these plugins and currently we only have support for building a feature which is basically a set of plugins or sometimes a set of features. In this case, upstreams are usually very good at separating each plugin to a folder with manifest, its source files and also it's very neatly separated. And the manifests have all the information we need thanks to the OSGI metadata. And in fact, just starting the build, the upstream build system which is a modified version of and on top of that will actually figure everything out itself. So the problem is as listed, it takes our ten arguments to invoke this to avoid it writing to your home directory which we can't do during a build deed even sort of other things. We have been working with Linux tools, specifically the Eclipse build start project which has provided us two wrapper scripts for this, one to be used directly and one that's used behind the scenes and they do a very good job of starting handling all of this but still it does not work very much like a Debian build, a Debian help out tool. And for that, I have created DH compile features which is a wrapper on that. Simply given a list of IDs to build and this can also be specified in a Debian help out file which is unfortunately not mentioned. It will generate all the features for you and just leave them for Eclipse install. Now DH install Eclipse which will then pick them up and install them. The only thing you need to do is list which packages each features goes into. We also have another problem with the build features is that they contain the open dependencies, a full copy of it. So when you use this, it will un-pack time when it installs into the package find these open dependencies and we replace them with a sim link. But other than that, it's just a fancy wrapper find zip that we had nowhere to find the given zip files and that was basically it on Eclipse over. So having gone over all of the individual programs, fortunately we're now in an enlightened age where you don't have to list all of these manually in your Debian rules file. We have instead a number of frameworks for doing packaging. So let's have a look first, DH7. That's a DH7 file, which Debian rules file which will do everything you need to do for building Java packages, hopefully. So the old dash with Java helper, obviously you depend on Java helper and you need to give it perhaps a class path and the path to a Java home. Now if you're using Ant, then Depp Helpers auto build support should hopefully pick up your Ant build scripts and invoke that correctly and if you're using JH build then that will all be picked up just by doing this. CDBS is reasonably similar. Instead we have environment variables to do the JH build invocation and then you have to include Java help.mk as well as the CDBS stuff. But otherwise it's the same. Obviously this is only the ideal world so you may end up having to override some things but that will just work exactly like you'd expect with Depp Helper you can do override some JH tool and that'll work perfectly normally. So in order to make this even easier for people I wrote a tool called JH make package and you can rock up in some build tree from upstream call that and it will try and build for you a Debian directory which will need a little bit of modification with copyright files and so on but we'll basically hopefully get a lot of this stuff right for you. One that doesn't need a lot of extra talk tweaking if you don't have a working build system and you need to patch it or give it unusual arguments or whatever then you're gonna need to tweak this for a bit. So Java helper contains documentation in the package which is installed in the obvious place. We have a website and a mailing list if you want to ask any questions about this and this will shortly be linked from the website as I mentioned previously, a series of examples which I'm writing which gives you all of the packaging you need to build and install a simple package of various types of various different build systems. So one of the things I haven't mentioned in this talk is Maven. Maven is a build system which is gaining a lot of popularity with upstreams. It works for the model that the Maven repositories will contain all the sources in the world ever and when you run your Maven script it will go and download them all for you and attempt to build everything which obviously is not great when you're trying to build on a buildy without network access. As such we have solutions for this which are separate to Java helper. There's a number of Maven packaging tools which is why I haven't mentioned them here but you'll have to go and look this up separately but if you do have a package that uses Maven we do have some support for that. So does anybody have any questions about this or anything else related to packaging Java? Do you know how many packages are using Java helper for now? I think at least 70 maybe 80 as I recall. What's the question in the back? So having not seen your examples yet can you point me to some packages which you think have good practices? Well you should have a look at the examples I've written if I'll go to, not there apparently. I will have a sort that out in a minute after some of the questions. All of my packages are done using Java helper and with that help of seven. A lot of the ones where I'm also upstream use make files as the build system which is atypical for Java but I like make. So there's also a free market Java for an example of how to link Java docs. It's one of the first ones that were converted to that part. Yes, one of the things I forgot to mention because it was written after we wrote the slides is that when you're generating Java doc if JH build will automatically work out if you have a Java doc package in your dependencies it will use links so that the generated Java doc links to the new one and if you do have Java doc which has links like that then we will automatically give you a dependency list or a recommend list containing those other Java doc packages. So again, this is all done for you. Yeah, that will first be in 232. Yeah, which hopefully we'll upload this week. Yeah, I just wanted to talk a little bit about man pages. So as with any program that's in Devian if you ship a program you ship man page for it. There's no specific support in here for generating man pages. There are other things which will do that if your upstream doesn't contain a man page which with Java software is almost certain. But yes, you should provide a man page for it. No, we don't have anything there but there are tools like Dotbook to man and the various other things where or help to man if you're in the lucky position that it has a usable help output which is also unlikely. But yeah, that's documented elsewhere. Dotbook to man, help to man. You can see the pattern. Yes, I'm looking at packaging Etherpad which is quite a big mess, I suppose. It's very beautiful. That's common. Yeah, so how much can Java helper help out with? And also a lot of the jar files, they bundle jar files and build with some shell script. And the jar files, some of them are in already existing Debian packages like openoffice.org-common. How? Ideally you're gonna have to package all of the dependencies independently and it may be the case that to package your program you need to then first of all package five or so libraries separately. Which I have started doing. Yeah, if your libraries are of the form here are some Java files, compile them and stick them in a jar. Java helper makes that almost entirely trivial. Everything gets done for you. If you're in a situation where actually you do have quite a complicated piece of software it needs to do various things at runtime which mean that you can't just run a jar file and so on then that becomes a lot more difficult. And there's not a lot we can do to try and help. However, where we can, if you have any suggestions for extra helper tools I'm more than happy to include them. But a lot of the things there are still gonna be useful and still gonna be able to help you package them more quickly. So I mean it depends precisely what is going on with your piece of software. It may be the case that you can have a rules file that looks like that and a few files in Debian and if so that's great. If it's reasonably large and complicated then you'll probably have to do at least some stuff by hand but hopefully we can get rid of some of the problems for you. Yeah, the Debian Java team is very helpful so far I've seen so. Thanks. So on IRC you had mentioned a preference for DH7 over CDBS. Any, can you elaborate on that a little bit? What Beedow said. I don't really understand CDBS very much. I wrote this by cribbing from some other things. DH7 just seems a lot more obvious as to how it works to me. All of the override stuff is very nice and you read it and it's pretty obvious what's going on if you understand DH7. All DH7 does is just run all the DH commands in order. So yeah, it's basically it's a long form old Dev helper but without having all of the tedious typing. Plus I suppose Java helper might do great slightly better because I understand what's going on. I was just gonna say, in addition to a nice simple DH7 file, I mean the other thing that seems to have to happen packaging up Java stuff is you need to patch up an upstream ant file if it's done with ant because they'll hard code jar versions and things like that. And I'm just wondering, this is just an idea I'm throwing out there. Maybe that could be automated where we actually, we look at the name of the jar maybe we sure as typically figure out well. We have this jar in Debian or we don't. I mean sometimes it's even knowing where to start. Yeah, because you gotta find all those jars to figure out whether or not they're actually in there. It's a version you need. So I'm actually working on free TTS at the moment which is the flight re-implementation in Java and contrary to popular belief if you leave the Java sound API stuff off the rest of it's completely free software and works just great. But I spent a lot of time angsting over whether to care about the existing ant build.xml or not because in fact about 90% of its content is a poorly written attempt to do all the things that you don't actually wanna do for a Debian package. But by which I mean it emits an incredibly heavy jar file that has all the docs and lots of demo executables and other junk in it. And a lot of the logic that's in there is all about building all these subservient pieces and making them part of the distributable thing. And after spending quite a bit of time trying to unwrap that and understand, get over my fierce aversion to XML for build logic and dig through it, what I realized is oh, what I actually want to have happen here is really, really simple and this is just the wrong approach. So I'm actually right now sort of sitting here doing it the other way which is forget the existing build file, treat it as if it just didn't come with one and craft from scratch and I think I'm gonna end up with something I'm much, much happier with. This course is obviously very dependent on the individual case of what the upstream gave you. So the URL is actually packagejava.aleas.org slash examples and this is what we have at the moment. So if you have a look at the simplest case here, then it'll give you everything you need in your debinder, which is that much. And the trivial debind rules file. So that's the sort of thing I've got for the examples and it does at the moment only cover simple cases with Ant and with using JH Build as the build system but that should give you enough of a clue to work from there and I'm hoping to add more examples as I think them up. And any other questions? Cool, well I'm happy for anybody to column me afterwards and ask me questions and or to come and talk on IOC or the mailing list. Sorry, I had a last question. So all these tools, you feel comfortable maintaining them yourself. You want more people to help you and isn't there written in pro? I'm always happy to have help. Niles is now helping working on them. The majority of, now about half and half, I originally started writing things as shell scripts. We're slowly moving stuff over to Pearlware where it's too complicated for that or where Niles has written them because he speaks Pearl and I don't. I'm perfectly happy for people to submit patches or entirely new tools or rewrite things in Pearl for me, that's totally fine. Yeah, one of the tricks of rewriting the things in Pearl is we can use the official debhelber backend which deals with a lot of problems for us like argument parsing, a lot of them just gets fixed by debhelber backend so that's one of the reasons. I'm happy for people to rewrite stuff in Pearl for me but I don't know enough Pearl to do it myself yet. Cool.