 Okay, it's three o'clock. I think we can start our session. So first of all, welcome to my talk about Diba and Java packaging. My name is Markus Krushani, and I'm a member of the Java team in Debian. Today I want to talk about our three build systems, about our three build systems, which we often use in our packaging work, about and, Maven and Gradle, and I want to introduce you to the most important Java terms and concepts. And I want to talk about Java Helper, which is very helpful if you want to package something which has no build system at all, or if you want to modify a package which uses a different build system. And in the end, I want to show you some common error messages and how you can avoid making mistakes. So if we talk about Java, then all you need to know is that source files have a suffix called Java. So our typical Hello World program would look like this, and if you wanted to compile it, you just have to type Java C Hello World. Now you can see that another file has popped up, a class file, and this file is just bytecode. And one aspect of Java is that everything is compiled to bytecode, which is interpreted by the open JDK. So if you compile it here in my system, which is AMD64, it is possible that you can transfer the file to any computer where an open JDK is installed, no matter the operating system, no matter the architecture, as long as it is supported. And you could run this program just like this. Hello World. So the next step would be something more complicated. Let's imagine you have a program which consists of three or four of these Java files. And in Java, it is convention to organize source code in directories called packages. So we have two packages here, compute and engine. And you would compile multiple files like this. And now you have two class files within the compute directory. Compiling class files is one thing. If you want to transfer those files, you have to make it more, well, some kind of, you want to transport it to another computer and you want to combine it into one single file. All you have to do is creating a so-called JAR file. A JAR file is merely a zip archive and all those class files are stored within this single archive. Plus one file, which is very special to Java, it is called a manifest file. And this is very important later on. So let's see how we can create a JAR file. The command is JAR. And something like C, V, F. The name of your JAR. And then the class files you have just compiled. Now you can see that OpenJDK2, it has added a manifest file. It's a so-called default manifest file. And you can take a look inside the JAR, like this, and see that the manifest file consists of just two lines, a version and a created byline. Manifest files can be much more complex and you will later see that it's very useful to modify this file because there are also information stored like the class path, another very specific Java term, which basically means where can my program find my libraries on the system? At the moment, it does not know. So you can change the class path within the manifest file, but you can also say, you can add it to the command line and tell Java to execute a program like this one. So you would call java-cp, stands for class path. And wait a second. Ah, first of all, let's move our computer to another directory. And then, no, that's not it. So you would call, of course, the java-c command, which is to compile. And then you put the computer on the class path and target the compute engine java file in engine. And now you can compile the engine file. And that's basically how all these java compilation things happen, how builds happen. This is the most simple case. So if you take a look at your compute engine java file, you can see statements like import compute dot compute, which just means the package compute and the class compute. The package compute and the class task should be imported. And you have just told your compiler that you should look for a compute jar in your test directory. And then he knows, okay, there I can find my classes and then compile my compute engine java class. This is what happens hundreds of times in Debian Java, only on a much greater scale. I just wanted to tell you the most basic aspects, but this is, in principle, what happens at the core all the time. Now, let's imagine you want to package an application which consists of 10 to 100 files, but you realize it has no build system. So what do you do? For example, there are many packages, API packages in Java that need, well, that we need, and they are available multiple of times on the internet. We just did one. So I have pulled two packages from main central, that's a repository for Java files, Java files, and then I see that there's only one directory. Again, that's our namespace, org, OSGI, and then you see a lot of packages, DTO framework and so on, but you see there seems to be no build system. They're just directories and files, source code. So what can you do? You could, again, use the Java C command and compile your packages, and you could create a simple script, a loop, and then, well, somehow you would compile all those files, but you don't have to reinvent the wheel because we've created a tool called Java Helper, which just does this work. So in OSGI core, there's a file called Java build, and as you can see, all you have to do is, you give your Java file a name, in this case, OSGI.core.jar, and point to the directory which contains all those source files, and that's basically it. Java Helper will compile all those files for you and create a Java file called OSGI.core. So how can you find more information about this Java Helper tool? So you would go to user shared doc, Java Helper, and then we have a nice tutorial in HTML and text, let's prefer text here, and there you can find all those commands. They are prefixed with JH, and it's pretty similar to the Depp Helper commands. At the moment, you only need to know one command, it's called JH underscore build, and that's what you actually do with your Java build file. So like with Depp Helper, you can choose either a file or you can choose to put those commands into your Debian rules file and execute it. So it's up to you, it's a matter of preferences and it doesn't really matter. So here you can see we use a Depp Helper and DH sequencer, and if you want to use Java Helper, you just pass with Java Helper to your DH command. What else is important in this file? You can see two variables. Yes, I forgot. If you have any questions, just ask, it's a workshop, so yeah. I have a question because I learned that frequently necessary that you add this UTF-8 thingy. Isn't it possible to make it the default because I always need to add it, so one. Good question. Well, I've asked the same question multiple times on our mailing list, but there's one team member which could win me, okay, not really, sorry, if you see this video. No, it's quite difficult because there are multiple build systems and they default to like Maven to USS key. And that's, I always think why do they do it and don't change it to UTF. But you are asking not the subset of UTF-8, so nothing should happen. Yeah, well, they have made it the default. So they argue that it is the job of the packageor or the upstream antenna to set the correct value. So they are not responsible for setting a language feature. Yeah, I agree with you, yes, it makes sense somehow, but we don't want to override upstream of the build systems and well, I think it's struggling against upstream isn't the best way to spend our time. So what you can see here is the JH build command is called and we passed two options to it, Java C ops and Java doc ops. And while we say, okay, please use UTF-8 as your preferred language, so otherwise the package wouldn't even compile. Yeah, what else can you see in this file is there are two variables called Java home and class path. I've told you about class path before. So here we tell our Java helper program that he needs to look in user share Java OSGI dot annotation dot jar to find the needed classes to build OSGI core. Our convention is to install system-wide Java libraries into user share Java. We have one unversion Java, a Java file and one version Java file. And that's mostly important if you use build systems like and or if you have no build system at all. Those, for those build systems, it makes sense to store it in such a global directory. You will later learn that Maven has a different concept and we install the same Java multiple times as we just similar to different directories because we have to keep a certain layout for Maven. But at the moment, it's just important to know if you create a package, you have to install at least one Java file and at least it should go into user share Java. A Java home is important. I'm not sure if you need it anymore. For Maven builds, you don't need it, I'm quite sure. But for non-build systems or and I suggest you still use this line export Java home so that the tools can find our default Java development kit which is OpenJDK 10 at the moment and will be OpenJDK 11 in a few weeks. Next question. If it's a default, why not setting the default and just do something else if it's... Yes, as I said, it is default for Maven builds already so you don't need it there anymore. But I put it stupidly in every rules file, this line Java home equals to this one. So it could be included into Java helper and then you have shorter rules files. Yeah, you sometimes want to override it. Yeah, and you override it. If you want to override it, do it. But if it's default, I don't know. Okay, let's think about it, it makes sense, yes. So sometimes you want to override it and then you have to look, just go to user, lip, jvm. So you can see there are multiple choices at the moment. Only we will keep only one for Buster which is OpenJDK 11. And okay, OpenJDK 8 will be kept just for development purposes and it will receive no security support at all. So, okay. So, what else? Yeah, so there's a second command here. It's called JH Maven repo helper. I'm just telling you about it but it's meant for installing these jar files I mentioned before into a Maven repository. So, and does nothing, does not know about Maven and if you have no build system at all, of course it doesn't know how to install a Maven artifact, a jar file. So we have created tools like JH Maven repo helper or Maven repo helper, just Maven repo helper that install those files into a new directory which is called user share Maven repo and here you can see different packages already installed on my system and the layout is as follows. So you install those artifacts into user share Maven repo and it is the same namespace as you would see in the package itself. So, let's see for Lib 12 monkeys. Oh, no. Where is it? Let's just take another one. Let's take commons codec and then you have commons codec, commons codec. So the first directory is the so-called group and the second directory is the so-called artifact and then you can see the version 1.11 and a Debian directory. So your artifact would be installed into the versioned directory but we also create a Debian directory for one purpose to be able to reuse one package for one package for everything. So in Java it is very common, it is very version centric and we use it to, well, we save us time to package all these different versions which is also not allowed in Debian. So we just package one version of commons codec and use that version for everything. So even if another package depends on another version like 1.12, we just use the Debian version. So this creates problems of course too. So you have to constantly modify or patch different packages if they are not compatible with our Debian package but it cannot be avoided because we want to reduce code duplication and it's also a security concern if we packaged multiple versions of the same library. So in Debian there's always only one library, of course there are exceptions but the general rule is please package only one library and then try to make all your reverse dependencies to work with it. So yeah, so that's basically one way JH, Maven repo helper is in Java helper. I just mentioned it here, you can use it but I recommend to use Maven repo helper instead which is a separate package, let's see. So this package enables Debian packages which are not using Maven in their build process to provide install Maven pumps and libraries in the repository located in user share Maven repo. If you just use Maven or Maven only build system then you have to use Maven Debian helper instead. So always remember, if you are not using Maven, if you are either using and or Gradle or no build system at all, then please use either Java helper or Maven repo helper if you want to provide also those Maven artifacts. It makes it easier for other maintainers who have packages which rely only on Maven to find your jar files which you have just packaged. If you only install them into user share Java they won't find those jar files but it would be much more complicated for them. So if you would like, want to be nice to them please also use Maven repo helper to install those jar files. Okay, so that's it for no build systems. Let's go to end. In 2012 I packaged a media take view which is a German application to view and download public television streams. And that was my gateway or my, yeah, well, my first package which I contributed to Debian. And as you can see, media take view back in 2012, 2013 used end, how can you see that? If it is end, there's always a build.xml file. Well, so xml looks some say quite ugly, some say it is human readable. Well, you decide. And you can also see in this case that it uses references another project. It's called NB project and that means the upstream developer used NetBeans at this time. It's an IDE for Java and it's a bit specific but if you come across such a package always remember this is a NetBeans project. And if you want to modify something you could go to the NB project directory and just modify those xml files. So what options do you have if you want to build an end package? So you can, first of all, you have to modify, you can either modify the build file. And this is necessary because most upstream developers include other libraries in their package. So before you start even packaging just take a look at the sources and remove all JAR files and all class files. They don't do this out of malice but because they want to make it easier for other developers it's more convenient to bundle everything together and yeah. It's easier for Java developers. So if someone uses and usually creates a lib directory and puts all those JAR files into it. So in Debian we can't do that because yeah well kind of violates our principles so just remove it. And then you can either patch the build xml file and change the path from lib to user share Java because that's our generic repository or you can recreate the lib directory which you have previously deleted and zoom link all JAR files which are in user share Java to this lib directory. How do you do that? So you can use Java helper again. Just go through the documentation. Again, our tutorial. And then, let's take this one. So you can either use a file called linkJAS in your directory which looks like this one. And add one line for creating a lib directory. Another line for creating a sub directory, lib build and a third line for creating a sub directory runtime. Usually you only need the first line. And then all your build dependencies will be linked to this directory. So I don't know. It's a matter of taste I think. You can either patch it which might look more clean or you can use this helper tool. The end product is the same. So your build system will definitely find your build dependencies and will run all necessary steps. So Debian rules looks like this. Again, we use Depp helper. And you have only to add with Java helper again. And if you want to create Maven artifacts then Maven repo helper is your tool of choice. And that's it basically. So the other steps are generic Debian overrides which could happen in every other package too. Another example, electric. Electric is a nice application for... Yeah, it's a CAD system that can design circuits and IC layouts. It's also written in Java. And it also uses the end build system. Again, here you see that we don't lose link jars. There's also no sign that we use it in Debian rules. Again, we tell our build system that the class path where it can find all those dependencies is end user to share out Java again. And here we list all our build dependencies with absolute path separated by a column and then we just add again with Java helper. And that's it. So in this case, I have decided to patch build XML and remove unnecessary stuff and change the lib directory to user share Java here. You can see Java C is our compiler and then you can put your build dependencies inside a class path element and point to your build dependencies and user share Java. It's one way to achieve it. You can also use link jars. Again, it's a matter of taste and your choice. What else have you to remember about end? Well, end is a very simple build system. It is meant as a build tool in contrast to Maven which is more like a framework and a project management tool. If you can avoid it, I think end is simpler. But our support for Maven is equally good at the moment so you can choose. But it is simpler for beginners in my opinion so if you can choose, I suggest you use end. But fear not, I explain Maven in a few seconds. It's not complicated at all. Oh, wait. Okay, that's Maven. Maven, yes. So as I said, Maven is more like a project management tool. It is declarative and it uses conventions. So if you look at the Maven project, you can always find a structure that makes it quite easy for Debian packages because you instantly know this is a Maven project. Let's take a look at PDF SAM, which is a PDF application. You instantly know that it is a Maven project if you see a pom.xml file. That's a Maven project. And the first thing you have to do is analyze the dependencies and check whether they are already present in Debian. So Maven dependencies are structured like this. You have a group ID, you have artifact ID, and you have a version. If you want to find out if it already exists in Debian, I suggest you, for example, code search.debian.net. Just search for the name and you could do something like hibernate, path, Debian control. And then code search would list you all possibilities, possible occurrences of hibernate in your Debian control file. And you know, hey, there's a lib hibernate file. So this is obviously packaged for Debian because it's a very important package. A question on versioning. Is the version in Debian always guaranteed to be the same as the version in the Maven repo for a given dependency? I think rather no. So let's say there's a version 1.1. It's exactly the same version as in Maven Central or else. But, well, sometimes we modify the content because of free software reasons for multiple reasons. But we only package one version. So you can be sure if we package version 1.1 that it is exactly the same version as in Maven Central. Maybe we have modified it, we added a patch. Okay, you can easily see this in our Debian directory. But, well, if you try to package something and you know the project requires version 1.2 of this specific library, you have to use 1.1 instead. Or you upgrade the package to 1.2 and then you have the same version. But then you have to make sure that all other reverse dependencies still work, continue to work. So, obviously, that's kind of a bit tricky. Is there a quick way to tell if there's a difference other than, like, diffing it, basically? I know there's a tool like ABI, a compliance checker in Java. So you could use that to find out if you break something, the ABI. Well, that's our daily business, I say. We have to make sure that everything works. So the more important the library is, if you like an Apache Commons library, which is used by dozens of packages, then it can happen that something breaks. Well, and you have to fix it then. There's no other way. So we can only use one version and you have two choices. You either upgrade the library or you fix all reverse dependencies and it works with it. That's it, yeah. That makes it a bit complicated. That's true. Okay, so you can see all those different dependencies and it's a good practice, a common practice, a good practice to check, do they exist in that way? So this is your first step, check if they exist and then decide if you want to package them. For example, you don't need to package everything. Sometimes you have a Maven project which consists of sub-modules like PDFSAM, but you only need one module. So obviously you don't have to package dependencies for PDFSAM merge if you only need PDFSAM FX. What you can do is the following. There's a so-called POMS file in your Debin directory and you can see it lists all POM XML files in your project. So if you want to ignore something, just add the ignore option to it. And you will be done. And so don't worry about too many build dependencies. Keep it simple. If you don't need Java documentation, disable it. If you don't need tests which require often different build dependencies, disable the tests at the beginning and add them later if you really need them. So let's take a look at a tool which is called MHmake which simplifies certain steps. It is a tool to create an initial Debian package. It is very simple and it only works if you have all build dependencies already installed on your system. Always remember that it can't automatically fetch them from the Internet and create you a package, at least not yet. So what can you do? You can run MHmake and then it asks a couple of questions. The name of your source package, the name of your binary package. Do you want to run tests? As I said, no. It's too complicated for the moment. And then it checks the license. It analyzes the POM. And now it asks you, I found something here, a POM file. Do you want to ignore it? If you see it as highlighted, the Y is capitalized, yes. Then enter the version. Then it asks an important question. Change the version to the symbolic Debian version. As I previously said, it is preferred that we change everything to Debian because you can just point all your other libraries and applications to this Debian version and they will still continue to work. They won't work if we would use different versions. So always use the Debian version unless you have a good reason not to. So include all models. Yes. And then it asks again the question with the Debian version and you can press just enter. Okay. Well, it also knows whether a plugin is useful or not. In this case, the Maven source plugin is not useful for us because we always provide a source package in Debian. So we don't need this. We don't need Javadoc. And other plugins. So just press yes. Ignore stuff. Test dependencies, yes. Okay, now this could be useful. No. So here's a problem. Ignore this dependency Java X surflit. If you're a Java developer, you would know the surflit API is packaged in Debian. It's kind of strangely doesn't find it. I have installed it. But in this case, just ignore it again and hope it will work. So we just looks for another version of the surflit API. Cannot find it. We will abort and just continue. If you're unlucky, it can happen that MAH make will just completely abort and you are left only with three or four files and they are not complete. And then you have to figure out how to add the additional information. But usually I recommend just take a look at other packages which are very similar to yours and just copy them. Just rename the package names and they look all very uniform and very similar. No. So we just ignore the test now and ask another question about the description of the package if you want to add a description and it fails. Okay, that is the so-called presentation effect. So what you can see now is there's a Debian directory and it contains several files and this is what I mean. It is incomplete. This worked before but in this case you are left with only the relevant Maven files. Let's see, what is a Maven rules file? In this case you can see that your group ID the file consists of group ID, artifact ID the type and the version. And we always substitute the version with Debian and the syntax is the same like SIT, a stream editor. So if you want to change something you use a Maven rules file. If you want to modify a group ID you would use this syntax just replace the word with another word you know. Yeah. So the another important file is Maven ignore rules. It's the exact opposite of Maven rules. Here you would just ignore artifacts you don't need. Again the structure is the same first column is the group ID, second is artifact and asterisk means it can take any value. So all those artifacts would be ignored and the build system would just think ignore them. Last but not least the next important file is POMS. The POMS file if you create a package there's always some text explaining what the options mean. And here you can see that we ignore the parent POM by default and we add another option and this package version means use the latest or tell your built reverse dependencies that they should add a strict version dependency on your package so they always depend on the latest version. Yeah, decide for yourself if you need it or you can use the option ignore to ignore certain artifacts. We are running out of time so I just mentioned two things take a look at PDFSEM if you want to package a Maven project an application of Maven if you want to package a library of Maven use for example Lib 12 monkeys Java which is a very simple package and model your package after this one ignore sub modules if you don't need them and if you have any questions just talk to us on our mailing list Devin Java or on our ISC channel and we can answer them. So last but not least I want to mention Gradle which is the ultimate build system very complicated and very powerful of course it's the only build system you will ever need they say it's also a declarative and it uses Groovy as a domain specific language so you can create your you can modify every aspect of your build system of Groovy it's very powerful but for us Devin packages it's also very complicated to create a proper package it is much easier with Maven because it uses conventions so you always find the same information in your source directory your source directory as the code your source directory with the application code and the test directory with the test code and so on in Gradle this all can be very different remember for Maven you use Maven Debian Helper and if you package a Gradle application use Gradle Debian Helper Gradle Debian Helper is not as powerful as Maven Debian Helper but it transforms all those dependencies which are in a build Gradle file well in a format which can be recognized by our Maven tools so you usually the Gradle upstream will pull all dependencies from a Maven central and Gradle Debian Helper will help you to transform the very specific Gradle notations into something our build tools can understand remember that and otherwise you have to patch the build Gradle file so you have to remove stuff which doesn't work like methods or plugins which we haven't packaged yet etc it's a bit more complicated so there's no magic tool which will create a package for you we have different Gradle packages in Debian just take a look for example for the recent version of MediaTikView that uses Gradle previously used and Mochito is a Java library that also uses Gradle check that out or apk tool okay last words some errors common errors if you see failed to execute goal one of its dependencies could not be resolved and the artifact XYZ has not been downloaded from it that usually means you haven't added the build dependencies to Debian control or you've got the Maven rules file wrong so the artifact name or the group ID is XY but you wanted to call it banana so it is named banana in Debian so you have to use Maven rules to transform XY into banana and always remember most likely a problem with your Maven rules file or you haven't added the package to build dependencies secondly the package XYZ does not exist but I cannot find symbols that also means either you haven't added the build dependencies to Debian control or the library is too new or too old and they have just removed the class in a newer version or remove the method and last but not least previously mentioned unmappable character for encoding USSK well then you have to pass one of the encoding options of and Maven or Gradle to Debian to Dev Helper to change that well that's all the last thing I want to mention is or documentation is not very good at the moment I want to change I want to change it and I will within the next days update our wiki at wiki.debian.org Java packaging and add more information to it so you can carefully read it again and you don't have to rewatch the talk and I hope we can extend on it and well create more information to make it easier for other packages to package Java applications I thank you all for coming and see you around