 Next talk is about Haskell and Debian. Going to be done by Joachim Breitner. He's been doing loads of stuff with Haskell, I guess, since quite a while. Since 2007. 2007. So that's quite interesting, I suppose. So please welcome Joachim Breitner. Thank you. First of all, this is not going to be a Haskell tutorial. I know a few people want to have that. So what I'm going to do is I'm going to do my talk. And if there's time at the end, which I have no idea about whether there will be time at the end, because I haven't tested it yet, I'm happy to answer any questions about Haskell itself. But still, it's an introduction. What is Haskell? Haskell is a programming language. It has certain features not found in other languages. And if you list all the buzzwords of language development, they are there. It's a functional language. It's pure. It means it has no side effects, which makes it easier to write bug-free code. It has lazy evaluation, which means that data structures are only evaluated when they are needed, and else not. It is strongly typed, but with type inference, so you don't have to type the types, which makes the name probably a bit wrong. It can be interpreted or compiled, so you can get properly compiled binaries that you don't actually see that they are Haskell programs anymore, unless you know how to look. And it has a large number of libraries available. And this is basically the topic of the talk. We want to have all the good libraries of that in Debian available to our users. And we have been doing it for a while, and I want to share how to do it. And last of all, Haskell is fun to program in. You don't have a lot of things you don't have to worry about that you have in other languages. And those things that you have to worry about in Haskell that you don't have to worry about other languages are more fun to resolve. This talk addresses you, especially if you either want to use Haskell on Debian so that you know how to use the Haskell packages for Debian and where to look for them and how to use them. That's, of course, also useful if you maintain a package that uses Haskell, some program. Then if you want to maintain libraries of Haskell and Debian, then this is the most important thing I'm going to talk about. If you want to do that, you are very much encouraged to join the Debian Haskell group, which is a packaging team, as many other packaging teams before. And I really discourage people to not join the group when maintaining Haskell libraries because we tend to do changes that need to affect every library in Haskell, every Haskell library in Debian. And we are happy to do this across our repositories. If you don't have your package in our repository, it's up to you to do the changes yourself and that only causes additional work and friction. So feel free to join the group, especially if you are a DD because you always need people that are uploading the stuff, people are building. Yeah, then the last audience, I don't know if anybody is here from that group, are those that don't actually want to know about Haskell but have to because they want to release a Debian release or want to get testing migrated or do some QA. And I guess it helps for them, too, to understand how Debian Haskell packages work. All right. Now Haskell code is organized in some certain way and the smallest unit of organization is Haskell modules. They have names, the names are hierarchically organized and can be either short or rather long. Then a bunch of those such modules that belong together are distributed as a package. And we have lots of packages here, so a bit of confusion. There are Haskell packages as in the package the upstream provides and then there's a Debian package that we built out of it. I hope you won't get too confused. If you are, just interrupt me. These packages have names like this. There's even a Debian package in the Haskell repository that we have packaged for Debian. So there's a Haskell Debian package in Debian or whatever. OK, a quick look how this might look. This useful page, which is generated as soon as you install the GHC Dock package, which lists all the modules that you have installed using Debian packages of Haskell libraries on one nice page. And you can see that there are lots of modules on the left and you can see what package they come from as in Haskell package. And you even have a index that ranges over all the Haskell libraries installed on your system, which is a very great feature if you want to develop stuff. You have only one page to look at to find the documentation. I don't know how many other languages provide that. Anybody standing up for his favorite language? No, good. Then there's the ghc-package, pkg, binary, which maintains a local registry of installed packages. In that sense, it's kind of similar to d-package, not exactly, but similar. And we can have a look at this. For example, we can list what packages we have installed. And we find that in the system-wide location, we have these packages installed. It's quite a few. And then locally also installed as a user, a few packages they are separately from the system. And they are there. So this is the ghc view of what packages are installed. And the Debian packages, of course, make sure that when you install the Debian package of Haskell library, the library gets registered within the system and unregistered when you remove the package again. Then cabal specifies the source package format of upstream Haskell packages. And we will later see how that looks like. It's declarative. It has one file, a dot cabal file, which describes what dependencies this package has, what seed libraries it needs, a description, a license file, the list of modules in the package, and then it builds the Haskell libraries. And this is great for us, because all the 200-something Haskell packages we have in Debian and all the thousands of Haskell packages we have not in Debian, they all build more or less the same way, which makes maintenance of hundreds of packages doable. That's a lesson we have learned since the days of the Perl team, and it's good to have that. Now, for additional confusion, there's cabal install, which is a binary. The binary is called cabal. So cabal can either be the source-formage package and library associated with that, or the binary which comes from the cabal install package. Great, but that wasn't my fault. Cabal install is a bit like CPAN for Perl. It can download your packages, build and install them, and it can resolve dependencies. So the locally installed packages you've seen on the shell before, I've installed them with cabal install. And then the last important bit for the ecosystem here is package, which is the Haskell package, hence the name, repository. So in a way, it corresponds to our FTP master, or also to CPAN as in the web page. OK, at this point, I have planned a demonstration. So let's assume we want to program something involving a very efficient representation of bitmaps as in mapping numbers to bits. So in that case, I would probably maybe first go to the package web page, which is this. And usually I don't look at the first page, but go directly to packages. And you see there's a lot of categories, and then there's even more packages below. And I want something that has somewhere related to bit arrays. Yes, is that better? Even more? OK. So I'm looking for bit arrays. And uh-huh, there's a library called bitarray. So I can look at the description. It's experimental, but people tend to name stuff when they release it first. And if it works fine, they just leave it like this. It's building on the current GHC that we also have in Debian. It's a good sign that the package is in a good shape. And it has three modules. I can also look at the API. And I see, OK, there's a data type bit array, a packed bit array. Internally, it is represented as an unboxed area of Word 64. Sounds like what I need. So let's try to get it on my machine. Now, leaving Debian out of the picture for now, I would go to the shell. I would use cabal install bitarray. And it will download the package. In this case, it doesn't download it because it has been downloaded before. It is doing something about its building it. It's doing something about documentation. It's updating documentation index. So there's basically a copy of the page I just showed you. It not only includes the system-wide packages, but also the ones you've locally installed. So even those are in the index. And because I have many packages installed, it takes a while. And it's finished. And we can run ghc package list. And I will find that at the very end, there's a bit array package locally installed. And now I can see if I can use it from which function would I want to test. Looks like I need documentation. List bitarray. Let's see if this works. 1 to 10, true, true, false. True. OK. Looks like it has created a data structure. It's serializing it as text in this way. And I can also do lookups. And I forgot how to do them. Lookup it. That's fairly easy. OK. Looks like it's working. So this is how I get new code on my machine to develop it. Now, from a cabal package to devian package, we first have to find consistent naming. So the general idea is that if you have a cabal package that's called bitarray, the source package name is associated with this. It's called Haskell bitarray, which is not completely enforced. You have a few package that derive from this, but the source package is not very important in that respect. And then we have three packages, three binary packages that are all named after the scheme libghc dash cabal name dash type of the package. And we have three kinds of packages. The first one is the dev package, which is the one you need to program with. Then the documentation is separate in case you're low in disk space or for the built-in teams. And then the profiling package, which you need if you want to build a binary that has profiling enabled. Then you need the profiling packages of all dependencies. And because these also tend to be not very small, we've separated them out because not everybody needs them. Cabal install, as I said, can be used on devian. One disadvantage of using it over the devian packages is if you, if something changes in the dependencies, you manually have to recompile your locally installed packages. Whereas if you use the devian packages, this is being taken care of for you. And we will see more about that later. Again, demonstration. Oh, yeah. Now we want to have this bit array package in devian. Because it's useful, I've used it last week to implement the setBritney tool. So I want to have it in devian. I can use cabal to get the source. No. I think I'm skipping the demonstration. Yeah, packaging later. I just wanted to show the, I'm confused. Sorry. I wanted to show the devian packaging stuff. So a package that's already in devian, for example, would be Xmonad. So we can search for Xmonad. And we see that we have these packages. We always, they always come in three of a kind. So here they are. And there's doc packages up there. The GHC6 are transitional packages from earlier times. And we had the version number of the compiler encoded in it. So if I want to install a package, and I think it might be available in devian, I can just do, for example, XHTML, it's a cabal package. And then it sounds very similar. I can just install this. I think I should have removed some package before demonstrating this. Well, it's obviously already installed. And afterwards, I can see it in the global list. So there's XHTML available for Haskell to use. All right, back to the slides. Haskell package dependencies. Now, the problem with Haskell from a distribution point of view is it provides zero ABI stability. So a minor change to a package, even a change that doesn't obviously change the interface, might change the ABI. So for those who maintain C libraries, it would be, as if you had a SO name bump with every upstream release and every change you might make. And obviously, treating this the same way as we do in C libraries, as in package renames and transitions and stuff, would not scale. So what we use is that GHT package already hashes the ABI. So it looks at all the things that are part of the ABI, creates an MD5 sum of that, and registers this together with the package in the database. And also, it marks for each dependency against which particular version of the ABI, which hash this has been built against. So for example, the bit array package we just installed has an ID. And it consists of the cabal name, the version, and then this hash. And if I would build another package based on bit array, this hash would be stored in the database. If I then would change bit array in an incompatible way, which could include bumping the version under, but doesn't have to be, then the other package would be marked as broken by GHT package, could not be used, and would have to be recompiled. Yeah, I think there's an example. This depends on array and base. And it records the hashes there. So GHT package has a, since last GHT release, a very tight grip on these dependencies. It doesn't break easily, as in binaries not running. But of course, we want our users to be able to be, what are users not to be able to have inconsistent packages in the first place. So we need to take this logic that GHT package provides and lift it on the Debian package level. And we do this using virtual packages. So if you would package bit array, then the tools that we use for building these packages takes this string and generates a virtual package that consists of libghc6, the cabal name, dash the type of package. We do this both for the dev and the prof package, dash version number, and then dash the first five digits of the hash. And then all the pending packages get this and that depends, and not the package name. Therefore, if I rebuild and package high up in the hierarchy and it changes the hash, people who have depending packages installed can't upgrade this package until the depending package has been rebuilt, which is good. Because Haskell packages installed in Debian are therefore always consistent. No, it's not. Yeah, let's fix it right away, because people might have heard of microsecond sleep. Therefore, we get the right thing here. Yeah, that looks better. Nothing we can fix in Debian. OK, the good thing is Haskell packages are always consistent. For users, that's good. For us, it's bad because it reads a lot of rebuilds in BNM use. I just uploaded a new GHG version two weeks ago, and we had to build 300 something source packages on all architectures. Luckily, since two years, we have support for BD uninstallable in WannaBuild. That means that our build demons won't try to install things that can't want try to build things whose build dependencies can't be installed, which kind of fits very well with our scheme of these hash dependencies, because in the WannaBuild and the build demons, they'll automatically get the build order right. This would have been horrifying before that, and was basically Haskell was my motivation back then to implement that. So in a way, Haskell is a stress test for Debian infrastructure. Then for our users, it's a bit annoying because libraries are temporarily uninstallable. I built a new GHG package, and until all the libraries have been rebuilt that the user has installed, it cannot upgrade. But yeah, that could be a minor issue. Testing migration is hard, because I have to get all the changed Haskell packages in the testing at the same time. So I need to manually create a large hint file for those who are aware of how Brittany works. Plus, if the Haskell package is also part of another transition, things get complicated. So if you want to see why packages are on testing and some Haskell packages are involved, this is the reason. The good thing, at least, is that binaries are not affected. Haskell links statically, so that the binary that's being returned at the end has no dependency on anything Haskell related. It looks like a regular C build library, a C build binary. So the binaries like Git-annex can migrate to testing and can be installed even if Haskell packaging is currently in a bad shape. As long as we use static linking, since the last version GHG does provide dynamic linking, we're not using it yet. It's new, and better be careful about that. But it has been discussed. OK, let's package a Haskell library. Now I'm at this point. These are the resources, in case you want to look at it later again. And I'm going to do a live demonstration, as I have hinted before. Yes? So you're saying the users of a Haskell program don't have these difficulties. So what you're doing is basically for Haskell developers who need all the libraries. Is that right? If everything's static linked, statically linked? The users of, for example, Git-annex don't need to worry about Haskell at all. They don't see a single Haskell package on the machine. It's simply binaries to them. It's simply a binary to them. Yeah, thank you. Of course, Joey, as the developer of Git-annex, has problems if I break Haskell, because his package won't build, and he can continue working with it. But yeah, that happens. OK, packaging has library. We have a wiki page of the group that lists a few things. We have a policy warning it's outdated. I think I've removed most of the audio stuff without replacing it. If somebody likes putting common practice into wording and likes to fill up the policy, that would be very welcome. We have a tools repository that we'll use in a moment, which has a template Debian directory, which we use as a starting point for packaging Haskell libraries. And there's a mailing list in our IC tunnel. And yeah, the rest will be demonstrated. So as I said, we want to package bit array. So we unpack it. This will download it and unpack it. Now we have the bit array source file, the cabal file. You see here's the cabal file. With the descriptions we've seen on the web page, including the dependencies. It depends on base and array, which is rather little. Other packages have much more dependencies. OK, so we need a Debian directory. So we just copy the template directory, which I have conveniently lying around somewhere on my machine. And then I just go through all the files and fix what's left to fix. Of course, I always have to exchange foo by bit array. Then I set the version number. I don't have an ITP for this package, so I don't have anything close. I know it's bad practice, but sometimes I'm lazy. And that's me. Then I go to the next file, which is combat file. Nothing to change here. The most of the work goes into the Debian rules control file. Who of you have seen the packaging for beginners talk in the morning? OK, at least somebody knows what I'm doing. Again, I replace foo by bit array everywhere. I have to change the uploaders because we need to have real uploaders for some reason. And if you know that I'm doing all this in great detail, just because I want to do upload right away, to make sure that you see some real work done. At this point, I would add all the dependencies on other Haskell packages. I would also look in the cabal file about the version range to encode this in the build dependencies. Now, this particular package doesn't have any dependencies that are not provided by the compiler, base and array arm. So we can remove them. Some packages are bindings to C libraries. In that case, I have to have the corresponding developer package of the C library here. Not in this case, so I can remove this as well. Then home page is usually just the page of the package file unless something better comes up. And we have the VCS fields for the team. Now, the binary package. The package name is, as I said before, libchip.it-ary-dev. C libraries bindings need to have the dev package here again as a dependency, but not now. Then all the other fields are filled by our packaging tools. We've put as little as possible logic in the individual package and as much possible logic outside so that it's less work. Description, I can usually just get from hackage, mutable and immutable bit arrays. And the long description would be this package provides mutable and immutable bit arrays implemented as packed arrays of 64-bit integers, or words better. It provides a pure interface as well as monadic interfaces for the IONST monad. I think that's fair enough for this package. I have to copy this to all three files. I have all three packages that are provided by the source. And then there's a common blob that's below that explains the type of the package. And I also have to add the subscription profiling data and here documentation. OK, am I done with the rules files? The prof package has nothing to change. The documentation package also nothing to change. So that was fairly easy. Then the copyright file. Somebody suggested we use the DEP5 format here. Not my choice, but I'm fine with it. Wait, upstream level. So let's copy that from here. And there's a license file in the code that I can copy here. OK, I think this should be sufficient for the copyright file. Do I need to have the space dot here? OK, let's add that as well. This is the rules file. And we're a bit proud of that it can be this for almost all packages. And sorry, Joey, it's CDBS, historical reasons. This is the sources format directory. We use quill 3.0 source packages. And then this is also important. It's a watch file. We rely on it with our tools. But they are also always the same for all Haskell packages because they come from the same source. Did I forget anything to you? Did I forget anything about packaging? OK, so this is my Debian file, Debian directory. So let's try to build it. Oh, I don't have an upstream file. But thanks to the watch file, I can just run use again for stone load. And it'll put the file with the right name in the right directory. And they're good to go. Yeah, now we see some stuff running by. It's compiling. It's generating documentation. It's generating the packages. And CDBS thinks it has to run a lot of DH commands. And at the end, we have the three binaries available to use. So let's test them. I install it. It says trigger for GHC doc, which is the global index. It's being recalculated. And this warning will be gone away with the next upload of GHC. And once this finished, we can go to the system wide index. And voila. C, C, C, D, data array, bit array. There it is. Locally installed with highlighted source available because sometimes need to read the sources of libraries to find out what they are doing, or very convenient, or very nice. And because it is such a fine package, I'm going to upload it right away. We use DAX as a version control system for our packages. I think I have to use VASCs by now. It works. I'm still confused about this migration. So I push the repository on DAX devian.org. At least I think so. It has connection because it's showing the fingerprints. And we also have to install a post commit hook that makes sure they get commit messages. This is documented also on the wiki page for those who want to join development. This has been the initial push. We only have to do this once per package. Now I can mark the package for release. And yes, thanks. Build it again with the release tag. Ah, when I was practicing this, I run the clean target, and now I forgot. So DAX, all right. Good. That looks better. Let's continue from here. Maybe I will fix the repository later. I'm building it again. And yes, thanks for my signed key. OK. I have four IDs, two keys. Have you seen eight mails by now? I hope the video team doesn't have an angle on that. And then I always have dev commit yes. I always have this command line to make sure that I don't forget to push my changes to the repository. So it's being uploaded and pushed. And we have package and uploaded a new Haskell library for Debian. And I think it was pretty quick. So it's not a lot of work, but there's a lot of packages. So we always need help. And I can go back to those slides. At this point, I would like to express some wishes that I would like to have in Debian. I know some of them are difficult or maybe even not desired, but from our point of view, it would be great if you could bin NMU arch all packages. Because some changes, yeah, some changes, and we don't have to change the 200 source packages, but you just have to rebuild everything. And doing this only by uploading a new Haskell dev script package and then NMU and everything would be really great. And similarly, create source-only uploads would be great. Sometimes you do changes where we are definitely sure it's not going to change anything problematic, for example, cleaning up the description files and having to rebuild 200 source packages just because of that. When the buildities can find out how to rebuild what and what order would be a great time saver. So that's a wish into the direction of Debian. In case some of you is going to be upstream for Debian for Haskell library, please write change logs. For some reason, it's not custom for upstream to write change logs, which is annoying. What we skipped today was everything about Haskell itself, but if you want to find more about that, there are good tutorials here, and I included them for those who would like to continue there. And you're free to ask me any questions during Debian or within the next eight minutes. Yeah, and the conclusion of this talk is Haskell packaging, Haskell for Debian, is easy, is not difficult, but always needs more manpower. So you're welcome to join the Debian Haskell group. If you want, we have a main list. We also have an ISC tunnel. I guess the main list is more important, more active. And maybe as a last picture, I'm going to show this. That's the Haskell package dependency graph. And you can see why you don't want to build that manually in the right order. OK, any questions at this point for the last five minutes? Well, first of all, when you showed us the locally installed documentation for Haskell libraries, I looked at mine and I didn't see any, because I hadn't installed the doc packages. And I also noticed when you installed your package, you had to install the doc package manually because nothing pulls it in. I'm wondering if you've thought about having a recommends in one direction or the other so we can only install one package and get the documentation too. I think we have recommends, or at least to just. You're checking you don't. You have suggests. OK, so you would like to suggest documentation. So you're suggesting to change the suggest to recommends? Well, in either the library package or in the documentation, I don't know which makes more sense. It might make more sense if you're installing the documentation, you probably also want the library. If you're installing the library, you may well not want the documentation. That's fair enough, I think. Now at this point, if you remember, the Debian control file had a recommends subscript, so we could change it in the central position and do binomous of all arch all packages, which you can't, so we will have to wait for the next time. We actually have to rebuild them anyway to implement this, but I think it's a fair suggestion. If you want to file a bug against Haskell Descripts, we won't forget it. And there's one behind you. Sorry. Hiding. I have a question about bootstrapping for Haskell. New architecture in Debian, I think bootstrapping is very difficult. I think new architecture of Debian cannot success bootstrapping. Do you think so, this problem? Sorry, I had problems understanding your questions. Can you repeat again? Oh, okay. Yeah, I think it's difficult because upstream says they have a certain set of architectures they care very much about and everything else is second class. And of course, it's technically highly challenging because it does strange things, like having a own calling convention and very strange things happening on the heap. And, yeah, but if anybody wants to do it, that would be great. What architecture are you considering? Is there any specific architecture you would like to see it on? So, yeah. Sorry. Sorry, I'm not good at English. Please. Yes, I'm currently porting a Super-H, Super-H is a new architecture and own and Debian ports.org, PowerPC, ESP and hardware porting now. Okay. Well, if somebody wants to do it, that's great. I don't know if we as the Haskell group which maintains that she's going to have the manpower and the skill set and the time to port to new architectures. But in a way, it's also upstream work, so you're very welcome to talk to upstream about that. And they take patches. It's just that they can't test it before they release, so things might break when they do a new version. So, I have a question. I apologize, I'm not a Haskell developer, but I'm curious, is can we draw an analogy between Cabal and Maven? If I would know Maven better, maybe. So, if you're not in Debian, if you're a typical non-Debian Haskell developer, do you use Cabal to just install all of your build-upends? Is that how it works? Yeah. Okay, so it's similar to Maven. In the Java world, we struggle with this because a lot of Java developers just use Maven and they expect they'll pull down all their build dependencies, and it makes it tricky for us in Debian to use that apart and package them appropriately. And so it's always interesting to me to see how other language libraries are handling that. No, FOS. FOS is pretty easy because the build-upends are pulled in declarative. So, there's this Cabal file. And maybe I would show some more complex Cabal file. Okay, maybe not because it's running out of time. And it's just listed in here. So, at one time, we go through a list, we look at the corresponding Debian package name, put it in the controls file. At our grade, we check if the list has changed. We put them in and everything works fine. So, we have every reason to be happy about this particular part of the Haskell ecosystem. Okay, one more question, if I may. Do you talk to Haskell packaging teams in other distributions and the problems they are facing? The Ubuntu team or Gaia more or less is part of our team. So, basically, this is the work in Debian and syncs it to Ubuntu. Very great, people should do them often. With other teams, no. There hasn't been any considerable collaboration. I think you can pass it back. If you want to stretch the time a few seconds. Yeah, your example of packaging a library, it looks fairly simple, but there is still a lot of boilerplate involved. I'm wondering if you have anything to automate generating the boilerplate or maintaining the boilerplate. For maintaining it, we have the template directory. For generating, I think we stretch the generatability as far as we could by using substrubbers, everything else. Well, I think that the Debian parole team, for example, I mean, well, first of all, they can build a package out of cPan directly, just curiously. I wonder if you've thought about doing that with package and also, I think they have something that can then maintain a package by updating the files as long as it's their packaging tool knows about, say, the format of the control file or the rules file or something like that. For the first question, somebody is working on a tool to generate Debian repository automatically from Hackage. Marco is doing that. But of course, we don't want these packages in Debian directly. And for maintaining the files, so far, we've been running fine. If you want to make a change, keep it low-tech, keep it simple, and then people understand it. I'm afraid we've run out of time. So, please, again, thank Joachim Breitner for this interesting talk. And then we'll move on. APPLAUSE