 Okay. Hello everybody. I would like to welcome you to the BSD Dev Room at Fossum 2017. Thank you for being such a large crowd here. I see a lot of faces I know, which is good. I'm going to tell you as our first talk today in this Dev Room about how we package source code of programs written in Go in package source. First of all, I would like a quick show of hands. Who here knows what package source is? Okay. So I don't need to explain it for a long time. Who here knows about Go? Most people. That's great. It's awesome. So package source, obviously, is the NetBSD packages collection, which doesn't mean it runs only on NetBSD. It means it's maintained by people in the NetBSD project. It contains over 17,000 packages. It's been growing. It's still growing after all these years. The really cool thing about it is that it runs on 23 different platforms out of the box. So that includes any sort of BSD you may want to throw at it, like NetBSD, OpenBSD, FreeBSD, Dragonfly, BitRig, whatever. It includes Linux, even Windows. The bash for Windows thing that they're doing is basically Linux, and Cycwin, I believe, is supported as well. Some cool features that package source will give you. If you're on one of the mainstream platforms, you can use pre-compiled binary packages. There's something called package in, which is a lot like apt-get. So really easy installation upgrade and so forth. You can set it up unprivileged. That can be super helpful. That means you don't need root for building packages or installing packages or using them. So you could do a container-like thing if you wanted to, like poor man's docker, so to speak, by having a private package source tree per major app that you just package together and it doesn't need root access to do anything, which is cool. There is a separate site project that's called WIP, Work in Progress. So if you want to get started with package source after all that I've told you today, it's super easy to get an account for that. It uses Git unlike main package source which uses CDS still, and it's very low barrier to entry and it's really, really very easy to create packages. To go a little more into detail, a package in package source minimally needs four files. There are makefile, distinfo, disker, description, and the P list, the packing list. So the makefile is essentially a bunch of variables that are being declared, and then you include one include line that does all the work. The distinfo is automatically generated. It contains checksums for the file you're going to download. Description is something you need to write. Well, it's one or two paragraphs describing the package, and the packing list again is auto-generated by the infrastructure is a list of files in the package basically. And the installation of a package source package or the build goes in these predefined stages. So that's a structure that's basically set in stone, but it's worked really well over the years and it's a good structure. So I probably forgot a bunch of stages here, but first of all, the first stage is fetch where you download the distribution file, or you copy it from somewhere where you have it in your local rep-hole. Then you check the checksums to verify that you've actually got the correct file with the correct size. Then you build tools, you install dependencies that goes recursively, does the same thing for every package you depend on. Then you extract the source code, then you apply any patches on top of it if there are any. Then there's a stage called build where you do the actual, I've got configure obviously. There's configure then build, then stage install which is where you install it into a staging directory, the so-called destir and then the package stage, which takes whatever is in the destir or rather the subset that's in the P list and packs it together in a package. Then package install takes that generated binary package and installs it on your system. Now, we switch gears a little bit and go into the world of Go. So, it's a fairly popular programming language to put it mildly. In fact, it's so popular that tomorrow there's going to be an entire dev room about Go related topics in the DePage room in the H building. In the world of Go, there is a tool that's also called Go. The Go tool handles, among other things, fetching, building, and installing Go programs. Let's look at how it does that. Here's a simple example. So, I chose a random package that was the last package I was working on. Don't read too much into it. So, what's happening here is you tell it to go get something. I'm sorry, it's a little bit small. The three dots at the end is a meta pattern. It's like a star except it matches sub-directory. So, that means I want to install everything that's under this path downloaded and install it. So, Go.get will download that. You notice import paths look like URLs, that's intentional. Then it sees that there is a dependency in there, that we don't have. So, it tries to download that. There's this little bit of a special case because it's sort of a sim link, so to speak, from an import path to an actual Git repo somewhere else. Then it downloads that, and then it builds the things in opposite order. It sounds similar enough to what we did in package stores, right? There's this thing called the GoPath at the top. If we look at this GoPath that we've just created, you'll notice that there are no two directories called source and package. There might be a third directory called bin if there was a binary. So, source contains a source code for those things that we downloaded and it's arranged by import path. That's a standard way that Go arranges its source code. Then the package sub-directory has another sub-directory for your platform. So, from the same source tree, you could build for as many platforms as you want, and it would not conflict. Then it has, you'll notice it has one level of directories less because each sub-directory in source is conceptually one package corresponding to one file in PKG. Now, there is some mismatch between that because it does download, then discover something, then download some more stuff, then discover some more, and then build stuff recursively in the middle. So, that doesn't quite fit with the model we have in package stores. Now, there's two possible approaches to this. One of them is to basically ignore the ecosystem, to say we only package Go itself, and then tell the user, just run Go get something something and you'll get your binary. In fact, that is what the people from the Go team have recommended to me, but I think that's unsatisfactory. That's clearly not what you want because at some point, there is some software that's written in Go that somebody wants to run and you don't want them to be special snowflakes. You want them to be packages such as all the rest of your system. In package stores, for example, they have a linter called package lint that's written in Go. So, it's actually become a quite important topic for a bunch of developers. So, let's deal with it. Some more thoughts on this is most of the packages or the individual, say downloadable units in this Go ecosystem are small and do relatively little. In that, they're similar to Pearl packages or maybe the tech live ones or so. So, there's like hundreds of tiny ones or if you've ever used Node.js and you've used NPM, like you install some random tool, you do NPM install Bauer, for example, and then you look in your NPM modules directory and all of a sudden, you have 350 modules in there and it's a little bit like that. Now, the binaries, once they're built, they don't depend on anything really because they're statically linked, except maybe Lipsy, but okay. So, once you've built your binary package, you can actually install it without having all the intermediate libraries there. So, that's kind of a nice property. So, we have a framework and package source called gopackage.mk that allows you to write these packages relatively easily. We're going to go through this Go OVH that we've seen before, see how that works. So, first step when you want to package this thing, we go to its homepage or to its GitHub in this case, and we download the latest release. Oh, look, there are no releases. Yeah, okay. So, instead, we search for the latest commit and take that date and take the SHA one of the commit, which is an endless hex string, and I'll stick that in our make file. Here's the actual make file for this thing, except there's one bit missing. So, you notice the version number is a little bit awkward, we've chosen a category, we have this weird tag thingy here, and then the more interesting bits for Go specifically is probably this block here. So, the Go source path is basically where in your Go path this thing would be placed. So, it's the import path you want for that package in the end, and the Go disk base is the directory name that your GitHub download will have, because if you just extract the file, it will not be in the correct directory. So, this does a little bit of gymnastics to move it into a newly created Go path, so to speak, to this sub-directory. Then you build and it looks like this. Again, apologies for the font size, and this year happens. Well, of course, there was a dependency, we don't have that package. But wait, you say, I have that package from earlier, right? I downloaded it, but yeah, you don't want that. You want your build to be hermetic in the sense that it uses only dependencies it has actually declared, so that in the end, your dependency tree makes sense. So, it's clear that it's a good thing that it's not finding it. Now, let's add the dependency. We add an include for the build link 3.mk of this package, which conveniently enough already exists. Then we run make and you see kind of unspectacular, not very spectacular it builds. So, what happened now? What did this include build link, something something actually do? So, build link is a framework part of package source. It was written originally for, I suppose, packages written in C or C++. Many of these use autoconf, and if you're trying to package something that uses autoconf, it checks for presence or absence of certain libraries. So, if you happen to have an extra library installed that the package may link against, it can add that as a dependency, and then it's not recorded in your package, but the binary does depend on it. So, that's a big problem. I think OpenBSD had a long history of sort of purging unwanted dependencies. Package source did this building thing instead, and it turns out it works beautifully for Go, because what we do we create a shadow Go path where we link in the entire source code and also the package files, like the entire contents of all the declared dependencies. Then we set the environment variable to tell the Go tool that it should look there for code. Networks just fine. I'll show you how the building 3.mk works. In this case, I show you the one for this package because we want to allow other packages to depend on it, obviously. There's a little tool called create-building that basically creates this thing on its own. So, what it does, this is the magic bit here, the building content filter. So, it takes the entire contents of the Go PKG sub-directory, which is where we install the source and PKG directories, and build links it in. The dependency method is built, so that means it will not be a dependency of the final package. It's only a build dependency, which is what it is. Here, we have repeated this include. So, if you depend on Go OVH, it will transitively depend on Go INE, so we can actually build the thing. This has another very nice property and also very important property, the compiled source code. So, the way the Go tool works is a little bit like make. If any of the source files are newer than the binary file, then the binary file is stale. So, it needs to be rebuilt. If your compiler is a different version than the compiled file, the compiled file is also stale, needs to be rebuilt. However, you can't just rebuild random things that are already installed because their file checksums have been recorded, and also the directories hopefully read only. So, when you build link the tree in, then it can replace those compiled files with newer versions if it needs to, and it doesn't count because you just delete the shadow Go path once you're done. So, that is actually, that solves a lot of problems, and it makes it kind of lower the mismatch between package source and Go. Now, I would say, so this is basically where we are, but the problem is it's still relatively tedious. So, to give you an example, because those anecdotes are totally what we need. To give you an example, I'm working on something called caddy. It's a web server written in Go, very high performance, very cool project, encryption built-in, zero config, and so on, has hundreds of other Go packages that it depends on. If you package them one by one, then you're going to be like me two months later, still packaging things. So, yeah, this could be automated, right? That's the next logical step in this direction. However, it's not trivial. You've seen a lot of metadata that I've manually added basically, because there was no release, the finding the latest commit and date and so on, and putting that in. The comment which I have mentioned is a one-line description of the package. I don't know where to get that from. What's the license of the package? So, unlike Perl packages, there's no manifest that has these things recorded in a neatly consumable way, which is very sad. Then the long description, readme.md files sometimes leave a lot to be desired. You can't just take the first few paragraphs and say, this is my description, it doesn't work well. Then the whole canonical import path versus source code location that the tooling can help you. So, you saw the Go tool discover where the thing actually is. You can use that programmatically. The same way you could do dependency resolution programmatically, you can ask the Go tool, which other packages does this thing depend on, but please filter out the standard ones. Then you match that against packages you have already in your tree, so you can write the depends files. That would be maybe a worthwhile first step. Then some of those have extra dependencies for testing. Unfortunately, package source doesn't have a concept of what OpenBSD called regress depends. So, you can't say package x, y is only used for the regression tests. The way I've been doing that is, I put it in the build dependencies in the make file but not in the build link file. But it's probably not the greatest solution. Sometimes things that depend on C libraries, like you have Go wrappers that wrap a C library. Yeah. So, I've previously written such a thing for Perl packages actually was called C-Pen to port or C-Pen to package on the package source side. So, maybe I should tackle that at some point. Would be a worthwhile thing to do or thing. Because the Go ecosystem is large and has many interesting things that may not be, I would say may not be known enough in the circle of people who were likely to use C-NetBSD. But there's some cool stuff in there. So, as my last point I'm trying to make, and that I also promised I would make in the description of this talk, is as an upstream author, like as the author of a Go package, what can you do to make it easy to package the software that you have written? So, my first point is, give us releases please. Like a lot of the Go community does never releases anything, they just push on GitHub continuously. And if you do releases, please also do release notes because we like seeing what changed. Then, we've sort of tackled the whole Go get process. So, it would be nice if your software supported that. And so, I've seen things, I don't remember what project this was. It delivered an install.go and said, just compile and run this thing and it'll do the rest. This is not likely to work unfortunately. And the most horrible part is when your software during build, for example, this install.go, during build decides to download more stuff. That does not fit at all with the hermetic building that we would like to have. And that is just necessary for making any sort of reproducible package out of your code. So, don't do this please. And also, please some read me so really bad as I said, please put more sensible descriptions. What does your thing actually do? Maybe for just libraries, it might be more fruitful to look at the Go docker output because many people have crappy read-me's, but good comments in the code that might be an idea. And avoid circular dependencies, that's a good one as well. So, the Go get works on the level of package, like it builds package by package. Whereas, package source typically works on per dist file. So, basically all of, for example, golang.org slash x slash net is one entity for us. And there used to be a circular dependency, might still be in there. There's a package in Go net that depends on the metadata package from the Google Cloud libraries. And then there's another package in the Google Cloud libraries that depends on something from the Xnet repo. So, if you try to build first one, then the other, you're going to fail. So, you have to split one of the two to get rid of the circular dependency. And that's really crappy. So, please don't do it. And that's mostly it. Thank you. Here are some links. The first one is the link to this event. You can leave your feedback here. I'll upload the slides a little bit later. Then package source org don't need to explain golang.org. Also, package source SE is super cool. It's a browsable package directory of the things that are in there. So, you can search for something like Go OVH or whatever and it will pop up a little page that gives you the description, gives you a link to the sources. You can click on the dependencies, reverse dependencies. That's really nice. And I have one bonus slide. Maybe if there are some questions first. Are there any questions? No. Okay, here's the bonus slide. So, NetBSD is kind of cool if you're using cloud stuff because of its Xen support. It runs great on a bunch of Xen-based cloud solutions such as Amazon EC2. And starting from NetBSD 7.1 are a release candidate one. We finally have enough stuff to be able to run on the Google Compute Engine, which is sort of KVM-based. However, there are still a couple of bugs. It's not quite easy to create such images. So, there's semi-official support coming. This is something I've been doing in my day job. It's not released yet. That URL doesn't work yet. It'll be there in one or two weeks. And it's basically a script where you can enter a release and a platform, 386 or MD64. It builds you an image and then you can upload. And there's a readme that explains how to use it. You can upload that to Google Cloud Platform and create your own NetBSD VMs. Thank you very much. Thank you.