 Welcome back everyone. Hi, my name is Benny Siegert. Today I'm going to talk to you about package source on Chrome OS. So it's not directly running on BSD, but we're using a component of NetBSD, which is package source, and running it on Chrome OS, which is Linux-based. But first I'm going to tell you why we're going to do this thing, or why I'm doing this thing for myself in this way. And then I'm going to present a little bit about Chrome OS, because it's quite an interesting operating system in my opinion. We're going to be looking at bootstrapping package source, so that we can use various packages that are in there. And then some future work, because as I said right now, the work of getting package source working well on Chrome OS is very much a work in progress. But we're going to see that. So first of all, why am I doing this? So personally I think Chromebooks are very attractive laptops. So you might be seeing that I'm presenting from a free BSD machine, because this Chromebook and this long chain of adapters didn't really work together. But other than that, they're kind of secure. They have secure boots on by default. Operating system images are signed. Updates are signed. They boot really quickly. They don't have a BIOS nor UEFI. Instead they have something that's similar to Linux BIOS, I think it's called something else these days. So from switching it on to getting to the login screen is like five seconds, which is great. I've never had one break from the software. So the updates come weekly. They're installed in the background. The only thing you notice that in the system tray you get a little arrow icon. And it essentially says the next time you're going to restart your computer you're going to have a new version, which is nice. The pricing for the hardware is often quite competitive. I like the fact that they run Android apps, so for those that don't know Chrome in a sense is Chrome the browser is your operating system. But these days you can run Android apps on them and that's super handy. For example I'm running the first and companion Android app on this laptop and that allows me to schedule or look at what talks I want to see. And finally one of those things that you either believe in or not is you could say it's the power of the cloud. And a Chromebook is mostly a stateless device, so it holds your account. The account credentials are Google account credentials actually, so if you have a Gmail account you use that to log in. But all of the settings are synced through various cloud mechanisms and most of the data you store is going to be stored in something like Google Drive. So what that means is if my Chromebook breaks I can get a new one, I can log in on that machine and then I get basically an empty desktop and then one by one like icons start reappearing and the Chrome apps are being reinstalled and everything and then like five minutes later it is as it was with the exception of things I've downloaded to the local hard drive. So that's kind of nice, however we're going to sort of get, remove this property in a few minutes by installing software locally. But sure why not. Now as I said Chrome OS is an operating system where the browser is the OS or at least that's what the marketing speaks as. So how are we going to get local software on this thing? There's two ways, there's really three ways. The first one is you can install something called Turmux, it is an Android app that pretends to be a terminal emulator but it happens to include more or less full Linux user land. So you can configure that and then your software that you install is going to run in that Android thing and you can interact with it through the graphical front end essentially or you can set up SSH and SSH into that container or whatever you want to call it. I'm not going to talk about that, that's the way that some people use but the thing I chose is you can set up the machine in developer mode. So developer mode removes certain restrictions and originally it's meant for people developing the OS themselves but we can also use it to develop software locally by installing software. And then developer mode is a USB hardware switch, these days it's a software switch. Some, like if you have a managed Chromebook for example if your employer gives you one it's very likely that you're not allowed to switch that on so that's very unfortunate. However there's this way number three and it's kind of the sneak preview. If you go into CROSCH the Chrome OS shell you'll notice there is a very poorly documented command that's just called C and it says run something in a container and it's very intriguing. You can look into the source code to work out how it works. I think the idea is that you can set up a local container containing a user land and it would be isolated enough so that you can use it even in non-developer mode. However it requires the container to be signed in some way and I haven't figured out how that works and it's also not officially released but the command is there and the source code is also there. So developer mode it is. So how does Chrome OS look internally? It's based on Linux. It's based on gentle Linux. That's probably the first big surprise. If you ever build Chromium OS, it's the open source version of Chrome OS. If you ever build Chromium OS from source there's a thing called Repo that will download like various deep repos and stuff. And then at some point like Portage starts up and starts emerging things. It's a bit weird but hey it's a really well integrated gentle system so why not. So on the Linux kernel, Chrome is your main user land thing that runs. There is no X server in between. There is no Wayland in between. I think it's Chrome runs directly on the frame buffer I believe. And the Android run time, so the whole running Android applications locally, runs in a local container. It shares the host kernel which is interesting. So if you do a PS in a terminal you're going to see the typical thingamajig demon, the kernel processes. And then there are all these processes with names like com.google.android.la. And they're all in the same process space. So this container provides some isolation. And this isolation between users is a little bit of a figment and a little bit not. It's a bit special as so many things. There is one non-privileged user in the system. It's called Kronos. Incidentally Kronos is the code name that Chrome OS itself had internally before it was released. And of course there's the root user and various system users. When you're seeing the login screen in the UI, it presents you with all the users that have accounts on the system. It's already a logged in user session from the Unix point of view. So the Chrome that renders the login window already runs as user Kronos. And when you enter your password and you quote unquote login in the UI, what happens is for every user that has an account, there is an encrypted file system image and the encryption key is based on your password. So with the password you entered, it's going to mount and decrypt this file system container that contains your files. And it's going to be mounted under home Kronos user. So the home Kronos basically is all the users on the system, whereas home Kronos user, which is also said to dollar home, is inside this encrypted file system. In developer mode you have virtual terminal number two, control alt F2, although F2 is really the forward arrow key, opens a text console with a standard login, looks fairly standard like a gentle system, and crush the Chrome OS shell, which you reach by pressing control alt T, gains a new command called show and that gives you a bash. And so that's what I mainly use for working on the command line. So you're going to have a browser tab and the browser tab runs a terminal, and in the terminal you have bash and you can run T marks or whatever you want to run. You could also, if you don't like this particular terminal emulation, you could also set up SSH and SSH into your local machine, but you don't really need to. So that's the story about user isolation. There are a couple more idiosyncratic features of the OS that you stumble over when you use it. So first of all, the first thing I notice is that there's no man command and it is kind of weird, but then again the operating system you use is not really meant to be used on a command line, so I can see them not including it. Package source whip does have a package for man. I haven't tried that out, but that in principle should work and should give you man pages. The AUK command on the system, I'm going to talk a little bit more about that later, it's very minimal and restricted for security reasons. The editor I found in the base system by default is VIM, and VIM is compiled with a tiny feature set, so it has no syntax highlighting. I think it does have several buffer, I'm not sure if it has multi-buffer support, but it has no syntax highlighting, no diffing, no language extension support, no whatever. But you can edit files with it, so it's close enough. And it has a somewhat aggressive thing that kills processes that have buffer overruns. I don't know exactly how that is called, but for example this VIM thing, when I tried to build a full-featured VIM from PackageStores, it gets killed immediately upon start up with there's been a buffer overflow detected, so I'm going to kill this process, and I haven't figured out why. And then the other thing is that they removed various things from the system, such as NIS, which nobody in their right mind would use, especially in 2018, or a bunch of libraries that are on the system are not delivered with headers, and NIS understandable, because it's a system that you use that you don't develop in theory. And that trips up a bunch of software that essentially is badly written that says, like, this is a Linux system, so there should really be an SE Linux library with its headers, and if there's not, then it just breaks. So that's something I've tripped over a few times. And that it's actually a surprisingly standard Linux user land. You're in for a surprise when you run the mount command, though. This is a tiny abstract of it. In a full-screen terminal, mount is more than a screen full. Most of that is because of all those bind mounts and null mounts and whatever they're called, where you can, like, graft a bit of the file system somewhere else, which they use for setting up container fests and so on, and, like, there's an emulated SD card in the Android container slash a SD card directory, and a bunch of nonsense like that. But the essential structure that I would like to work you through is on slash, you have this device manager device, and it's read-only. When you set your machine into developer mode, you can set it into some sort of extended developer mode or whatever it's called, where everything gets read-write and you can SSH in without a password and a whole bunch of other things. I don't recommend that. So I recommend you to leave the slash partition read-only and keep it managed by the OS updater, essentially. So you're going to keep getting these weekly updates and every time it's going to just do random things that you don't care about in the slash partition. And you can even, once you're in developer mode, you can re-enable secure boot because by default it gets disabled and then your operating system images are verified, which is good. The other two important partitions are the one that's called stateful partition, and that is read-write. And the OEM partition, which on my machine has almost nothing. So the manufacturer of the Chromebook can put their own crap in there. And the stateful partition is the one that holds the encrypted file system image for all the users. They're just loopback mounts on that partition. And the stateful partition is also the thing that is wiped when you, as they call it, power wash the device. And it's the thing that has the main part of the internal storage. So the slash is relatively tiny and most of it is on the stateful partition so you can use it for your data. So you see home chronos user, that's the encrypted file system thing. User local is one of those bind mounts. One thing to notice is that your home directory is no SUID, no dev, and no exec. So developing inside there is no fun. However, the only bit of the file system that is exposed to the UI of the local file system is the downloads folder under your home. So what I do is I have a home chronos user downloads source where I put all my source trees so I can use like GUI editors and Chrome apps to work on those. But if I want to compile something, then I need to put in some sort of temporary directory or package source, all of it is in user local for me. So here's my recommendation. As I said, leave the root partition read only, play around in user local or the stateful partition, which is the same thing really. Or you might choose to put an SD card or something or even a USB stick into your laptop and format that as X2 without no exec and then play in that. Maybe that's also sensible because you can make a very large SD card these days. Could you jump on the bind mount bandwagon and just bind mount stuff and tickets? You could probably. I haven't tried that. But package source is not very opinionated. It allows you to install it anywhere. So that's nice. So now let's jump to package source actually. So what is it? It's the NetBSD package collection. And at the time of writing, we've passed the 20,000 package mark, which is quite a lot. So these packages are the main part of package source is essentially a large tree of make files that encode how to build all these packages, how to what their dependencies are, like what packages you need to have installed before you can even try to build, and so on and so forth. Package source is released once a quarter every three months. The last release was called 2017 Q4 was around the New Year's Eve. These releases are kept up to date with security updates. Although the stuff I'm going to show you later is all on package source current, unfortunately. And in principle, it has the possibility of using binary packages or building things from source. However, because this is a new platform, there are no binary packages yet. Package source itself supports 20 different OS types and several variants and a whole bunch of basically any architecture that those OSes might run on. One thing I forgot to mention, Chrome OS itself has four different architectures. It has essentially 32-bit x86, 64-bit x86, and then ARM in 32 and 64-bits. So this Chromebook over there is the Samsung Chromebook Pro as AMD64 architecture. The Samsung Chromebook Plus is a 64-bit ARM. So you have all the variants. I've only tested AMD64. Challenge number one, when you want to build stuff, whether you get a compiler or whether you get a tool chain from. So Chrome OS has a command in Dev Mode that's called Dev Install. That's supposed to install tools for you to develop. However, it installs a whole bunch of crap like Ruby and Python and scripting language nonsense. But no C compiler. Somebody asked on a mailing list, like, what is this? And they said, we're working on it, but we don't have anything yet. So that's bad. If this were an ideal world, we would have somebody else, me would have already built binary packages for, say, GCC. So you could just install a binary bootstrap kit at the packages and have a tool chain. However, I'm not at that point yet, unfortunately. So we're going to need some sort of a bridge to get a C compiler, now that we don't have one yet. So I found there are different things that you might try. I found this thing called Chromebrew. And it's pretty intriguing. It's based on homebrew, except for Chrome. And it's written in Ruby because it's homebrew-based. And you install it with this fun command, like just double you get a thing and pipe it into bash, which I did. And its redeeming feature is that it has binary... So this is the home page of Chromebrew. It has binary packages available for all four Chrome architectures, Chrome OS architectures, so that's nice. CPL licensed. So good shout out to them, Dave. They're quite a bit further down the rabbit hole than I am. So installing, once you've installed Chromebrew, like you pipe this command into bash, you use the command called crew, install GCC and Linux headers. It'll say it needs to install various other things. Then it says, oh, look, there's a binary package available. Downloads the things, installs. It's actually really nice. Now we're going to do the following. We're going to install... We're going to download the package source current, terrible from the package source, FTP, extracted in user local. As I said before, you enter shell to enter the shell. And then you change into the bootstrap directory and call the bootstrap script. And immediately exits and says, your shell's echo command is not BSD compatible. Oh, look, you should have another shell. It turns out if you export bin bash as your SH, it works. It doesn't seem particularly BSD, but okay. So try two. This is the bootstrap... This is actually try 10 or something. This is the current line I settled on. I'm going to explain to you what all those parameters mean. The first one, unprivileged, is really nice. It'll install everything as user chronos. So you don't have to use sudo all the time. You might like that or not. I do. Then the next thing is the installation directories. So I said the source tree unpacked under user local package source. I use user local package as the prefix. And then I put the var and package db inside that prefix. So everything is nicely self-contained. You can backup user local PKG and have your entire installation there. Then cwrappers equal no is a fun one that took me a while. So cwrappers is kind of hard to explain even. Package source uses a set of wrappers around things like a compiler, the linker, and so on. And it can do various things such as adapt the command lines to different versions of compilers. You can tell it to remove things from the command line. If the build system insists on adding a dash p thread and you don't want it, you can tell it to remove that parameter. And so there are two implementations of the wrappers. There's the old ones, the old and busted ones that are written in shell. And the new hotness is written in C. It's quite a bit faster too. However, because of the way the bootstrap detects the presence of various things, it detects there's no patch command, which is true. So it installs diffutils. And diffutils depends on a whole bunch of other things. And so it ends up creating a circular dependency where to install cwrappers, you need to have diffutils installed, but to build diffutils you need cwrappers installed. So you bootstrap without cwrappers. And then you delete once you bootstrap the line that says cwrappers equal no from your make conf and then the rest will be built without. And then prefer package stores equal yes, because the libraries, as I said, that are delivered by the base system are often incomplete in various ways. And here I put six jobs. This machine has four threads, so six make jobs is a good way of getting the maximum build performance out. And then this happens on Chrome Beta, Chrome 64 and later. It will run happily for a while and then it says I'm going to install the bootstrap mk files and then you get this fun message, awk runtime arrow pipe execution not allowed in sandbox mode. You go, what the hell is this? It turns out the awk that Chrome OS delivers has been used in an exploit as a vector of running a command. And I don't know all the exact details, but you know that there is this thing called Pwn to Own or something and Chrome OS regularly has good performance, sometimes not so good, and one of those exploit chains that did the full compromise of the system used awk. So they said, we're going to fix that and they implemented the sandbox mode for Mawk, which is their awk, and they install a version of awk that is permanently set into the sandbox mode. So it cannot run, cannot do networking, it cannot run pipes, it cannot open files with programmatic names and so on and so forth, which is kind of annoying because the dependency management of package source needs those features. So I fixed that on Friday in the train. Essentially, just the bootstrap kit now detects that you're on Chrome OS and says, needs awk equal yes, and then we build a net bestie awk and everything is fine. The other thing I did in this work is to properly support Chrome OS specific customizations by making it a sort of first class citizen in package source. Package source has the concept of an OS variant and that was invented for Solaris derivatives or SunOS derivatives. So you might know there is like, yeah, there's Illumos and Open Indiana and a bunch of others, so they are distinguished by setting this OS variant variable. So now in package source current on Linux, we essentially put the lowercase distro name in OS variant and so you can check if OS variant equal Chrome OS, then you put some customization. Some things that break on the platform, when it's upstream's fault, I try to send patches upstream. There have been a couple of ones where I did that. So back to our little use case, now that we have installed package source, we can install some software. First of all, you want to add this path thing at the end of your Bash RC so that all the commands you install in and package source are available from the command line. Then you can just go to any old directory, such as langd slash go, my personal favorite, do bmake package install. It'll download the sources, check for dependencies, install those first, then download the sources, build everything, install and create a binary package. You can take those binary packages and move them to other machines. It's all nice. If you go to packagesource.se, it has a searchable web catalog of the 20,000 plus packages. So once you find something there, you can just install and use. And in my last 40 seconds, I'm going to quickly summarize the future work for this. And I also want to say we're open for contribution. So if you have a Chromebook and you're interested in joining this effort, please get in touch. We have a debug tracker and we have a GitHub even, although it's a read-only mirror. So my first point that I want to fix is I want to have a full tool chain as package source binary packages. So that means binutils, GCC and so on. And then we can get rid of the Chromebook requirement and you will just be able to download a bootstrap kit and binary packages. And I want to package some things that are not in base. So again, there's some software that thinks if you're on Linux, you must have Posix extended attributes. However, the library is not there on ChromeOS, so you can add those by package source. Man is another example. And finally, look into support for X applications. So as far as I understand, you cannot mix graphical, I know, graphical X applications on the ChromeOS desktop, but I think you can launch an X server on a new virtual terminal such as VT3, and that should work in the future. And that's it, and we don't have time for questions. Thank you.