 Hello. Good afternoon, everybody. Thanks for having us again to speak on the main stage at FOSDEM. It's great to be here. My name is Guido. I'm a long-term FOSDEM attendee. I've been developing them spare time. I also work at Google. And we're going to speak today about Linux Desktop on Chrome OS. The code name is Crosstini. Myself, I joined Chrome OS a few months ago. So I couldn't miss an opportunity to come to one of my favorite conferences to speak for the first time in my life about Linux on the desktop. So what will we go through? We'll discuss a bit like this introduction, right? And then we'll go through a bit of getting started with Chrome OS and Crosstini. We'll go through the architecture. And Michael Dylan, who is in the Mountain View team that implemented the backend of Crosstini and the machine management on which it works, will explain the internals. And then we'll go through a small advanced usage demo or failing that miserably, we'll go through slides that shows what the demo should have looked like in case things like go. So this is our plan. You've all probably heard of Chrome OS. This is a Linux-based operating system that is quite widely available but is also geared at the cloud type of work based on Chrome. One of the most exciting things that happened to it was the development of, recently for people like us, was the development of Project Crosstini, which is a Linux Debian container natively running integrated into Chrome OS so that you can run anything that you run normally on Linux in a Chrome OS machine. You can run a Chrome OS machine to other people or you can use it and deploy it in your enterprise with the advantages of a simple and managed operating system while at the same time maintaining the flexibility of a Linux system. Given how widely available Chrome OS is, which is not like amazing in every country, but still in the US you can just go to a shop and buy one, right? Like very easily. I would say that this is probably the closest, most available, Linux desktop just like comes there, right? So it's an interesting development. So starting with Crosstini once you have a Chrome OS machine is very, very easy. You go to your settings, you turn on Linux, or you search for the terminal and you click it, right? And then this is installing and then you get dropped at the probe, right? And then this is Debian and that's it, right? So this is probably the simplest installation of Linux that I have seen. On Control Shift P, you see preferences, you can, like, adapt, like, size of the terminal and so on, and then you can install your applications, install your own terminal, do whatever you want. Once you install applications, they are integrated in Chrome OS, so here we installed a few development environments, right? And then you can start them just like you click on the clips, it starts as a window, any other development environment, or your terminal and anything in the terminal, any other app, everything works. On top of that, File System and Google Drive sharing were integrated, so you can share a Google Drive folder with Crosstini, or vice versa, you can access Linux files from the file manager inside Chrome OS, and so then you can save from your Linux programs inside, like Drive shared with Linux there, and then see it in Drive, see it on your other Chrome OS computer, all these kind of things, or from Chrome OS you can manage Linux. Clipboard integration works kind of. There are some restrictions related to the fact that we don't want Linux to programmatically access anything that runs on Chrome OS, so you have to manually paste things in. This is to avoid attacks in the Linux container that extract data out of Chrome OS without your wish. So networking works, and once you start something in the container, for example Apache, you can just go to either Pangolinux Test, or for many ports just on localhost, and you'll be redirected to the container where you can test your development results and so on before uploading. Clicker is good. You manage your settings just inside the Chrome OS settings, and there's, like, right now in Dev, a few settings are more coming depending on what we implement. I'll have to do it manually. Sorry. Here. We have some roadmap of things that we're already looking at. USB pass-throughs so that you can share a USB peripheral from your Chrome OS directly to your container, support for audio so you can play your favorite games, but also so you can, like, listen to things anyway. Like audio integration, GPU integration, also for any sort of development with 3D, or anything else, and also Fuse is something that didn't work in the first version, but is coming in the next couple of versions. Working myself in Chrome OS Enterprise, I'm also going to mention about the enterprise and manageability features. So when you buy Chrome OS as a consumer, you won't see any of this, but if you buy a fleet of Chrome OS devices with Enterprise management of them, then you can manage the whole fleet centrally from Google, or you can export your policies in other way, and they will all behave depending on that. So for Crosstini, at the moment, we have only a few policies versus device policy to turn it on and off. So if you don't want any of your people to use Crosstini, you can avoid it. Then if you turn it on, you can decide whether only members of your enterprise can use it and which users can, or whether a user that is logged into the computer in case you allow friends or other people to log into the computer that are not your enterprise users, log in whether they can also use Crosstini or not. This is just really a small step. Of course, as you can see, there is more enterprise features coming to make the container controllable centrally, but we're going to talk about them when we have them and when we have designed them. This is what is coming relatively soon. I'm now going to pass on the microphone to my colleague Dylan, who's going to speak about the internals of the architecture of Crosstini and how it came to be and what is inside. Thank you very much, and see you later for the next one. Thank you. Alright, so Chrome OS is already kind of a desktop Linux. We've run on a Linux kernel. We have a user space that's built from Gen2's portage build system. We run a slightly modified version of Chrome for Linux as our main UI. A lot of times people look at me kind of crooked and say, you spent a whole lot of time making Linux apps run on Linux. At their core, they're kind of right. There was a lot of complications in doing this in a way that preserved Chrome's core driving principles, speed, simplicity, and security. Security being the big one that we really didn't want to mess up. Chrome OS has been at the forefront of desktop security since it launched. We didn't want to take any steps backward on that front to add this new feature. One of the key principles we have is that we don't let any unsigned native code access to our kernel. There's too much kernel attack surface to expose to random code. So the only code that we run that isn't signed by us that we're sure is safe runs in a Chrome renderer process and is very well jailed. So to preserve that kernel protection layer, we decided that instead of trying to make these Linux apps happy running in a jail, we would just give them their own kernel and put them in a VM. The solution we ended up with actually ends up with two Linux kernels and three complete Linux user spaces. So if you look at the overall architecture layout, it looks like that. But actually, a little bit more like this. You can see we've got our two kernels. There's three user spaces. The main Chrome OS user space. There's a user space inside the VM, which basically is there to bootstrap up an unprivileged Linux container, which is our third user space that the users actually interact with. We're using stock upstream LXD to start the containers, which is nice. It gives us a lot of flexibility. We've got a lot of pre-manufactured containers that we can run. And with that, we start a Debian stretch installation up top. We've got our installation, which jails the Debian installation. And then there's an extra boundary, the VM boundary, which is the boundary between red and blue in the picture. And that boundary is the most important boundary. And to make sure that was as secure as possible, we went and really evaluated how to make a virtual machine boundary that was as safe as Chrome's current security principles. And we looked at the available technologies, and we decided we were going to use KVM in the Linux kernel, but that we weren't super happy with the user space options that we had. So we started from scratch and wrote a new virtual machine monitor in memory safe language. We wrote it in Rust, just to give the maximum amount of protection we could from guests trying to escape into the host. We called this new VMM Cross VM because we were not creative that day. So the principle of this was that it was really focused on security first. Performance is important, boot up time is important, but nothing trumps security. You have to make sure everything is as safe as possible. So the good thing about this is whenever the guest needs to request anything from the host, the normal path is it pokes an event FD, something on the host side wakes up, and we have memory safe code written in Rust that does the parsing of all the untrusted buffers from the guest, et cetera. So we don't have memory overflows or weird exploits going on. To take that a little further, we also decided to run each emulated device in its own Linux process. So when Cross VM starts up, it's got one main process. It actually forks off a bunch of children that can run in independent jails. They run in independent sets of Linux namespaces with different views of the mount tree and with different access to different system resources. So the main benefit here is that our, like, Verdeo random device, if it was to get exploited, wouldn't have access to the networking stack. Only the Verdeo network device has access to that. And if you exploited the block device, it couldn't get to the data in Verdeo random. So all of that combines to keep the guest away from the host. And that's great. That's really what we wanted. But in a perfect world, you would never need your guest to access your host. You could just keep that hard line there. But we really wanted to integrate things. We wanted to integrate the UI. Sharing and network. So we had to punch a few holes. And to do that, we added some host-side demons to Chrome OS to help share things with the VM that we're running. The biggest one of those is there's a Wayland compositor built into Chrome now. And we share that Wayland interface into the guest. We've got a custom Verdeo device that lets us have a zero-copy display pipe from apps running in the container in the VM out to Chrome. So you can get really good frame rates if you're doing video decode or playing a game. We added something to manage the VM's life cycles. So Chrome by itself doesn't have privilege to start a VM. Again, just added a security boundary there. There's a separate process there which manages bringing up and taking down VMs as the user needs them. We added... Cicerone here is really interesting because it's the part that integrates the UI bits. So if you install an app in your VM, it notices that there's a new app in your VM and says, hey, maybe I should put this icon in your launcher. So now you can click on that icon and it just launches your app. And that communicates over... GRPC over VSOC into the VM. Finally, we had file sharing access. So you can selectively, as Guido showed, select files you want to share with the VM. So, hey, I've got this thing in an email in Chrome OS. I'm going to download it. It's in my downloads folder. And now I want to edit it because it's a C file or something and I don't have a good web editor for that. So I can click on it, share it in. It gets shared over 9P into the guest and then the guest can open it up and edit it in Vim or whatever they want. All these host-side demons also need endpoints inside the guest, so they need somebody to talk to. So on top of our default Debian installation that runs inside the guest, we've added a few... I'll do this one first. We've added a few demons that talk to the host and they're aware they're running in a Crosstine EVM. The main one's called Somalia and it's used to help X11 apps run against our Weyland compositor and to kind of bridge standard Weyland into something that works over VertiO Weyland. We actually have to run three of these. One for standard Weyland apps and then one for a low DPI and one for a high DPI because most X11 apps are awful at running at high-res screens like we have on some Chromebooks. And in addition to that, we added Garcon, which is the guy that talks to the UI component outside of the VM. So this is the thing that notices when an application is installed and shuffles its icon out to Chrome OS or gets poked by Chrome OS and says, hey, please launch this application or please install this application. So the VM we run, we called Termina, this is the thing that starts inside the VM, so it contains the init for the VM. And we wrote a custom init that was, again, super stripped down, knows it's running on a Chrome OS machine, knows that its only job in life is to get LXD up and running and start a container and do some lifecycle management things. Once it starts, it brings up LXD. We've got one more daemon we wrote which is there to kind of bridge the standard LXC container commands out of the container and out of the VM and into Chrome OS so that we can poke it with GRPC and it can talk to LXD over its native interface. The main advantage here is that all this VM side code is completely read-only, signed and trustworthy, so you can always get and start up to this point. No matter how corrupt you blow up your container or anything, all this is all unmodifiable and all signed by Google as well so that you can trust it as much as you trust any code on your Chromebook. Okay, so that gets you to the point where you can run our default installation. Our default installation is Debian Stretch. It's pretty basic. It suits my needs fine. It suits a lot of users' needs fine. Although I imagine there's a few people who want to do something different or want to explore more what they can do in this room. Some people like different distros. And all this other stuff. And we've allowed you to do a fair amount of playing inside the VM. So we're going to try to do a demo of one of these more advanced usages now. Thank you very much. I'll try now to do a quick demo of this. If it doesn't work, I do have some slides. I'm going to publish then the slides with basically screenshots of the demo, but now I'm going to try to do it live. Do you have an idea how I can just replicate the screen? Because it's going to be very... Do you want to hear it? Yeah, I want to hear it. Okay, so we've already seen on the screenshots how to just install it and do it normally, but if you want to interact a bit with Chrome OS, you can just press CTRL-ALT and this Chrome OS shell will pop up. Is it big enough? Should I maybe increase the bit? I'll increase the bit. Should be better. So once you have this shell, you can type a lot of different commands to, like, see and interact with your Chrome OS system. Right? I'm going to just type VMC and this tells you how to interact with these virtual machines. VMC list should show that there is one terminal virtual machine. You can create more. There is not a lot of need because then inside each terminal they all look the same and you can create different containers. So right now, I'm just going to type VSH and this allows you to join either a VM or a container inside a VM. If I type VSH terminal, this just connects me to the Chrome OS user inside terminal and this Chrome OS user basically can run LXC, right? So if I do LXC list, it shows my virtual machines. On a normal installation, you will just see Penguin, which is the default container which runs Debian and it's IP address, right? From this, you can also see, for example, the network site and you can just use LXC in various ways to study your situation, like as you would normally and as you can check in the man page. One thing we're going to try to do is list our remotes and this shows you basically these are the default remotes that are published in Chrome OS. You can add more. Images is like images on Linux containers. So if we use that, we should have a good amount of random Linux distributions. I'm going to list AMD64 images that are available there. There's... Yeah, good luck with the Wi-Fi. If it doesn't work, it's okay. I remember a few... I don't know if we had a network connection here or something. Okay, we're giving up on that step on a good Wi-Fi, like when not everybody is downloading their updated apps at the same time, it works. From there, you can copy an image. So for example, LXC image copy, Alpine Linux, which is very small on local. You can copy it, so it should work very fast just checking. If it doesn't, I'll just move on. Okay, so it's completed successfully. It didn't have to download it because it already had it. Then I can go and see Image List. And I have some, like, versions that I had already copied from before, but you can copy anything that you see in that images, where Subuntu, where it's made. There's a lot of various distributions, and otherwise you can find your own container images and just put them there, right? Just adding another remote. Then from there, I can just launch a new container, right? So now I'm going to launch it for real. So I'm launching an Alpine 3.8 container still because it's small, and they're going to call it Alpine 1. I have a few that were already running, but this one is new. So it's creating it, and it started it. If I list now, I used to have Alpine 0, Alpine 7, and Penguin, where Alpine 0 and Alpine 7 I was just experimenting on, and now I have Alpine 1, and I can just connect to a console, and route on my new container, and of course I need a bit of network, but hopefully. Otherwise I have a container that is already configured. You know. What am I to do? This morning it worked, but everybody was sleeping after the party yesterday. Unfortunately, everybody's awake and I touched something. We have a bit of time. That's where it's in. So I now have my packages available. Then I can go apk.add apache2. Not too bad. Also open SSH. Maybe we'll just use apache2. Give it. Okay. Then I can start them. I'll start apache2. I'll just add a user. Of course, of course. It's a very demo. I would use a real password, I promise. I could also have changed, you know, to allow a direct route to log in without... Don't do that, right? Okay. Cool. And now that we have that, I can go in my normal container, right, and then I can just go SSH on my new container, and that's where they work. Of course, we trust it. And... I'm in there. Or, since I installed apache2, I should be able to just open Chrome. I don't know... It shouldn't really work at alpine1.linux.net first. I haven't managed... Exactly. So... I'll just go AlexeyList. Alpine1.1 was this just going Chrome to my IP address, and it should just show me my default installation of apache. This... Thank you, thank you. Amazing. And of course, this is mostly meant for developers, so you can publish your dev things and then try them in Chrome directly. You can use it really for anything. You can run your little services, experiment, perform. At that point, you are free both on the distribution side, deploy your containers, pretty much anything except the things that we're still working on that I listed in the roadmap. This concludes our talk. There is some time for the Q&A, and we're going also to leave you our contacts and our list, so that... So this is just what we've just done in the... network now. So thank you very much if you have questions. Can I say... Hi. I have a little question. Okay, round to the question directly. Thank you. Before Christine was released, I was using something else called Crouton, I think. It was like a community where someone who made it in his leisure time, did you collaborate with him or did you use a totally different approach than what he did? Thank you. From what I know, Christine was inspired by Crouton, but the approach was different to allow it to run in a non-modified, non-dev version of Chrome OS. Maybe Dylan can speak more about that, but we have several of the Crouton maintainer types are actually on our team and had done that, so they contributed a lot of ideas and we're excited to see us basically make a verified mode Crouton, which is what we started calling this and hence the cross-dini name. Hey, I've played around a bit with it and as far as I understand the port porting stuff is only between the container and user space and it's on. Okay. Okay. So I've played around with the cross-dini and as far as I understand the port forwarding is just between the Chrome OS user space and the container user space inside the VM, so there's no way for me to SSH in and out of my container to another device. Is that right or is there a way to do this? Yeah, that's right. That's on purpose, at least for now, we haven't come up with a way we're happy with to expose ports from inside the VM to the outside world yet. It makes us feel a little creepy, we don't have any open ports now and adding them is kind of scary from a security standpoint. Thank you for that. On the other hand, you can for sure connect from inside the container outside, you could do some SSH forwarding or other things if you need to. This is mostly meant again in this case. If you want to run an onion service on Crescent, you could and it would work, I guess but the idea there is you connect from there from your own machine or among all the containers everything also works right so you can SSH between different containers on your machine and so on you cannot create a network of Chrome OS devices unless you have some other layer there but that's not by default one is how strong is isolation between containers they are running in the same VM or it's possible to have separate VMs for each container and the second question about GUI integration, is it possible to extract single windows or from application running inside those Linux containers or it's only just some full desktop something. You can start multiple VMs and have end containers in each VM didn't quite catch the second part of that question any of them can display windows if you'd like so they all get the connection to Chrome's Wayland socket but you need to if you're going to run a non standard container you need to go do the plumbing that has the special demons I talked about running inside your non standard thing so it can talk to a slightly modified Wayland protocol or of course you can port all of your tools for your favorite distribution I have a little question about security you said that you're trying to keep your software as secure as possible however when you buy a Chromebook the security update stops after seven years between five and seven years so how can you say secure and is there a dependency between what you're proposing and also the Chrome OS you have on your computer. Yeah eventually they stop updating but if you really want to keep using this ancient Chromebook you can keep pulling builds and updating it yourself if you'd really like and you can sign it yourself obviously we're a big believer that the hardware is yours you can go pull the right protect screw on your seven year old Chromebook and start signing your own things if you don't trust Google's signature either so how secure it is is really up to you if you want it to be or you can do the easy thing and let Google worry about it and you can do it too does this work on a Chromebook that isn't in developer mode sorry could you say it one more time does this work also on Chromebooks that aren't in developer mode yes it works in developer mode everything that works in regular mode works in developer mode it works in normal mode by default yes you don't need to be in developer mode if you want to be in developer mode you can be in developer mode feel free so half my question was answered by the FAQ it's not supported on the ChromeOS machine I have so I'm wondering why and if I did like try and build do my own builds like what would be stopping me from what makes it difficult to support like a 2013 pixel a first gen pixel which has like all the horsepower you would need to do VMs but Google has stopped updating our goal is to get it to as many Chromebooks as we possibly can we really do want to bring it to as many Chromebooks as possible there are some challenges on some of the older ones some of them don't run the newest kernels and don't have some features that we need and updating their kernels is rather expensive because there's too many bugs in new kernels and the combination of the old kernel with the recent Intel speculative execution issues has made it very hard for us to update some of those older big core devices in particular so it's something we want to do we're just trying to figure out how could you say a bit about your approach about GPU acceleration inside the VM we're working on it it's a big chunk of work to get the GPU exposed into the VM in a way that we're happy with from a security standpoint but we do have people exploring that now so hopefully someday soon it's about as non-committal as I can be with the schedule I have a question about just about distribution of apps I mean if we have a Linux app that we want to make available to Chromebook users but we don't want them to have to bring up shells and go through that whole interface is there a plan to have an app store type of mechanism for installing apps our initial pass here has been very focused on enabling developer use cases and not so much on app use cases so today that's not really super smooth so if I just want to go click on an install package it won't automatically get me through this whole setup flow if I've gone through this setup flow and then I can double click on a .deb package and it will just install but you still need to go through this initial setup which is kind of a barrier as this has still got a beta tag on it for some good reasons and it's really only tested for these development use cases we wanted to enable people to develop webpages, develop progressive web apps these types of things on a Chromebook first and then we're looking to expand those use cases in the future I'd like to encourage that thanks have you looked at the performance overhead do you have anything virtualized do you have any numbers about it there is a there is somewhat of a performance aid it depends on your workload the biggest things are normally IO heavy operations that do a lot of VM exits so if you're looking for like the absolute utmost performance from the hardware it's not quite not quite native but it's really close to native and it's been fine I've been using it as my only development device for months and months and I've been okay with it for things like building kernels, building cross VM building audio servers things like building Android I still go to my big beefy desktop okay thank you so in your Apache test it was connected via HDP and IP addresses and you can't open ports you can't open ports so let's encrypt isn't really an option for HDPS I'm wondering if there are any thoughts to making like a local let's encrypt kind of style certificate authority that maybe Chrome would trust and some sort of way to create like local domain names that could then be used in that scenario yeah so for development we trust the container like we trust localhost so we're cheating a little bit on that and that's probably good enough for the use cases where someone is developing a web page or a PWA it lets you install PWAs and that was our main goal and we don't have a lot of plans to enable this as something you'd actually run a production service on because it's still just a laptop right we're not looking to take over the server space here great okay thanks a lot everybody