 OK, next up, we've got Simon Maravid here who's going to be talking to us about containers and Steam. Hello. So unlike most of the people who've been speaking in this room, I'm not talking about mission critical services. I'm not talking about securing your database or anything like that. I just want to play Half-Life. So I'm sure you all know about Steam. It's Valve's app store for mostly games, a few non-game apps as well, on Windows, on Mac, on their own Debian-derived operating system, SteamOS, and on generic Linux. The games are labeled as SteamOS, but actually they work fine on most distributions sort of. I work at a company called Collaborer. We are an open source consultancy, and my current project is helping Valve with the Steam runtime, which is how their Linux games manage to work mostly correctly on SteamOS, on Debian, on Fedora, on Arch, whatever. The problem is, if you buy a game quite reasonably, you want to play it. You want it to work on your computer. But Linux distributions are all different. Does it use libJPEG 6 or libJPEG 9? Does it use libSSL 1.0 or 1.1 or something from the distant past? You can't really know quite what you're targeting. Or you sort of can, because the LSB exists. But that's got like three libraries in it, and no operating systems actually use it anyway. So it's not a great baseline for running all your games on. So Valve invented this thing called the Steam runtime. This is from like 2012, 2013, something like that. So you have your host system. I'm a Debian developer, so my host system is obviously Debian. And inside that, we have this box. This is, there's a horrible shell script that sets up an LD library path about this long. So that a bunch of libraries from the Steam runtime get used instead of the ones from your host system. So this is the traditional solution for portable games. You bundle all your libraries with your game, except Valve have bundled all their libraries with their launcher so that all the games get to share them. And the Steam runtime releases, there is one so far, unnamed after Team Fortress 2 characters. So this is Scout. And if your operating system is similar enough to Ubuntu from the distant past, you can run your game and it works. Yeah, that LD library path is a massive oversimplification. The real one is huge. The problem is, you know I said if your operating system is similar enough to Ubuntu from the distant past, it turns out in a modern operating system this doesn't work. So the major things we have to get from the operating system are the graphics driver. One of them is the graphics driver. So in the open source world, we all want to use Mesa or potentially some other open source driver, but yeah, it's Mesa. And Mesa needs a bunch of dependents. No, I don't want to restart Steam. Right. Thank you, Steam. Offline mode, well respected. So we need dependencies for the open source drivers. We need the LLVM libraries, which are enormous. We need the standard C++ runtime because of that, which is pretty big. We need LibDRM. We need LibGCC. That's kind of core thing, kind of important. And we need GNPC, obviously. And if your GPU is from like this year, you need a version of Mesa that supports your GPU, so really recent as well. And there is kind of anecdotal evidence that if your kernel and Mesa are about the same age, that's probably going to work best. The kernel is meant to be perfectly backwards compatible at all times, but that's really hard. So it's just mostly backwards compatible. And then the other side of things, we have the proprietary NVIDIA driver, and historically the proprietary FGLRX driver for ATI stuff as well. And these are similar to Mesa. They have a user space part, and they have a kernel side part. Unlike Mesa, they require the kernel module and the user space part to be exactly the same version. The communication between them is not backwards compatible. It's not forwards compatible. They have to match. So you can't bundle the user space part of the graphics driver because then it won't match the kernel part for most of your users, and that's not good. Also, I mentioned G-Lipsy. G-Lipsy is kind of fundamental. We can't really get away from that. And it has this thing, the dynamic linker, which is what actually starts your program, like your game or whatever. And the path to that is hard-coded. It's part of the elf headers of every binary. You can't change it. It's part of the ABI. You don't get to choose what path people are using there. It calls into libdl, which is the actual implementation of loading shared libraries. If they don't match, their assumptions are going to fail because it's like, call this function, this private function in libdl that does the stuff we want. Oops, it's not there. Call this other one. Oops, it does something different. Bad news. Similarly, libdl is inextricably linked to libc, and libc is linked up with a libp thread, which does locking and stuff, that kind of thing. And if they don't match, you're in a situation that the G-Lipsy maintainers have never tested, can't really be expected to have ever tested. It's like, this is not going to work. So we don't have graphics drivers in steam runtime, and we don't have G-Lipsy. And unfortunately, if you are using the graphics drivers from the host system, they will depend on a version of libllvm and all that that is at least the one they were compiled against. Reasonable assumption, except if you have replaced all the libraries with Ubuntu from the distant past, that assumption breaks your game crashes, which is why the 2013 era steam runtime stopped working. So what we now have, you'll notice the same diagram, but the box is a bit paler now, because instead of having, we're using all the libraries from the runtime, we are now looking at those libraries, comparing them with what's on the host system, and picking whichever one we think is newer. So if you are running the steam runtime on something really ancient, like Ubuntu 12 or Steam Host 2, which is pretty old by now, or something like that, a bunch of the libraries in the steam runtime that Valve consider important for games will have been upgraded. So for example, we have a newer version of GNU TLS, we have a newer version of SDL, and it will prefer to use them. Others of your system libraries are going to be newer, like in practice, you will have a newer, like libJPEG or GTK or whatever, and we'll use those from your host system instead of ours. And again, the LD library path is a mile long, even longer in this case in fact. And it works great, except, comparing versions of libraries, you might think, oh yeah, yeah, you just look at the sim link, it's a sim link to like libcurl.so.4.1.2.3, you compare that with libcurl.so.4.3.8, and you can see which one is newer, except a few libraries, the so name, the like canonical name of the library, is just a regular file, not a sim link. So you can't easily tell which is newer. We have some code in progress to improve on this by looking at the symbols exported by the library and the versioned symbols and that kind of thing and making a better educated guess, but it's still fundamentally a guess. If your libcurl is newer on the host system, as it often will be, it might not be the same ABI we've got as a result of some with hindsight bad decisions made several years ago in Debian that have since been fixed, but the steam runtime is based on a version before that was fixed, so yeah. And OpenSSL removes symbols from its ABI when you turn off support for insecure SSL versions. So if you have a library that calls into it, like libcurl or something, it may not work with the host's libSSL because even if the so name is the same, that symbol has gone away. So we have to hack things like libcurl to not try and use the old SSL versions even if it detects them. Also, the dotted line on my diagram is not a hard boundary. Game vendors will often compile their game test their game, it works, good, ship it. Even if some of their dependencies are in fact things that came from outside the runtime and we never shipped a copy of that. If you happen to have that on your host system, their game works, great. If you don't, problem. So, big thing I have been working on lately is pressure vessel. This is a little open source program recycling a bunch of code from Flatpak. So Steam is still in the old style runtime. But as a child process, we have bubble wrap represented here by its mascot of a bubble wrapped cat. This makes a little container. So quite similar to what Flatpak does. Much of the same code. In this container, it's a very strict scout environment. You get the libraries from Scout and no more. If your library is correctly done, was built in a scout environment like Half-Life, for example, it works fine. If your game is one from a less strict publisher, it probably won't work. But it didn't work for some people anyway, right? So at least they can QA it. If they test in this very strict container, it should work for everyone. Hard part of this is we have to hoover up the graphics drivers from the host system. And because they have dependencies, we still have to get the dependencies into the container somehow. And there's some mad hacks in pressure vessel to make this work. Remember I said the LD.so path was hard coded. We have to find out the right LD.so over the top of it to make this work. We wish we didn't have to do this, but it's kind of necessary. Because the graphics driver in your container wouldn't support your hardware because it's old. As an experimental side benefit, we can do other fun container things. So we can give you a separate home directory for each game so that anything in that home directory, we know it came from that game. We can send it to the steam cloud. You get your save games everywhere. Except currently this feature, the actual steam client doesn't know about the separate home directory. So cloud autosync is worse with this feature. The steam workshop also doesn't know about this yet. Unlike a lot of container stuff, we're not trying to do security here. It's not a security boundary. It would be nice, but it's not the priority right now. Meanwhile, you may recognize this guy giving a talk in a room similar to this one two years ago. So the flat pack has come along. People in the community, not me, have made the steam client into an unofficial flat pack app. So here we have flat pack on my Debian host system. Starts the container using bubble wrap. This container is using the free desktop SDK, which is a set of libraries done by the free desktop.org people. And inside that, we have the traditional scout runtime just the way it was a few slides ago. So this is a security boundary, at least a little bit. It's kind of porous right now, but we'll get better. As far as the steam runtime is concerned, the free desktop SDK is in the role of the host system. So we get the graphics drivers from the flat pack runtime. We get some libraries from the flat pack runtime. And a unique feature of this thing is that there is some madness with an LDA audit plugin inside the container to force broken games to work, even if they're bundle libraries they shouldn't, which is great and I'm amazed it works and we should steal it. Unfortunately, combining these two is pretty difficult because some of the constraints that are put on flat pack containers mean we can't enter another container. So the stuff we want to do in future, old games have to keep working, even if your GPU is new. For new games, we want new runtimes, so more team fortress characters. We currently have a we currently have a runtime being tested called soldier, which is somewhat newer than scout. Heavy and spy are slightly older ones that we tried but abandoned. And when we go into this newer runtime environment, we don't want to repeat the mistakes of the past and have your game not work reliably in the, in a new distribution in like five years time, or ten years time. So we want everyone who is testing their game in soldier to be testing it in a strict soldier environment that they can't accidentally pick up dependencies on libraries we don't have and that kind of thing. So one strategy for this is that because we've got pressure vessel, we don't need to pick the same runtime every time. So if we imagine that half-life had been ported to be built in soldier rather than in spy, we could just make a different container and this means not all the games have to have the same runtime. The games don't have to have the same runtime as the Steam Client, everything's great. Another possibility is where, because mistakes have been made and blame was ascribed, some games have dependencies that are not in the scout environment. But those games basically work on scout in Debian 10, we have like empirical evidence that works. So why don't we put the old runtime inside the new runtime and use the fact that the new runtime is pretty similar to a current version of, for example, Debian. And that might well make games more reliable. And again, we don't want to do this for future games. It's just for current ones. And finally, we have this setup that I have been trying to get working. Flatpak now has support for what they call sub-sandboxing, where instead of making another container inside your container, which you're not allowed to do, you can send an IPC request out to Flatpak and say, please make me a container, make it look like this, go. And so you have the Steam Client in your original container in the Free Desktop SDK environment. And then over here, you have half-life in your scout container. Flatpak can't currently do this. There is not all the code yet to be able to ask it to use a totally different runtime. But the Flatpak developers seem to be fairly receptive to the idea that that's something they might do in future. So hopefully we can get that working and have these parallel containers. And Flatpak users will be able to benefit from this multiple runtime thing. Any questions? Or alternatively, do you want to demo? Right. So here is the reason Steam was nagging me to update it. This is not a special magic hacked-up Steam. This is just the normal public beta of the Steam Client. I don't even have access to the private ones. And the only thing I've done a bit differently is I've run it with this magical option to ask PressureVessel to give me a GUI instead of just running the game straight off. You can get the magic incantations for how to control PressureVessel from the source code. So this will take a moment because it may have to unpack my runtime. So I have this little test UI that I've asked for. Normally you wouldn't get this, normally you'd just get a game. This is my development version that has all sorts of inadvisable options, like making a process ID namespace which doesn't even work because it confuses Steam. So I'm running it in the Scout runtime. It's giving me a choice of runtimes. I have a copy of Soldier available. I have this option here to put an LD library path runtime inside the container. That doesn't work yet. And I'm going to tell it to run a shell instead of actually running Half-Life. So this is a thing I put in for debugging the environment. This is an X term inside my container. So if I catetc.isRelease, you can see that this is not my Debian host system. This is a purely Scout container and slash user and all the rest of it came from Scout. Except for this override directory which has all the stuff I've yanked in from my host system to get my graphics drivers working. And this shell has the command to run stuffed into this variable. So you can run your game. A bit of logging happens. And welcome to Black Mesa. And we are out of time. Thank you.