 So, this is about something called Toolbox, and if you have heard about something called Silver Blue, then I'm going to show you how you're meant to write programs on Silver Blue, which kind of sounds odd because Silver Blue is supposed to be a Linux distribution, so it shouldn't be too hard to at least write a program on Linux, right, if not anything else, but we'll get into it. I'm Rishi on most IRC channels, on email, on Fedora. I work in the Red Hat desktop team. These days I'm working on Silver Blue. I'll tell you more about Silver Blue if you don't know, if you haven't heard about it. Silver Blue is, and I have been working on the Fedora workstation, which is kind of related to Silver Blue on various aspects of the workstation, and as a side effect of that, I also work upstream on GNOME, like many of my teammates. So, I'm going to give a kind of, I'm going to try to stay a little bit high level, but please do ask if you are interested in some specific points. So, Silver Blue. So, there's a website called silverblue.fedraproject.org, but like most free software websites, they don't really tell you exactly what Silver Blue is. It's essentially in short the next iteration of the Fedora workstation, or maybe the next generation, because iteration doesn't sound as disruptive as it actually kind of is. So, the idea is that it promises robust upgrades, so you don't need to worry about like your DNF update failing and then your computer is no longer booting, because I don't know, you lost power or something went very wrong or something like that. And it's easy to roll back. So, in that sense, it's kind of like your operating system is now part of a Git repository, though it's exactly not that, but kind of gives you that feeling that you can, like reverting back and forth is trivial now. You don't need to painstakingly figure out which package was updated or which wasn't completely updated and do all that. It's just, it's trivial enough that a very non-technical person should be able to do it. If not by himself or herself, you should be able to call that person up and it's just a button press away. And another big idea is that the operating system is separated from the applications. So, like historically, all the LibreOffice packages and the Firefox packages and the GCC packages and the Grubb and the GNOME shell and the Mutter and the DBA packages and the Kernel package, they're all part of one thing, right? I mean, you can't really update your LibreOffice package without updating also your G-Lip C and everything. You basically move from one version of Fedora to another and you get the whole thing as one thing. And they, like, the LibreOffice that you're running is linking against the same G-Lip C that something else is using, like, say, GNOME shell is using or so on and so forth. So it's just one, it used to be one thing. It's kind of no longer like that. So, when you install SilverBlue, you have the operating system and the application sort of decoupled. And I'm using the word application in a little, like, I'm like fuzzing around a bit, like, I'm not exactly defining what it is, but if you're talking about applications in terms of graphical applications, then these applications are a little bit more secured than they used to be on the legacy Fedora workstation. And the operating system itself is an OS3 image, similar to CoreOS, I believe. And the applications, and the graphical applications are distributed as flatbacks. Could be Fedora flatbacks or could be from somewhere else, could be from flatup. And just so you're sure that SilverBlue is a real thing, I'm actually running SilverBlue right now to give this presentation. So it kind of works. I mean, it already works very well if you are like a non-technical user, like, because the graphical things and all these things work. But things get problematic if you are like a developer, if you are actually working on the OS itself or if you're actually writing programs to build things on top of the OS. Then this whole lockdown aspect of the whole thing starts to get into your way and it can become annoying and things like that. But the basic idea already works and if you need to give it to somebody who's a casual user, it works. It's a lot harder to break and so on. And so if you want to know more, there's a talk tomorrow at two o'clock in Panorama. It's called On the Way to the Future of Fedora Workstation. It's given by my teammates, Tamash and Yixi. So I don't want to talk too much about SilverBlue. Better go to their talk. So those were the good parts of SilverBlue. So now it's time to face the problems. So the one thing you realize when once you like, once you open up a terminal on SilverBlue is that there is no DNF. There's no like the traditional Linux package manager thing, not there. You also slowly realize that SlashUSR is mounted on a read-only file system. So just having root access also doesn't really help because it's read-only. The mount is read-only. And that's where you figure out that it's kind of difficult to actually start hacking on this machine, right? I mean how do you install your compilers like or your... I mean how do you install Node.js or Ruby or GCC or Strace or whatever it is that you want or Emacs or Veeam or whatever it is that you need. So I mean this screenshot was just supposed to be a place holder for a demo but I'll try to like actually show you what the problem is. So I'll start a terminal, try to increase the font size a little bit. So yeah, so there is no DNF here, right? And there's definitely no GCC and no Emacs or it's kind of, now what do I do? And there is RPM though, although you'd also realize that RPM is kind of in a read-only mode, so you can query the database but you can't really install new packages, which is sort of a lie but let's put it that way. And you also soon come across this strange thing called RPM Oystery. Sorry. Anyway, I'll try to just scroll. Yeah, so there is this thing called RPM Oystery and it shows you some things about your machine but it doesn't really help you to solve your GCC or Emacs problem. But what it says is that you have you're booted into this version of silver, blue, et cetera, et cetera and things like that. So I just showed you this thing to show you that this is actually something strange that's running on this machine. But it's Fedora though. I mean, if you go to Fedora release, you see that it's Fedora 30. It's Fedora but not really the usual thing. Yeah. So one thing, so there are some hacks and niche solutions already out there that work to various degrees of, I mean, the degrees of success depends on what you really want and how hacky or how niche your needs are or so on. So one hacky solution is to use something called package layering. So this RPM Oystery tool that I was playing with, it actually lets you install RPM packages like random RPM packages so you could do RPM Oystery install GCC. But the problem is that you would have to reboot to get GCC. So the reason that you need to reboot is kind of actually sound by design. It's kind of the thing that makes silver, blue, so robust. But in this case it can get annoying. You are in the middle of a debugging run or you're just trying to build something and then you need to reboot every time for every dependency of your project. It's just not the right fit all the time. I mean, usually for a non-tectangle end user the dependency problem is already solved and you just hit one command and everything comes in. But as a developer who is just playing around, it becomes a problem. And if you do too much of this RPM Oystery package layering thing, you also start to like, you also start to damage the benefits of silver, blue because the more RPMs that you layer on, the more flaky your system starts to become because of various reasons. I mean, if you're not convinced I can get into that, but it's better you go to the silver, blue talk or ask questions there so that we don't get too sidetracked. So that's the hacky solution. So you can just do package layering and reboot. And then there's the one niche solution that I am aware of that if you're just interested in building graphical applications, targeting GNOME and flatpacks, then you have this ID called GNOME builder that basically knows how to survive in this kind of environment. So it's kind of really if you want to write a graphical application that you want to distribute as a flatback, it won't work if you just want your development prefix where you just install and build random projects that you want to like, I don't know, like maybe you're a GCC developer and you want to build GCC, for example, then definitely GNOME builder won't work. So we have this thing called Toolbox. That's where the Toolbox comes in. It even has a logo these days, so it's kind of a real project, apparently, because yeah, you need a logo these days. So there's the GitHub page. So in short, what this thing does is it gives you back your familiar RPM-based environment. You have your DNF and you can do all the things that you're used to doing so far. And this environment that you get is somehow decoupled from your host operating system. So even if you completely damage the development environment, you still can boot your machine. And I think it should just keep working, I would say. And it used to be known as Fedora Toolbox. We dropped that prefix because, yeah, I mean, it was Fedora Toolbox because it was kind of driven by Fedora. And then people thought we can whatever. I mean, it doesn't matter. But the thing is, if somebody says Fedora Toolbox, you know that it's just Toolbox these days. So back to our demo. So we have this situation now. So if you did something like this, nothing really happens but you get a new shell which looks slightly different with that purple thing in the front. And now suddenly you have DNF here. And you can do something like strace. Yeah. And so that works. And there is no RPMOS tree here anymore, right? But this is also still Fedora, right? So this is the deal. Like you type in Toolbox Enter and then you get a new prompt and then you get your DNF. And then it's just how it used to be, right? Hopefully. Yeah, they should also work. We'll get to that. But yeah, but they should also work. Maybe I have something here to show you. Maybe. Well, I don't. Okay, let's just leave it running in the background so that I can show you later. But we'll get to it. So by now you might have been able to guess that there is something about containers at play here because it's decoupled from the host and there's no virtual machine involved. So it sort of sounds like containers. And it is. It uses... It's based a lot on all these open containers initiative standards and everything around that. Specifically it uses something called Podman. And even more specifically is Podman running without root. So rootless Podman. The reason the rootless part is important is because not long ago, I mean, you couldn't run Podman without root. And so was Docker. I think there is something called rootless Docker now, but I don't know how mature it is. But Docker is usually it needs root access of some kind to run the thing. So yeah, so there is no root involved here. And it shouldn't be involving root because we are essentially trying to provide a replacement for a shell running inside a terminal emulator. So it would be really weird to have your bash running as pseudo bash or something. Because I'm looking for root access inside the container. Yeah, because I'm looking for root access inside the container. Right. So should it just be... I mean, it probably won't work. I mean, I honestly haven't tried. I mean, it's kind of muscle memory too. But the pseudo inside the container is not real pseudo on your host. It's a map to some other user and it's kind of limited to that container. You can't really use the pseudo inside the container to, I don't know, do something that really needs a UID zero on the host. And yeah, so it uses Podman and containers and all these things. And yeah, so why the toolbox? So containers are kind of like what, like namespaces and some file system magic and stuff like that. And things like Docker and Podman already make it a lot easier to work with containers. So why toolbox? The idea is that like, like the average developer using Silverblue might not even know container technology very well. And even if they do know containers very well, because they maybe work with containers all the time, they probably don't know how their desktop works very well. Because we want this kind of seamless integration between the development environment and what you're running on your laptop. Because what we are really trying to do is like replace the shell for development. And if things don't work very smoothly, then it will again be just annoying. And we don't want to burden people with like asking them to run Podman with like 30 different command line flags like Podman run, dash this, dash this, dash this. It just doesn't work. You need something that will just pop up in your terminal emulator and sort of just work. So that's the whole point of having this project. Yeah, like you know, so that you don't need to figure, I mean like he was asking, like you don't need to worry about whether your graphical apps will work or whether your agent will work or whether this or that or this little thing will work or not work. So the Silverblue toolbox was inspired by this thing that Coros had back in the day. Well, not back in the day as in like a year ago. We are talking about a year back. It sounds like a long time, but not really. Yeah, yeah, they did. Yeah, and they used, and it used to use something called RKT or Rocket. That was a Coros project and SystemDN spawn, but it also needed pseudo back then, which was a problem. But this thing served as our inspiration to what we actually needed because I remember we spent a lot of time discussing what we really need to solve this problem and nobody really had a clear idea and there was a lot of strange jargon flying around like container technology, this and that. And then we came across this thing on GitHub and we were like, yeah, this seems like the kind of user experience or developer experience that you want. And then we built from there. Like, of course, the internals are not, are entirely different and yeah, but the basic idea that you type something and you get a prompt and then you work there. So I think it's still a relatively new project. I mean, given the fact that we didn't really know what we were doing when we started, we just wanted to sort of see what will happen if we do something like this. And it literally started like I think a few months after Rootless Podman was even a thing. So it kind of, they got developed in lockstep and things would break and basically like two very young projects like driving each other forward. And for better or for worse, this prototype got really quickly adopted by Silverblue users. And I kind of thought that there weren't that many Silverblue users, but apparently there were. And they are all kind of like early adopters. So they're all like programmers or contributors of some kind, like technical users, I would say. And I guess you hit this hurdle very, very soon if you are someone with that mindset that you can't really, how do you unlock this thing to do your own thing? And yeah, like I was kind of a little shocked that this little shell script that we were playing with suddenly had real users and people were filing bugs and whatnot. And they were expecting like, I don't know what, but yeah, I mean, like it's still like just a shell script because I don't think we ever, I think we are like not much of a prototype anymore these days. Like, but yeah, I mean, so that flavor is still there. So yeah, like I don't know if that other thing. Oh, sorry. So I'll quickly wrap up. So what works? Yeah, so well and the next applications work. So if you have Emax inside your toolbox, it should work. SSH agents, like if you don't need to keep typing in your SSH password, if you have divas clients and demons running like you usually do on your desktop, they should keep working as you shouldn't feel the difference. Kerberos should work, assuming that you're using the KCM credentials cache, which I think most of you are because Fedora switched to it for a while ago. So K need K list, those things should keep working. If you plug in your USB or camera, those things should work. Those things should, I mean the contents of your USB should be visible inside the toolbox. Things that don't work. Usually like, I mean usually people filing bugs is how we figure out how what doesn't work. Like long ago, like somebody filed a bug that they couldn't compile LLVM or something because SHM underscore open that system call wasn't working or something was running with some weird permission and or whatever. So people file bugs and we figure out what's not working. So these are some random things that we know don't work. If you need services running on the system divas instant, they don't really work. Various Nmap options, they need root, real root on the host, they don't work. We have seen some glitches in TMAX, for example, some really strange things. And I am not sure, like, we tracked it down a little bit, but not yet. I mean, it's not clear what's going on. The NVIDIA driver always a problem. It's like, it can be a stock answer to everything, like what doesn't work NVIDIA. But we also have a rough solution for it. Audio probably doesn't work because, I mean, just didn't look into it. Should be easy. It's just something that, yeah. We have seen some strange errors from this locale command once in a while. Yeah, something to figure out. I don't know. And that's it. So what's left to do? We need to write tests. I mean, we basically don't have any tests. And we are working with the podman team to somehow, like, keep our tests aligned. We need to write it in something that's not POSIX shell. Possibly Go, I think, because the whole ecosystem is written in Go. And, yeah. And we would like to have some wrappers for well-known commands to make things a little less rough. Because the problem starts is that if you have plenty of these shells on your computer lying around with toolbox and whatnot, then the other problem that you have is that you don't know which is your toolbox shell and which is your host shell and what is what. And then you're like, why is my RPMO history not working here or whatever? And I can type. And all these, like, annoyances build up. So the idea is that we could have some wrappers inside the toolbox environment which basically sort of magically get you what you want. Yeah, so that's it. That's all I had to say.