 So good afternoon, guys. I hope you had your lunch, and you're all ready for the next session. I'm Shumanthro. I would be talking a little bit about Toolbox. And if you don't know what Toolbox is, I would give you a brief intro of why we have Toolbox, what's the use of it, and preferably where we stand with the project. At this point, me and Debershi, another guy we both are maintaining Toolbox, RPMs, and the images. And I'll go ahead and start this presentation. So a couple of things about me. I am Shumanthro. I work for the Fedora QA team. And my primary job role is to test packages, images, as in the compose images, and anything which comes new as a part of test days. Other than that, I do a few other things in Fedora. One of that is making sure that council understands, or rather works with whatever new community stuff we want to roll out, so objectives, Fedora ambassadors, and a few other things. I also am somebody who coordinates outreaching in Fedora. So if you are willing to do internships in Fedora, that's one of the programs I coordinate with COVID that has not been a very successful one. But yeah, we keep having more and more projects. If you are someone who wants to have projects, initiatives, or very small, bite-sized tasks you want to accomplish, reach out to me. I coordinate internships as well with GSOC as in Google and Outreachy. So I would be able to help out. I usually hang out on LiberaChat and a couple of channels that's mentioned there. And that's mostly what we have. So let's get started. First up. So a long time back when we started doing Linux distros, everything was based out of a specific packaging format. So you have the devs and the RPMs. And you would basically package everything in a dev-RPM style. The problem is everybody's machine becomes or starts becoming a snowflake problem, which is if you have a machine and you're learning Fedora something, Ubuntu something, and you have your own package set or the packages you use, it's very hard for me to debug what's happening in your machine. So if something crashes, it's nearly impossible unless you give me a full trace back of what has broken. It becomes very hard for normal general users to provide us with that much amount of logs every time they file a bug. Hence, most of the bugs that people file are marked as invalid. And they do not contribute much to us solving it, in short. So as a result, it becomes very hard for us to test what these problems are, how this problem started. Other things with when you have RPMs and devs, mostly these are packages. And updates are not really fault tolerant. So if you ever try to upgrade your machine when you run out of space or your power goes out or your battery fails, well, that's a whole different issue after that point. So historically, we had exactly these problems. And another problem is we had almost no separation between the apps and the operating system itself, which means think about it this way. If you wanted to have the latest version of Firefox or latest version of dark tables on your machine, you would probably have the latest version of operating system. If you were to expect Fedora 39 to have X version of Firefox running, and even if that's until that's packaged for F38 specifically, you cannot really, really have it. That becomes a little challenging because there is no separation between apps. And you have to might upgrade your entire production system just to support your apps. And that's a problem that has been there for a really long time. One advancement that we had a few years back was we started having these things called OS tree. Now all of you who don't know what OS tree is, OS tree is think of it like Git, but for your operating systems. So when a package maintainer or whenever a maintainer commits something, they actually package the entire thing under one hash. And every time you upgrade, you upgrade to that particular hash. So if something is failing, let's imagine a particular hash that we all are sitting in. And there is a very high chance if my Firefox crashes, your Firefox will crash as well. Because we are sitting on one simple hash. And which means if I were to fix something and deliver you a fix, it is very easy for me to tell you, OK, that hash works. Just upgrade it. And it would just solve your problem. That's a very simple way to detect problems and fix it or rather try to find a root cause of it without going into what package has changed between last two days before the failure started happening on your system and you could actually register. So that started happening with OS tree. And OS tree used to maintain a lot of flat packs and Podman apps. So all the apps with OS tree were very sandboxed. So technically, since OS tree is immutable, all the things came as a part of either flat packs or Podman. So all the client applications were flat packs. All server applications were Podman apps. And then it used to make life easier for most of the people who used to use OS tree or at the still use OS tree. The only problem with OS tree is there's no DNF and there is no, it's immutable. So you cannot really modify slash user. I mean, you can do a lot of things, but that is not really how OS tree is meant to be used. Now, as a result, one of those things that we started talking about was how can we solve this? So like for example, if you were a C++ developer or you were a Golang developer, you still need binaries. And those binaries are not in flat packs anymore. They are still RPMs. And you would need to install those RPMs. But without DNF, that's not really possible. So the other way that you could go around was installing that onto a container and then pulling that out from something. Problem is when you use or keep using Podman for multiple things, you make, your commands start getting like one sentence or one paragraph longer. Because it sandboxes its own self. And other thing is when you try to SSH between Podman and Podman and Podman, now that gives you another level problems. As a developer, when you start building apps, you really do not want such things. So the simple implementation that we came up was to ensure that all these compilers, debuggers, SDKs, were kind of set up using one single place. And you wouldn't have to spin up containers after containers after containers for doing the same. Example, these were the few questions that a lot of people had. The last one is actually a hack that is layering works. But every time you layer, you restart. Every time you restart, well, you lose things. You really don't want to install NPM dependencies and keep restarting over and over. So, yes. Moving on, we started having a new thing called Toolbox. And that was the point when we decided Toolbox would help developers run all of these things inside a Toolbox and use it system-wide however way you want to use it. That's the point when we started making Toolbox more developer or debugger-friendly. So if you are someone who is debugging code and you do not have a particular system, like, for example, I want to debug something for, or rather, I want to build something for CentOS. But I'm running Fedora. Well, my original option is to run a VM. Here's the thing or a container. Anything, for that matter. But every time I keep running Podman and keep giving it commands and more commands and more commands, that's a problem. Also, a lot of integrations with SystemD, Avahi, Neto Drivers, they need to be explicitly set up most of the time. Toolbox, you don't need to do that. That's implicitly done for you. It increases the quality of life for a developer and that's exactly how. So as part of Toolbox, we kind of have all these images that we are talking about. Hosted in registry.fp.o, right? Some images come from access.fredhead.com, specifically the rel9 images. So if you are somebody who is using Fedora and you want to run rel for testing your code or building something, you can have a rel image. If you are on CentOS, you can have a Fedora image, vice versa. Toolbox makes it really easy for you to go ahead, have a container, a bunch of containers, running very specific things, and then you can go ahead and run like your graphical applications, because it supports well ended. You can run your graphical applications as well as your CLI applications, use it wherever you feel like. This is one of those things which we kind of started talking about for a long time from a developer perspective very precisely. But very recently, we have also learned the fact that it becomes really hard when you are supposed to, one of my use cases back in the day was whenever I used to commit something for a repo, it used to, let's say that's a Ruby repo and I need a bunch of dependencies to run a lint, and I don't have 1.9 GB to give on my system for it to download things and which I would probably never use after that point. In that case, I would probably have the entire thing run on a toolbox and then run whatever commit I want to have. That way, it creates a very simple separation between things that I don't want, want it one time and go on. And I can still, if I want, because it uses Podman on the backend, I can actually snapshot all of that, keep a tar off it, import it every time I want to use it and reuse it again with toolbox and that makes developer life easy. So a couple of things that we have decided with toolbox or rather we kind of are trying to do with toolbox. One thing is we are trying to go ahead and give you a very simple command line debugger for OS-free and OS-free based distributions, namely IoT Core OS, Nome Endless OS and goes on. It is actually very important for adoption of these OS-free images or rather OS-free based operating systems because without DNF, a lot of the people who are actually working and developing apps, they wouldn't get the libraries they would want. Without the proper support of DNF, it is going to make adoption curve really hard for folks who are using OS-free. Also, there is a major thing that a lot of our developers look forward to, which is whenever the run applications on let's say toolbox, it is not actually, it is very seamless. It doesn't make your life revolve around multiple flags and parameters. I will come to it how. So other thing is we have grown our code base or rather the test code base, which is currently batch, which expands to bash automated testing, which is like 250 tests and it's growing. Mostly supports CentOS, Stream9, Fedora, and Fedora and Fedora raw hides and Ubuntu. Raw hide is the latest addition we have here because toolbox used to just get images of Fedora for last like three releases now. But in the recent one, we have added it to the workstation by de facto, which means the RPM would still be shipped in the workstation. The image won't be, but the image that you can pull this time with toolbox is the 39 image, which is the latest image. And that's exactly what a lot of people in Relenge are working. So Mikal is here. So thanks for helping us build that. All of this would also be moving to quay.io in some time. So it would not be res.fp.o like it is today. It will be moving out. And that's one of those primary goals we have with toolbox. Now that's mostly what I wanted to cover up. We are looking at four specific areas we want help from contributors. And that would be with test coverages. When I mean test coverage, this is exactly what I want to point out. We have the basic, yeah, that's a fire alarm, should we go out? Yeah, I think we are not burning. I mean, we are alive. We won't be burning. So yeah, we are looking at actually test coverages, a couple of things on test coverages. We already have the basic test coverage running currently, which is fine, which is not extensive. It is just working and it is enough. But as we go on increasing the number of platforms, which is Arch, and if we move beyond Arch, which is Cali and rest of the things, we do not have batch testing enabled for all those. And we would want contributors to come and help us over there. Other thing is as we go ahead, add more features to toolbox because it still uses Podman on the backend. We can actually do a lot of things with toolbox, which are currently not being done, mostly because we want to enhance whatever is today, make it offer stable quality before moving on to giving multiple options that Podman can still expose for you. So essentially we want to have those features, but for that we need contributors who can help us write test cases and manage those, both on the manual side of things and on the automated side of things. Open QA is one of the good place where we are trying to dump most of these test cases, which would automatically run it for raw hide because now it gates raw hide, so it would be more easier for us to catch something which has changed, which then breaks toolbox. That is one of the places. We are looking for anybody who is willing to go at their conference and talk about, because toolbox is a very small piece of software. If you have a conference around you and you want to speak about toolbox, get in touch with me or Rishi. We would help you with a slide deck which you can use to go ahead and talk about toolbox in your local developer communities. Other thing is we have a bunch of documentation, mostly upstream. We don't have things in Fedora docs yet, but if you are somebody who is willing to write documentation for us, reach out to me, I would be all game to help you write something. Now, the last part, and this is where it gets a little tricky. Toolbox relies on a lot of dependencies and when I mean a lot of dependencies, it means if something changes in system D, Podman, or rather whatever Podman depends on, it usually would break toolbox and Jens here has filed multiple bugs which has broken toolbox previously. Some of these issues are easy to debug and some of them are not and this is where we would probably want you to test toolbox with multiple OS environments and let us know if something is breaking because that way we would be able to fix it in time. This is just me and Rishi working on it, so we still need more people to come up help if things get broken. One more thing, toolbox is blocking, which means if toolbox is broken by some chance, Fedora workstation would be blocking it. In other words, we would block the Fedora release until that issue is solved which brings us to the point of how critical this can be. So having said that, that's mostly what I wanted to cover at the state of toolbox, right? And I would leave the rest of the time for the questions. So if you have any and every question that you have around toolbox, I'd be more than happy to take it. I'm just wondering how one of these test cases working for you, are they giving any interesting results? So the BASH automated testing, the BATS which is upstream, they give us some results and those results are usually in a very binary format which is either ones or zeros saying this particular parameter did not work at all or this worked and all of this is done using Fedora CI so which is fine right now but with the time as we extend more and more features, I think we would want to rewrite some of these test cases, refactor them and make sure something interesting comes up but my concerns with the bugs are not coming from what test cases we have today. It is coming from a concern of if something changes in let's say podman, it is supposed to break us, right? And which intently means that before, you know, before chain sets are filed, we, me and Rishi, we probably have to look through the entire chain set once and see whatever new is coming and if that depends or if that has a possibility to break toolbox in some other fashion, like if there can be a regression on that and we don't have a blocking policy for it currently but very soon with the fact that workstation would be blocking toolbox, we have to look at bugs well in advance which might be coming from a new chain set which has not yet been deployed to Fedora but when done, that will break toolbox for sure. Toolbox is that sometimes very long-lived toolboxes can have problems and I think this is quite difficult to detect in CI so I'm wondering if it would be useful to have some kind of archive of older toolboxes or something like to test on. Yeah, it's a slightly tricky problem but... I couldn't get your question like... The problem is that sometimes over time, toolboxes stop working because of changes. So you can't detect that in CI easily. So I'm trying to have some archive so all the images to all the container to test these things. Yes, so that's actually a very, very nice observation. So the one CI that we run, we explicitly put it to Fedora 34 and we test Fedora 34 and see if 34 works and then on a 34 we test let's say the last N plus one. So the recent F37s and F38s and sent us Nines and REL10s or REL9s, right? So in that way we have one point of understanding if toolbox works with F34, if F34 works then everything else works. Usually if that test case fails, that's usually a direct blocker for us. Like that's the... That's one thing that is done by the CI right now. So if you were to look at... I don't know if this can be... So let me show you something. See that line? That's the one that got added very recently but that's how we are making sure. So there's an older version of REL as well just to ensure that things are not broken. So we still have REL base image 9.1, like UBI 9.1 but we still maintain or rather test it on the CI. Okay, my question is why Fedora 34? Because it's no longer a support. It's an oblige. Yes. That's actually... 34 is not actually... It's not very static. We can make it anything but we want something which is outside the supported variant tree, right? So that we know that this used to work. So if something has changed between that and the latest image, right? Then we know exactly what changed. Remember the toolbox only comes with a very specific set of packages that are one-to-one compatible with the operating system that you use. So it probably won't have package... Binary is like uptime on it but it would definitely have things like DNF which are like whatever you would find. Use libdnf if that's DNF3, you will find DNF3. That's DNF4, DNF4, and DNF5, DNF5. But that would be one-to-one compatible with whatever but we just wanted to keep it 34, keep it outside the series that's supported right now to ensure that we can have an understanding. Plus, this was done when we did not have raw hide. So this still doesn't talk about how we would test raw hide stuff. So the moment raw hide lands and we actually have something that is shown up on Quay, maybe we can add that as an image and see if it runs on raw hide. So it would pull base as raw hide and then try deploying whatever we want to, yes. Thank you. So that's mostly what it is. If you guys have any questions, reach out to me. Thanks for attending my talk.