 One welcome to the last session. So as we announced yesterday, we are going to have a round of lightning talks So now is your chance to have your five-minute elevator pitch to Tell like your colleagues and friends something interesting like a productivity hack or the project you're working on So as I said, how this works is its first come first serve I've got one slide submission so far But of course everyone can make up something on the fly so you can talk about anything except for over five minutes That's generally the rule. So who was who wants to present something then please come up to the front row and then we go in order So I know Mark or you have something prepared Otherwise well, then it's Mark or first I would like to talk about System containers Where there is a root user and it is untrusted. I have been discussing these in the past few years of this conference with many people and usually I'm told that This is insecure, but then there is a lot of unwavering and there is never a Full explanation of why? You may want to take this as a challenge. I will probably lose it But I hope that we will learn something useful in the process so My little experiment I used the Debian stable system. So slightly old system D and kernels and spawn to create a container and In the container there is another normal Debian system built with the bootstrap, which is booted the user the usual way There The idea is that whoever has root in the container should not be able to escape to the main system I Think that it will be useful to identify two classes of vulnerabilities just a plain Denial of service like a crash in the system or booting it something like that and Actual access to the host system This is the configuration as you can see private users so username spaces Root user inside the container is not the real user the real root user of the host system. I drop Some capabilities which I felt will be insecure Most obvious is the Capsis admin which is root equivalent Capsis module still root equivalent and others that I felt they were not needed in for a normal system Are still left quite a few other capabilities. I'm still not totally sure about the Cup F set ID which allows setting file system capabilities. I Have learned today that there is some work in recent kernels that allows virtualizing that with username spaces But anyway, if that's a problem it can be removed easily because at least the Debian systems Can work as well without the file system capabilities Temporary file system is a workaround because at some point systemd expects to mount 10 pfs over there but it cannot because the ability to mount and unmount file systems has been removed by removing the associated capability so I mount I have and spawn the mount one and this works around the problem When I start the container there are a Few errors the first two are totally cosmetic and spawn already sets the Container host name so it does not matter that systemd cannot do it again second error does not matter this is a Setting which is system-wide and it has already been set by the host system systemd the other two the well there are then a few errors like the last two ones about Assessing some C groups. I'm not sure about that, but the system still appears to work So I would like to know from you if this container Is secure or is it possible to escape from it to the host system? And if it is possible why can we work around this with the current kernel features or not? Because I think that this is a pretty useful feature that should be supported. Thank you. It's up to you now Yeah, I expect I expect that somebody will have Will tell me how they plan to escape this container. Yes, that's 490 is the kernel in Debian stable. Please update your kernel that one's Basically, you're basically dead if I if I can access the local TCP calls if I can access If I can execute elf binaries this kernel has known issues you should be on four nine hundred and twenty years at least Yes, but it does not matter because that's just the Debian version Security patches are applied on the on the internet. No, no, they don't come to something Irrelevant detail Yeah, but otherwise I think Stefan grab air has just done a talk about like Lxt and how to do user containers like the non-wood containers And I guess he has a lot of answers about that too like how to use second filters and what to exclude so But thanks for that experiment Nobody else wants to go then I will go next So I'm out in pit. I work in the cockpit team and as you see in my very carefully designed nice slides here We are all over design Actually, so cockpit if you don't know it it is a Linux session in a web browser So essentially it allows you to administer like administer your your Linux servers from a web browser And all to do this we talk about that we talked to about a hundred system APIs ranging from user ads shadow the docker socket or like you discs and deep as API system the you name it and We test on more than a dozen of operating system like every single change and We want hundreds of test cases and as you can imagine with this combinatorial explosion We essentially test all of these systems a lot and we find a lot of bugs and I want to show you how we find and and track those So this is a pull request where like an automatic bot Rebuilds a fresh image of a dollar 27 in that case that happens about once a week for like all of these operating systems And this is the source where we find a lot of problems So in this case we see Here like the first naive attempt just failed. There was if there were some test failures so Then of course a human has to look into this so you stare at this for a while and figure out what the problem is So in this case it was a regression in a silly note So the next step that we do We are good citizens. We follow back downstream to the place where it belongs So in this case to the fedora as a Linux policy So you we find a reproducer and describe the problem, etc. Etc. And so the trick is we found an upstream issue That essentially reference this downstream report So that we have a pointer and we can essentially link To all of these downstream reports to Ubuntu to Debbie and to rail to fedora you name it And we have them all in a kind of github issue database on our side in cockpit and we give them this special known issue and And then we essentially create a pattern which we call a multi override That looks at the test output and if the pattern for this override matches We know that like we hit this bug in this operating system So it applies to fedora 27 and it is issue 9507 So whenever our bots then encounter encounter this particular issue They file an update to this a cop to this github issue with the exact test output and We have the pointers to the logs and artifacts and core dumps and whatnot. We have the particular time And what this allows us to do is we can automatically recognize once the bugs gets fixed because otherwise these known issues would just pile up over time So we see there's been a lot of those so Regular we want another but it goes to all of these issues and checks when it last happened and If it didn't happen for three weeks, then it proposes to close these So and once again, you know, of course a human has to confirm these so I see okay Hasn't been looked hasn't been Occurring in three days So I can check the downstream bug and see yeah indeed. There was an update here So and there was supposed to close the bug and now we have proof that it doesn't happen anymore Because on our current for dollar 27 update. It doesn't happen. So we can close this and the issue goes away and This is a problem if you would do it manually We would just go crazy like you see these are all young only known issues. So at the moment. We are tracking 49 Regressions in some operating system and the under 23 they got closed recently So this allows us to master this thing at scale without having any human effort, which is not strictly necessary So thanks So next up then wash If you came to my talk earlier, I was talking about pod man. I also mentioned build a So anyways, this is now and I by he's on my team and About a year and a half ago at Devconf We were sitting there and we're talking he's the maintainer of container storage and while we're meant to talking about container storage I was basically telling why I don't like Docker file the biggest problem to me with Docker file is it forces all of the Tools to build the container have to be inside the container image So basically you end up with like yum and Python and all this stuff if all you want to do is run a web service Why do we have all this extra content inside the container? So I We're talking about container storage and I basically said, you know, why can't I just have core utilities for? For containers, so I basically wanted just a bash Capability give me a directory. I put content in the directory and I make a container image out of it And I pushed it to a container registry So he asked me You said yeah, we could probably do something Actually, this was like the first day of the conference and the third day of the conference. He actually demonstrated it But he asked me what to call it and of course I told him just I said, I don't know I can't name things just call it builder so of course he Made fun of me and he created builder Made fun of my accent. So what what can you do with builder? So builders now a product in our suite. This is actually the The latest coloring book we represented as a dog It's a Boston terrier for anybody that wants to know and I think it looks just like now and so But anyways, it's a builder is a core utilities for building container images That's the way I like to describe it and so that one of the big commands you can do we should can do builder from Fedora, so this is based on the darker file thing But basically from this will go out using containers image and pull in it pull the Fedora image off of a container registry Storing on top of copy-on-write usually Usually over villa FS and actually creates what's called a build a container. This is somewhat different than other types of containers It's an overused term So the next point is we mount the container. So there's a build amount command this returns a mount point where the container is mounted quick segue In Docker, there's a thing called Docker copy and this thing actually allows you to Copy content from a container image to your host or from your host into a container image So they built this really cool tool to do it called Docker copy. So I decided to build my own tool I called it copy And so we're able to copy content from the host into the container or you can copy can't content in the container out to The host, but I didn't stop there. I also created DNF All right, sometimes I call it yum sometimes DNF and you can actually install root you can stall into the container directly I didn't stop there. We also created make so you can do make install dester and you put it into the container and So when you're done putting your content inside the container and of course nothing goes in the container except what you You know put into it. There's no extra stuff You can do build a config and this sets all these other fields that you see in some of the darker file like the entry point commands Environmental variables you want set basically configures the image spec When you're done with that you call build a commit that actually takes the container built creates a local image on your system Once you're done with that you can push the container image from your host to any of your favorite container registries Moves it up and basically this image is as small as possible Based on packages the amount of packages that you're gonna put in there You can put a single binary in you can do whatever you want so If anybody came earlier anytime you see red on the screen you have to say it Well, of course we support darker file because that's what everybody wants you to do So we support darker file we have build but build a build using darker file Of course, we use a shorter name for that because we're lazy engineers and we call build a bud And of course and he's a brush does not have any rights on that name. So we use build a bud which uses sports darker file Actually, it does I wrote a big scripting language for it. It's this is one of my proudest works I wrote bash So you can actually build containers using bash And and the really interesting thing is the way we can run and build containers. There's no demon No, no way the way we can share information is I built this really cool thing. It's called a file system Okay, and I have lots of different versions of it But you can basically share storage between containers and between processes and you're gonna pardon me and build a cryo All these different tools using it. So anyways, that's build up as quickly as I could do I don't know if I did in five minutes, but thanks. Oh, does anyone else want to go? Otherwise then yeah, let's have another coffee for before the closing session as opposed things You