 Good morning. Good afternoon. Good evening and welcome to another edition of the Level Up Hour. I am Chris Short, executive producer of OpenShift TV. I am joined by two wonderful red headers, the illustrious Langdon White and the always brilliant Dan Walsh. Welcome, both of you. Langdon, I'll let you take it away from here if you want. Sure. We are here with Mr. Walsh, as self-referred to as Mr. S. E. Linux. We all know that to be true. As always, I'd like to introduce the show just a little bit, just to say what we're doing here and why, and maybe share some slides. This, in case you are lost, is the Level Up Hour. We'll see if everything is laid out correctly. The Level Up Hour here, we talk about containers for everyday usage. Why you as a systems operator or a Linux sysadmin or an OpenShift operator or whatever role you have, not just developers. Developers are welcome too. I come from a pretty strong development background too. I think the use case for developers using containers has been pretty well made in a lot of places. I use containers for all sorts of stuff all the time. That's what we talk about on the show. That gives us a little introduction to the show. You can find myself on Twitter at Langdon with a one, and Chris Short. Actually, Dan's Twitter handle is our hat Dan, if you want to follow him on Twitter. We're always also available to chat most of the time. We're awake on our Discord, which you can see the link there, but I bet Chris is going to use his magic to share it in the channel so that you can go and check it out and join us there. A little bit from a little bit more introduction. We have the show notes from last time are at that URL, which I will throw in the chat with some fancy clicking. Look at all that copy-pasting that I did this morning. Look at you. I'm so proud of you. Today, we want to talk about SE Linux containers and OpenShift. That's why we asked Mr. Walsh to join us, because he knows stuff. I think that's where we'll stop with the slides, because we don't want to give away the internet points quite yet. Dan, if you are unaware of what the sweet internet points are, we will talk about them later. As I sip my coffee, we usually try to do some announcements. I know myself and Dan are both giving talks this weekend at DevCom CZ. It starts on Thursday, ends on Saturday. I don't think there's anything on Sunday. Dan, you want to tell us a little bit about what you're talking about this weekend? Yeah, I have two. One is a BOF, so I run a containers BOF every year in this on Thursday. You'll have to look at the schedule. I think it's around 9.30 Eastern Standard Time, whatever that is, and CET. It's basically just a conversation that we have every year. Just get together people, talk a little bit about what's going on in the container world. And then try to get, answer people's questions. I try to get as many people from, by the way, I'm the chief architect of all containers at Red Hat. So I bring my team and get them on and get them talking about different stuff. Usually we talk about futures at it. Well, I will say, unlike a normal BOF or whatever, that BOF usually is like 100 people, right? 200 people. It gets packed and there's lots of stuff and it usually runs way over. So flocks of a feather, not birds of a feather. Yeah, exactly. So it is highly recommended, especially if you're doing something kind of esoteric with containers. You know, there's a lot of good discussion there. Unfortunately, we will not be serving beer this year because it's hard to deliver. But please join the virtual BOF at least. Yeah. Then I have a talk on Saturday morning, believe it or not, at 6.30 AM. I just love this pandemic since it's a European conference. It's rather early in the US. But that talk is just basically looking at it's sort of like a year of view of Piedman. The talk is talking about all the new features that have gone to Piedman. We'll be talking about some of them probably in the next hour. But basically it's going to go through lots of demos and just showing you all the cool stuff. This past year in Piedman has been incredible. We just released Piedman 3.0 this past week. Yeah, we're pretty excited about it on the show actually because we talk about Piedman all the time. And I complain about it and then I go file bugs. Sometimes they get fixed, sometimes they get closed and I'm told I'm an idiot. But whichever works. I also, I'll just pitch, I have, I'm actually hosting a keynote of some student research that's going on around understanding, sorry, we got some spam in the chat. We basically looking at microservice textures and trying to understand what's actually going on in them. It's this open source project called profit. So that's at 8am on Friday. And then my talk is at 9.30am on Friday talking about event driven architecture really using service mesh, serverless. I'm going to try to work in some demos. I didn't realize how short the talks are this year. So I may be doing less demos than I expected. It's probably better because doing virtual talks that are long is rough to watch. But Chris, how short? Like 20 minutes? Yeah, 20 minutes. I was thinking it was more like 30, 35. And but so I'm going to, I'm going to, that's tight. Yeah. So, so that should be good. There's lots and lots of other stuff. You know, obviously we're going to, we can't highlight too much of it here, but I threw the link to the website, which has a link to the schedule. It's free. It's virtual. It is on central European time. So, you know, if you're on Eastern, you're going to miss a bunch. But if you're in Europe, which I know some of our audience members are, or, you know, in Asia, parts of Asia, maybe the time zone will work out better. So definitely recommend that. The other thing we're going to talk about was container plumber's conference. Yes. Container plumbing, or I'm sorry. It's called container plumbing day. Plumber's day. So something like that. It's at container plumbing.org. So this, this all came about a little while ago. We had some ability at Red Hat to basically throw together a virtual conference. And I, it was, it was brought up to me to do this. And I, a couple of conferences I love to go to, you know, it used to be the plumber's conference in Linux, which we talk about. So low level, you know, stuff in the operating system and booting up. And another one is all systems go, which is used to be the system D conference. But basically I was talking about things like system D and new features of the operating system, but basically all low level parts of the operating system. And so when this conference Justin came to me, I said, why don't we do a low level containers plumbing conference? Not I want to avoid Kubernetes, OpenShift and all the high level orchestrators and stuff like that, but really talk about what's happening at the really at the operating system level to make containers better. There's, there's lots of changes in the file systems, there's lots of changes in your system D, lots of change features being added. And so our goal was to really get a deep low level look at some of, you know, some of those core technologies for running containers. And that's, that's really what the conference, that that conference is, it's got it. So it's first time we've done it, we've got a whole bunch of people submitted talks and now we're going through the painful process of figuring out which talks to have at it. And so I guess to be clear, the container plumbers day is two days, March. So much like the level up hour. It's, it's not exactly how it's advertised on the on the team. The one of the things I always found interesting about the Linux plumbers conference, right? So basically the kind of software engineers that drive, I don't know, 80% of the low level internet, let's say, always a conference that is kind of done on somewhat of a shoestring budget. So for example, it was held in New Orleans during hurricane season, a couple of years ago. And I was like, you know, that whole VP rule of like, you can't put two VP's on a plane together. Let's put all of our top kernel developers in one place during hurricane season and just see what happens. So yeah, sometimes I wonder about the choices we make. But so we, so I actually think that'll be really interesting. You know, like, I think a lot of people realize is like, we've had some pretty like some stuff that could have landed in the Linux kind of kernel a long time ago, that but has really been driven by containers, right? So like overlay file systems, you know, or like the concept of them being in the kernel level really didn't materialize until containers started pushing them into the kernel. And so I'm not sure what's our what's our current favorite kind of the what's the current choice for the overlay file system? I know it's an overlay FS. No, right? Yeah. So well, overlay. So right now, I mean, to give you an idea of what's going to be talked about at the conference, going back over like file system is the thing that almost everybody uses for containers in the world right now. And one of the interesting things with with the pod man effort is pod man is is rootless containers, right? So basically allowing you to contain us as rootless. One issue that's always been in the Linux kernel is controlling who's able to mount file systems. And with with the advent of user namespace, which what pod man takes advantage of and running rootless containers, there's certain file systems that are allowed to be mounted by inside of username space. So you can't mount physical file systems, you can't mount NFS, you can't mount lots and lots of file systems, even if you're in username space. But inside of username space, you're able to mount the sysfs, procfs, slash tepfs, bind mounts, and fuse file systems. So over the last few years, we've been using fuse overlay. So basically one of the guys on my team, Giuseppe Scravan, rewrote the overlay file system from the kernel is a fuse file system. And that's what everybody in the world, in rootless world, is taking advantage of containers. The problem with that is the user based file system is going to be a lot slower than a kernel based one. And so I've been talking to the kernel file system guys at Red Hat and pushing them, mostly overlay guys actually work for Red Hat, and to figure out why we can't mount an overlay file system is what we call rootless mode inside of username space. And as of the 4.11 kernel, you will be able to do that. And so we'll be demonstrating and talking about how we get things in the kernel to, again, as you just said, to support new security features or new features of the operating system. Now we're going to continue, give you another idea. One of the issues we have with rootless containers is using NFS as a home directory. This comes out with enterprise customers more than non enterprise customers. So because of that, we have to add lots and lots of support and looking at different ways. How can we get NFS home directories to work in our environment? I don't want to get too deep into it, but basically, when you're running a container in your home directory, you actually get allocated multiple UIDs to run your processes. So just imagine you're doing a pull of an image. So when I pull down an image, that image is going to have multiple UIDs inside of it. So you're going to have a UID of, you know, most of our files say a row by root inside of your image, but say you have one that's owned by the patchy user, say the patchy user 60. So when I'm pulling down an image into my home directory, what's happening is I have a process running as my user Dan Walsh by the inside of user namespace. So it looks like UID zero. And when it's pulling down the zero files, it's writing them to disk and, you know, everything's fine. You know, it's just basically writing out my UID to disk, but now it gets that pulls down the Apache file and it pulls it down. And the next thing it does inside of the tower, you know, expanding top thing is to chone it to UID 60 inside of user namespace. So that sounds all good. And that all works really well on a Linux operating system with user namespace because you're allocated a group of UIDs to use the namespace. Now let's look at this from the NFS service side. So you're doing this inside your home directory. And what the NFS service size is sees my UID, say my UID is 1000. It says 1000, writing 1000 files. And all of a sudden it downloads a file instead of writing it to 1000. It writes it as 100,000 and 60. And the NFS server basically comes on and says, no, that's not allowed. UID is not able to chone to non-UIDs because NFS doesn't know about user namespace. And so therefore it denies that everything blows up, which causes anybody that's trying to run our containers. You know, podman and builder and all the container tools as rootless won't work if you have an NFS home directory. And it turns out lots of enterprise customers still use NFS home directories. So we have an issue. So one of the things we're looking at going forward is how do we get, you know, continue to potentially use the fuse overlay file system and our fuse overlay file system sort of fake out that it's writing out UID 100,000 and 60, but basically use an extended attribute to write out a file. And so that's the type of stuff we're investigating and working on all the time to solve these sort of tricky issues that, you know, don't really exist when you're running rootful containers. But it's, you know, how do we continue to improve our security by allowing more and more workloads to write in rootless mode? So it's, it's actually really interesting that you bring up that example, because I suspect that we actually ran into that exact problem on the show, because I use NFS mounted projects directories from like in my home network. And I discovered that if I tried to run a container with control or with colon Z, if that if the where the files were for that container were NFS mounted into my computer, it wouldn't work. And I wouldn't be able to have the right kind of right access from inside the container. But if I dropped the colon Z, it did work, right? Because I think it was just passing it through. But if I did the reverse and it ran it all local, if I needed the colon Z, so it was really interesting about like how it switched depending on that kind of writing. It's funny because, you know, NFS has been around since, you know, since I was a young pup. So it's been around since I don't know the 70s or something. And, or 80s at least. And NFS moves glacially slow as far as adopting new features. And in a lot of that's just because, you know, there's a huge install base and trying to get these things updated. This is incredibly slow. So we're talking about like AC Linux labels, we worked to get AC Linux labels to work on NFS for, I know it took six or seven years. And even to this day, as you see NFS, you know, basically NFS is denying you the ability to trade those labels. And that's all because you're not using a recent enough NFS server. You probably need NFS version four dot one on both the client and the server. And they have to be set up in such a way to get access to work to get the AC Linux labels to work across it. So this NFS is a constant pain. And even as we, you know, as I talked about the new features that we're trying to get into NFS, because it's a client server operation, we have to get both the client and server. And in the enterprise worlds, the servers are usually owned by companies like NetApp and other big companies. And so we have to figure out when they're going to have a more recent enough Linux kernel that we can start to take advantage of this stuff. But we get some of that static in RHEL, as I recall, as well. You know, what the enterprise software doesn't like to do is change, you know, with, you know, usually good reason. Chris, nice to see you. Narendra says that NFS is vintage. But not in the cool shirt sort of way. Right, right, right. In the wine, you shouldn't have made vintage kind of way. NFS is, you know, I would, we're picking on NFS now, but I would think if you were doing this with Lester or Zeps or any of that network-based file systems, you can have simple issues to, you know, yeah. Because again, username space is sort of a new feature. Even when we look at this being five or six years old, it's just the servers don't understand that all this user has actually got allocation of all these UIDs and that communications is not going back and forth. Well, I mean, yeah, and it's like, it's fundamentally wrong, right? I mean, like, you know, like a user should have a user ID, not a set of them, hence user ID. So, you know, I could see why it would take a while for, you know, kind of the rest of the software to kind of respond to that, you know, like, that's a pretty, you know, conceptual change to your, you know, your software. Did I see some other questions go by or? Not really questions, just kind of statements. Okay, that's cool. Kind of digging through. If you did ask your question and I missed it, please just re-ask it and I will get it caught, but I don't see any necessary questions. Just more comments on NFS v3 versus v4. Right, right. SCD needs faster disks than what NFS can really provide. So, I dropped a link to that SCD FIO article from IBM that really kind of saved my butt when I was standing on my cluster here at home. So, yeah. So, I mean, I think long story short though, is that if this conversation is interesting and you want to kind of really delve into it, you should definitely check out the container plumber's day, right? I mean, like, this is the kind of conversation that's taking place there. And it would be really, you know, I am now more convinced to want to go because I do like some of this stuff. Kind of talking about, let's back up a little bit, you know, kind of moving a little bit higher up the stack. My biggest question about Podman v3 is what made it three? Like, what is it that triggered going from two to three versus just another minor revision? So, one of the things we're trying to follow is what's semantic version of. And with semantic versioning, the way semantic versioning is supposed to work, this is a goal laying thing. So, if you think about your version of a product, you have an X, a Y and a Z. And with semantic versioning, if the Y and Z change, then you theoretically just, everybody should be able to upgrade and there's no breakage. So, existing tooling should always work. If the X changes, that means that, you know, there's a breaking change, right? That something we're going to break something in previous versions. My way of example, not PHP, for example, which does breaking changes on the Y version just for entertainment value. Yes. So, we had a, when Podman was first started out, we decided the first remote client, the remote capability in Podman was, we used a tool called Violink. And Violink was the mechanism for us to be able to start sort of remote containers. We have Podman on a Mac and it could talk Violink to a Linux server and it would talk to on demand Podman instance on there and launch it. And so, we were sort of the cutting edge with Violink and but we ended up having a lot of issues with Violink. And simultaneously with this, a lot of people were trying to push Podman to, you know, Podman was originally announced as a replacement for Docker at the CLI level, the command line. And a lot of people said, well, we need Podman to support the ability to do Docker type stuff, basically implement the Docker socket. So, we looked a year ago at our, a little over a year ago at our implementation and we decided that rather having two remote APIs that we would move to what we call the Docker API or we call it API B2. And basically, in compatibility mode, we met to the Docker API so that you can actually use the Docker client and talk to Podman and Podman bunch containers for a while. But now we can support things like Docker PY, the Python bindings, and we had the big announcement for this past week is that we can now support Docker Compose, which talks to the remotely of the socket. Other things we can cover now, lots and lots of people built into the CI-CD systems, you know, basically tools that talk to the Docker socket. And now we can just put Podman in there listening to the socket and implement it. So, since we wanted to go to one remote API, we were dropping Violink as support. So, one of the main drivers for going to 3.0 versus 2.0 was that we were going to drop Violink. So, that was going to be a breaking change for anybody that was relying on the Violink API and moved to this. But we added untold amounts of new features and the main, probably the biggest announcement is Compose. And we've gotten a ton of people that are interested in, you know, a shocking number of people who love the Compose language for running containers locally, but also, again, having support for Docker PY. We also have sort of all the extended features in our extended remote API. So, you can do things like create pods and do all of the fancy close to the Kubernetes type stuff with Podman using the traditional API. So, we have, you know, really, you know, that's probably the biggest features in Podman. We have much greater network support now, so that you can do rootless networking, you can set up additional network branches, you can do MACD lands, you can do pretty much, we consider ourselves as close to compatibility with everything you can do with Docker right now at this point. So, one of our goals was to help our customers, Red Hat customers move from REL7 to REL8 and in REL7 we supported Docker, REL8, we don't support Docker, so we need to have features. So, Podman 3.0, which will be released in REL8.4 has all the features to allow our customers to easily move their existing bases into REL8 without having Docker available. And that's sort of our goal. So, right now, we're saying we're, you know, when I say we're compatible, sometimes I get in trouble for this because there's certain things of Docker that we'll never implement like Docker Swarm. Oh, right. Yeah, of course. And there's features that also they've, Docker, Docker has this thing called link, where they allow you to link multiple containers together. Well, they've deprecated that, so we didn't implement it, you know, so, right. So, there's things like that. And our goal now is to take it beyond, you know, take it way beyond what Docker had the capabilities to do. And we've already driven that through our support for originally C Groups, C Groups v2. Speaking of DevConf two years ago with DevConf, I announced that Fedora was going to move to C Groups v2, even though Docker will run on it. And that caused a little bit of a stink in the world. And I'm happy to pronounce that two years later, you know, pretty much the whole world is moving to C Groups v2. Docker who works on C Groups v2 now. And Kubernetes, I think, either in the summer release or the fall release is going to finally support C Groups v2. And it was all driven by pushing Fedora to turn it on by the fall. Well, this is, you know, we have Fedora Fs, right? I mean, this is the problem with the first one. But it was, I think Fedora did, you know, it was awesome that we were able to finally move because we were in a chicken and the egg situation forever where the kernel guys wanted C Groups v2 and the container world was stuck in C Groups v1. And, you know, it needed a nudge or a shove off a cliff or whatever to finally force us through. And Podman was the solution to this because Podman worked instantaneously with C Groups v2 and we gave people a way to move forward. And Red Hat, a lot of engineers in my extended team worked on making Docker run with C Groups v2 and obviously Kubernetes, you know, that's sort of where we also concentrate because we had to get them to C Groups v2. And so those are all the great features that have come in Podman 3.0. Yeah, in a few weeks, I can't remember the date off top of my head. But in a few weeks, we're actually going to have Brent Boudi on the show to talk about it kind of some more in detail as well. I think I'm going to try to get Ravashi Mahana. I can never say her last name. Yeah. But who is one of the co-chairs of DevConf US, but is also a contributor to Podman. So we think she may also be on the show. I have to finish twisting her arm to be on the show with Brent, but we'll see. So let's see. What I did want to talk about, you know, kind of, we've been talking a lot about kind of the general world of containers. But I'm kind of curious about like, what do you see as like, what is OpenShift and Kubernetes in a sense? What is its relationship to kind of security? And when I mean that, like, how, where does that security come from? Is it coming from Linux? Is it coming from the containers? Is it coming from Run-C? Is it coming from something else? Is it a combination of all of them? Like, how does that stuff kind of work? Okay. So security comes, especially in something like OpenShift and Kubernetes, it comes at different levels. And obviously, things like role-based asset control, who's allowed to launch what kind of containers and what kind of systems. And that's all stuff that's way above my head. So I don't deal with that level, because that's sort of APIs and stuff like that. I deal at the operating system level. So I'm actually looking at how do we control what containers can do, you know, what the processes that are in containers can do at the operating system level. And what we try to do is take advantage of as many features of the operating system as possible to guarantee the separation, right, that we don't have a privileged escalation and the ability to break out. And these separations are controlled in lots and lots of different features. And I tend to look at sort of breaking the operating system into different features of the operating system. Most people understand there are routes able to do stuff and non-routes not able to do stuff. So OpenShift out of the box runs all containers as non-route, right? And you have to set special flags to run containers as non-route. So that's sort of running, relying on sort of the traditional mini computer design from Unix that originally developed, you know, so time-sharing systems. But basically that's proven to be a very good security mechanism that you run each one of the containers with different UIDs. And then you get a decent amount of separation. But sometimes you have to run with some route capabilities. And route also has the ability for, basically, route was divided into, I think, called Linux capabilities many, many years ago. And sometimes we'll give you a little bit of route, but we won't give you full route. And so that the amount of route that you're running containers can get, or amounts of capabilities, Linux and Docker actually sort of standardized on way back when about 14 different capabilities. And so when we looked at this, looked at those capabilities, we said, well, let's examine each one of those. And as we've gone forward running, you know, I call it containers in production, you know, Kubernetes world is like four or five of those that I just don't believe you need. And so we've trimmed those off. So we give you a little less route than you traditionally did. So another way of looking at containers, if I'm in a container, and I want to, I'm a bad guy inside of a container, one of the things I want to do is I want to track the operating system, right? I want to attack the kernel. And so there's mechanisms of attacking the kernel or through file systems. So kernel file systems, things like slash this and slash proc and stuff like that. And so what one of the things we do in that environment is we basically take those file systems, we mount them as, as read only. And so, you know, process inside of a container need to be able to work with read only work need to be able to work with slash this and slash proc, but they don't need to write to them. So and then with the the limited power of root, even though you might be rude inside of your container, you know, we take away the ability to mount and unmount. And so, so just those mechanisms, the other thing is certain parts of the file system were never intended to be hidden from users. But what we can do is we can mount dev null over certain types of files and the slash system slash proc and we can also mount empty directories over some directory structures that most processes don't need. Second, another thing we do is is controlling which devices. So another way to attack the operating system is through, you know, current devices. And so we we eliminate the optimum devices that your container can see. So you don't see dev sd zero and stuff like that. Lastly, another way you can attack the operating system is through syscalls. And we take advantage of a thing called set cop to instead of, you know, x86 64 machine, you this about seven or 800 syscalls available to you. And what we do with set cop rules is we can actually cut down the amount of syscalls that are allowed to your processes down to say 300 syscalls. And so now if there's a bug in a Linux kernel that involves a syscall that isn't available, your set cop table, then you can use that to take advantage of it. The last one that obviously I've been involved with forever is another way I want to attack your operating system is just to break out into the file system. So inside of a container, you sort of get a small snapshot of a file system, group of files in the operating system, you don't get the full view of all the files in the operating system. So but if I'm able to break out of my container, maybe I could get out to the parts of the operating system to be able to read. So if you think of just a time sharing system, right, how often do people have world readable files in their home directory? And if I have two users running, I might be able to read that file in your directory, your home directory might have your dot ssh directory set up too loosely. And if you think about that, there's lots and lots of files all over the operating system that probably don't tend to be written, read by other entities, and you especially don't expect them to be written to. So how do you protect the file system if I'm able to break out the container and get out to the file system? And the best tool in the world to do that is, is SC Linux. So SC Linux is all really about controlling access to the file system. There's lots of other features of SC Linux, but in the container world, what we really take advantage of is, is its ability to protect all the rest of the operating system file systems. So think about that years ago, I did the coloring book where I talk about the cats and the dogs and you know, SC Linux is you define a cat type in a prop dog type, and then you define cat food and dog food. And we write to Linux kernel that a cat can eat cat food and a dog can eat dog food. But, and everything else is denied. So if the dog tries to eat the cat food, it's blocked. So if you think about it from a container world, what we do is we define all the containers as being a container type, and we define all the content instead of containers being a container file type. And we say containers can read and write container file types. Now if a container breaks out and tries to read that's a shadow or tries to break out and read random stuff and other directories on the system is says no, you're only allowed, you're a cat, you're only allowed to eat cat food. And so that's one part of SC Linux protects the operating system, you know, the entire file system. The second part of SC Linux is that not only can we say you're a dog, but can actually need the dog. So I can say you're your dog and your phyto and then this is the dog food phyto. And, and then we could block, you know, say another dog named spot from BNLD phyto's dog food. And that's and this is my quick analogy to MCS labeling is so when we launch containers, we actually pick a random MCS label basically it's a little indicator that names the container. And in SC Linux, we can label the content similarly. And now we say if a container breaks out, it can't read other containers data because again, this is the dog spot and only read the container spots content. And so fundamentally, there's no other tool in the world that at the operating system level that protects the file system. And that's where SC Linux rules. Now, as we get further into security, one of the things years ago, I did another coloring book where I talked about, let me actually interrupt you for a second, Chris, can you can you pop the ebook box command? Because the coloring books you're talking about are on that. Oh, that's right. Go check out the coloring books. But sorry, so the other coloring book. So the other coloring book talked about it's actually based on the three pigs. And the whole idea there is you have your applications and the pigs are the analogy. And you're basically choosing where the pigs should live. And we talk about the difference that pigs should live. The most secure is going to be on physical different machines. And then you have them living in virtual machines and virtual machines is like living in a duplex. And then when you get to containers, it's like living in an apartment building or a hotel. And that is a single point of failure. The front desk of the maintenance man, he has the keys to all the apartments. So you have to protect the containers world. That's the kernel. And we talk about those. But one of the things that we talk about there is, you know, there is a big effort to take advantage of virtual machines for separation. And so Cotter, which was originally led by Intel, Cotter containers are becoming more and more popular. And they're starting to be available inside of Kubernetes and OpenShift where we can actually take advantage of KVM for separating back containers. And with KVM separation, it adds the additional feature of instead of getting sort of access to the full kernel now, you only get a small slice, right? So you're only able to talk to the KVM devices of the kernel and you're running your own kernel. So it gives you this really nice separation. And this is sort of like we've been relying on security for the last 15 years and virtual machines and talking KVM. So we start to run containers, not full operating systems in this, but just individual containers inside of these KVM slices to give us protection. But one of the intro we talked before this started, one of the interesting things that happens in any type of container is you usually want to be able to write to something, but you want to write to storage of the operating system. And so this is what's in Docker calling the term volumes, right? So this is volume mounting in. So I'm running a database, I might have an image, I'm saying MariaDB, that is the MariaDB executable and all the stuff used to support that. But I actually don't want the database in the container, I want the database stored on local storage. So what I do is I volume mount in into that in storage. In this case, we talked earlier about overlay, right? I don't want to have a copy, I'm right file system for my database. I want to have a great best file system I can. So when I volume mount in content into a container, this is where all of a sudden the security becomes a concern. Right. It's going to impact the performance of the database itself, right? But it also, now, sort of when I talked about the SE Linux and the cats and the dogs, I talked about everything inside of the container is cat food. But now I've taken something from the operating system and stuck in the container and that is not labeled as cat food. So I need a way to basically relabel that content to be the content that this container can use. And in the SE Linux case that's using, there's a special colon Z and colon capital Z that I can put on a volume mount to cause the container wrench and docker or podman to go out and relabel the content. So now it becomes cat food content, right? Right. Right. The containers content. So that's where SE Linux works great in containers until you start volume mounting in and then you have to start to understand that you're taking some random part of the operating system saying this is going to be used inside the container. Well, when we move to virtual machines, just like that, right? The virtual machine container separation, how do I handle volumes, right? And in the case of, you know, when you're running in a virtual machine environment, you have a different kernel than the host kernel. So I can't just use standard bind mounts to take a volume in like we do in traditional containers. And we can't really do it in KBM anyway. Right. So KBM causes issues around that and it basically ends up being a network-based file system. So I get FS or this I think called plan nine file system. There's all these different files. There's all these different file sizes to do that. And so the kernel file system team at Red Hat started working on I think called Vert IOFS because most of these network-based file systems are thinking that you're on a separate network. And 30 years old at a minimum. Right. So Vert IOFS is basically realizing that this is all happening inside the same kernel, even though it's a virtual kernel, how can we take advantage of the type communications path there? And so Vert IOFS is a really cool feature for using things like how to containers and taking volumes off a disk and putting into it. It's going to be a talk at containers plumbing on Vert IOFS. It's really kind of cool to look at that. But as soon as so dropping back to SE Linux again, as soon as I take that file system, right, so I have Vert IOFS which is running a little demon that fuses demon on the host. And that's communicating with the kernel inside of the container, the KBM separated container and doing back and forth communications. So we basically opened up this pipe for processed inside of the container to communicate indirectly with Vert IOFS demon running on the host, running his root most likely on the host. And all of a sudden we've opened up a security hole, right, or a potential security hole. And you know, even though KBM separation. So it's sort of like we go back to my duplex analogy earlier, I'm in a virtual machine and the only thing I have is a shared wall of the, you know, that's KBM separate into, but now I've poked a huge hole through the wall. Right, right. Yeah, so two containers to potentially talk to each other. So what do we do with Vert IOFS? We wrap it with SE Linux basically say, now, you know, we're protecting the, again, we have to protect the file system from potential escape. We take advantage of other security fee, all the other security features. But, you know, anytime we, even though we have KVM separation ends up being, you end up having to pull walls through, you know, the better separation. So we end up having to take advantage more and more of the operating system. So that's all features that we're taking advantage of right now in an open shift in containers and, and, and stuff like that. And that's all different parts of the operating system I deal with and work with. One of the big features that we're adding to open shifts going forward and we're working hard to get into Kubernetes is, is taking advantage of user namespace. So user namespace, again, it's, it's key to Podman. Podman is great because you use a namespace and do all these really features, really cool things with, you know, running containers with different groups of UIDs. And so they're separated again using the traditional UIDs. And when we have a container that requires multiple UIDs or requires something like root inside of a container, username space, because it's the ability to do that in the most secure way. Well, username space has never made it into Kubernetes. Okay. Oh, I don't think I need that. Yeah, okay. Yeah. So it's just not supported inside of Kubernetes because Kubernetes says, you know, how do you, how do I translate from the Kubernetes YAML file down to the runtime that I want to run in a separate username space. Right. And so there's lots of effort going on. We've actually implemented some advanced features in cryo now, container runtime for Kubernetes to support username space, but the Kubernetes isn't consuming it yet. Yeah. Well, it can, it can do it. We can do it like a side door. So we can do something called attributes. So we can say that special attributes will say, I want this thing, but Kubernetes doesn't know what's happening. It's just cryo underneath the covers is doing it. So we're, we're going through the standards process and go through the Kubernetes teams to get username space to be a full first class citizen in the world of that. And I think that'll be out by the fall. I think we'll have full username space features. So we're going to have username space, SEO Linux, KVM, separated containers, set comp rules, you know, so this, but the main thing here is there's used to be a conference I go to every year called defense in depth, the federal team, and military, they're always looking at, you don't have just one wall, you have multiple walls and you have lines and you have different types of stuff. So in container security or any type of security, you want to have as many levels of defense as possible so that the bad guy might get through the first line, but he can't get through the second line. Right. Right. So that's, that's, you know, that's why when we talk about containers, it isn't just one thing. It's, it's, it's lots and lots of different. So one thing that occurred to me while you were talking, which I just kind of wanted to ask you about and see if it was accurate or not is when you talk about root full or rootless containers, in a sense, like root almost doesn't really exist in Linux anyway, or any more kind of as a, right, because it's really just a set of capabilities that are granted to a particular user, right. Is, so the concern or at least is it really what we're doing here is that why a root full container is a problem if you could just take those capabilities away is because you're going from I have a high level of power and then taking away abilities versus having a low level of power and adding abilities. And so when you do the root full container, you're, you're extracting privilege, which you may make a mistake on, whereas if you're taking the rootless container, you're adding privileges and the mistake will only result in that process suffering, not the system being compromised. Is that, is that a good understanding of kind of why root folder is better? Yes, that's, that's a great way of looking at this is it's always harder to, you know, as you Linux, well, they used to, you know, people wanted to have an admin and they used to say, I want an admin that can, cannot do this, right. I said, no, no, no, you can't define something you can't do what you have to define is what he can do. And, and if you start with the assumption, you know, this is a user that can send a ping message, then that's easy for me. If you say this is a user that all he can do everything in the system, but can't send a ping message, I can't make that happen, right. Because something else that he's able to do can turn off the security that prevents him sending the ping message. Now, so yeah, that's a good way to think about it, but I want to go back to your first assumption, you talked about rootful versus rootless. Now, when you talk about Linux capabilities, this is a kernel concept, right. So the kernel is divided out the power of root. These are, you know, treating root like it's a non root user, but there's a couple of other things to realize about a root running process, root running process running UID with no capabilities at all on the system can still read and write all files owned by root. Right, right. So, so, so, so fundamentally, you don't want it to be root for that reason, right. So, you know, I might be able to, you know, modify the Etsy password file, because I'm able to write the Etsy password file without requiring any additional capabilities. Secondarily to that is the kernel has basically said that, you know, I'm lowering the power of root. The problem is anytime you do a process-to-process communication in user space, user space has made huge assumptions about a process on the other end of a Unix domain socket is being owned by UID zero, right. They're not looking, if you look at even system D, it is not looking at what capabilities the process that's connecting you to it is running at, it just sees the UID equals zero and says, oh, you're root. So I'm going to allow you to do stuff. And, you know, even if that, even if that root had all its capability, the only thing it was allowed to do is do ping. So, fundamentally, getting rid of, you know, UID zero has merged itself into the operating system, even though it's not really much different, right. It's so embedded to the assumptions of your operating system that getting your processes not to be able to be run as UID zero is credible even without even if you eliminate all the capabilities. Yeah. So, I suspected that. I mean, because, you know, like I see it all the time, right. I mean, even if you're writing a batch script that you want to run as root and you want to test for it, you know, you basically look for UID zero. You don't, there's no, you know, there's no like easy batch command that says, is this person root? So, you know, and years ago, even years ago, there was efforts to allow a process to say, on the other end of the connection, what capabilities does he have? So it would be really nice if like, when you connect to system D to do some, you know, start up a service or whatever. If it was able to come to you and say, do you have any, do you have this admin capability? The problem is that that's racy. And the kernel developers, so there's no way to, inside of the single communications, be able to figure out whether or not which capabilities are available to a process on the system. And so every effort in Linux has been torn down by Linus Tobalus, because he doesn't like, he's not going to allow any racy security assumptions to be built into the other races. So fundamentally blocked from ever getting this feature. But I literally just filed a bug against system D the other day, saying it'd be really nice if I was not privileged user, if I did system control, you know, status, some service name, it would assume dash dash user. Because I'm not root, right? So why would I be asking about anything else? And yeah. And so it's kind of funny, because yeah, we fundamentally don't really have that understanding, you know, kind of in Linux, it's like, you're either zero or you're not, you know, which, you know, that's the problem with, you know, we could just go write a new operating system, and then we could solve, you know, this and so many other problems. If we had the way back, if we had the way back machine, we right, right? Yeah, when Unix was created, when Linux was created, right? Exactly. Exactly. Chris, we kind of were talking for a long time there was there. Yeah, we've got some questions lined up. Okay. Yeah. So is there any major difference? Any major difference between the tool kind of Kanako developed by Google and Podman? And I think it's more of a Kanako build a discussion there. Kanako is more of a building container images. Right. And the real question that would have been Builder versus Kanako is probably the correct thing. So Podman has Podman build and Podman build under the covers is basically build a code sucked into and into Podman. So most of the time when people report bugs on Podman build, we always redirect them to Builder and Builder So Builder is a tool. So if I'm going to compare Builder to Kanako, Builder or Kanako do somewhat similar things, especially when it comes to handling of Docker files. And Builder's term now we have, we also support container file because I want to get the word that the DOCKER adjective out of the world as much as possible. But basically, so the format that Docker files sort of standardized, there's lots of tools out there to be able to build container images, those CI images out of them. And Kanako's one, Builder is one, this is one from OpenSuzi. There's lots and lots of tools to build up to do that. What fundamentally we wanted to do with Builder, so we wanted that support obviously because that's the vast majority, but we also wanted to experiment with Builder to be able to create images from scratch without requiring you to build the Docker file. The second everything I wanted to do is be able to take advantage of the operating system. So if you've ever seen me give a talk on Builder, and I talk about you know, you create a directory, you copy some content into it, you create an image and push it off to a registry. There's no reason to have a Docker file to do that if you want. And then so Builder has all of the sort of low level intrinsic commands to create a container image without having to go through a Docker file. So you could use bash, scripting, or you could use higher level tools. And so Builder. Because you want to bring Rocket back. Whatever. But I mean, the idea is to be able to build using fun amount of the little tools. So some ways Builder is sort of like a library. And we've entered into Podman as a library. And then we also it's also gone into OpenShift. So any of the OpenShift source damage Builders and things like that are using Builder on the covers. It's been pulled into Ansible. And lots and lots of people use Builder as a low level tool. But probably the vast majority are still using it as a mechanism for building Docker file, which Kenneco is doing the same thing. And bottom line in all these tools, like, you know, I mean, the other nice feature about Builder is that it integrates with Podman, integrates with cryo. And, you know, we really can sharing all the same content, whereas Kenneco is sort of a standalone tool that doesn't work well with other tools. It just has the ability to build images and push it. But I don't really care. And, you know, if you want to, as long as you're building OCI images, then all these tools can work together. And sometimes this is sort of like saying, you know, when should I use, you know, I think versus using SCP or like, there's lots of tools to do the exact same thing. It's just each one of these tools is sort of developed for slightly different use cases. And, and they all can come together as long as you know, the files of the disk or the OCI images of the shared content that we could all work together and sort of experiment and build a new tool to do this stuff. Gotcha. Were there, was there another question? There's another question. Yeah, yeah, yeah. Is why did Podman decide not to support for our link? So, so it was too, too thick. Violink didn't have, Violink hid a lot of stuff from our engineers and, and we'd be randomly getting random breakages and we couldn't understand the low level functions that were going there. The other thing is Violink didn't seem to have great upstream support at the time. And lastly, we, you know, so probably the biggest one is we didn't want to have two ways of doing this. And most people were demanding that we did, you know, an API based on the Docker socket API. And so, you know, it was just like, it just makes it at a certain point. It paid more sense to us to just move to, to, you know, rest, restful API for our remote communication than embedding another tool in the middle. So we could just use standard HTTP communications like Docker designed for the Docker socket. Nice. So there is literally now a socket as well. So you can set up a Podman service with basically it's a socket. Again, we don't have a demon, right? Right. Yeah, we have a system D will the system be serviced that when you connect to a certain socket will activate a podman instance and that podman instance will service, you know, basically launch a container or give you a list of containers or whatever. But basically each one of these communications causes a podman instance to be fired up and then communicate back over the channel saying, here's my containers, I created you a container, I killed you a container, stopped you a container. I don't, yeah, for some reason, I don't think I ever realized that the socket communication that Docker was doing was like HTTP kind of rest, you know, socket communication. I always assumed it was like a loopback kind of binary socket, you know, that was, yeah, that's interesting. Okay. So was there another question or anything, Chris? One just came in. Actually, can you use Docker composed with podman rootless since podman can do the socket? So this is a, this is a question you can ask Brent when he comes on in a couple of weeks. There you go. The answer to that is, is not yet. So again, we were rushing to get the thing out. So we got the full support for rootful and no one's ever looked at running compose rootless because it was always talking to the Docker socket. But since we are the champions of the rootless container, we will make it work. The biggest issue we have right now is, is composed as a lot of network configuration. When it starts up, so most people don't ever configure and do stuff with a network or compose does a lot of stuff with configuring the network when it starts up to get to containers to communicate and stuff like that. And a lot of that code doesn't work well with rootless networking at this point. So that that's just something we're going to fix. And I would figure that would be in fixed and say podman 3.1 or podman 3.2. So I had a question about that. Ask Brent that one and I'll be well to find out what he's thinking about that. So he did most of the work on compose. So one of the things that I actually think was really cool and didn't realize about podman, you know, before I actually started doing the show was that basically being able to describe, you know, a pod, but you know, like a Kubernetes pod as YAML and kind of, you know, essentially do the equivalent of podman compose, but using YAML that would also be translatable to Kubernetes or OpenShift. Is there a consideration around how like what I thought was kind of cool about that is I could on my local machine, then I could set up, you know, my little database, my little web server, whatever and kind of set it up all correctly to eventually be deployed in Kubernetes with the same syntax. Going to Docker compose syntax to kind of semi-accomplish the same goal means I lose that kind of translatableness to Kubernetes. Has the team been thinking at all about how, what does my migration look like from, okay, now I'm tinking around with my Docker compose version of the thing, but now I want to go put it in OpenShift. Do I have to go rewrite it now in YAML in this new language I don't know anything about or is there a way I can convert or is that something you thought about yet? Well, take a step back. So I've been dealing with Docker since Docker started back in 2013, 2012. And I know how to run containers very well at the host level. And I've always been trying to push off going to Kubernetes because I like to live in my little single node where I'm an expert at. And but one of the things when I've gone to the Kubernetes world, it's always like, oh, God, I got this YAML file, which basically I go out somewhere and I download someone else's pre-prepared YAML file. And now I go stop mucking around with it and trying and failing miserably trying to figure out how to write a container to run Kubernetes. So one of the fundamental things we did with Podman, we said, let's sort of build the best practices to run our containers into it. And we have a thing called Podman Generate Kube. And what that does is takes your running containers and running pods on your system and takes what those are and translates them into a Kubernetes YAML file. Oh, we did a show about it. In fact, yeah. So it's a kind of cool feature. And the secondary thing we also said, you know, sometimes debugging things in Kubernetes is difficult. So I might want to take my Kubernetes YAML and just run a Podman Play Kube, which will take the Kubernetes YAML and run it. That's basically what you talked about here. And one of the goals with Podman is to make the roadmap onto OpenShift and Kubernetes easier, right, to allow you to transfer from your sort of Docker knowledge to your Kubernetes knowledge. So reasons interesting, the question you asked about Compose is actually Brent, who's coming up to give a talk with you guys, is also giving a talk at the containers plumbing called Going from Compose to Kubernetes. And what he does is he uses Compose to generate containers running onto his system, and then he uses Podman to generate food from that. And then he takes Kubernetes and runs inside YAML. So that's the whole workflow that he's been looking at. But the thing is we can't ignore, you know, Compose probably has, I don't know if it's, but it's huge. It's a huge user base that relies on Compose and tons and tons of big companies rely on Compose. And as you said, it's very difficult to go from a Compose YAML to a Kubernetes YAML, but using Podman in the middle, it actually makes it easier to at least get to, you know, so sort of like, I've never written a piece of code from scratch in my life. It's always a great copy and paste. And so, you know, at least with the Kubernetes play, you know, generate Kube, Podman generate Kube, I get to the point where I have a Kubernetes YAML that's, you know, that I can sort of map back to what I understand here. And now I can stop looking at deployments. So the advanced features of Kubernetes from that point on. So that, yeah, so that's, that's our goal. And our goal, if a person comes up to us and says, you know, we're going to run multiple containers on a single host, I would say write it in Kubernetes YAML to start. You never know that when you're writing the code to begin with though. But I mean, your goal, the goal is to get, you know, our goal is to get, you know, your application to run on thousands of nodes and have it orchestrated around Kubernetes. So your end goal is that, you know, but you might start out with just wanting to run a couple, you know, couple instances of your application on a single node, then, then you're writing a Kubernetes YAML and just launch it with Podman. So just kind of on the note of never writing your own code from scratch, I have to plug one of my favorite pieces of software of all time, which is a little Python script called how do I, which on the command line will actually search stack stack overflow and all that stuff for how do you do this thing that you can you can cut and paste from. I will throw that in the chat because I like to plug it because I think it's awesome. Let's pause there. I'm actually old enough to really remember programming before you had search engines and we would have you'd have racks of books on the wall and you'd be pulling it down and then you would read from here and type it exactly what to be clear. I'm old enough to remember Yahoo launching. So, you know, and yeah, not having to guess websites anymore. So that was one of those ones where in retrospect you were like, Oh, this seems obvious. Why didn't it occur to anybody to make yellow pages of the internet? All right. So we're pretty much out of time. So I think we should try to hit the points. We try to do that. We're going to say our but Dan, if you need to jump because it's after 10 a.m. That's fine. Yeah. Appreciate you coming on and everything. Do you want to stick around? That's fine to hear about the internet depends on your schedule. No, I'll drop out now. Nice talking to you guys and thanks so much. Any additional questions or anything else? I'll be at DevCon this week and then the conference afterwards. So thanks man. You guys later. Yep. Thank you so much. Yeah. Appreciate it. All right. So I will just share the internet lines. First and sweet, sweet internet points. And here we go. So Narendra is still slightly on top with Netherland Takham right behind him. 4,300 and 4,200 points. That's ridiculously. Yeah. So much. I will point out Noah Friction made a big jump this time. I can't remember exactly what he had before, but he did make a big jump over this notch. And yes, Noah, I'm assuming it's a male. I apologize if I'm incorrect. But then we also have Joe Fuzz and Detective Conan Cudo and Bacon Fork still going strong, starting to catch up. We got to see some, get some more points. We have on the screen right now the way you can get points for this episode. And I'm throwing it in the chat right now. So definitely go out and go after those sweet, sweet internet points. And it's definitely neck and neck. Do remember there are other ways to earn points besides just watching the shows. You can file issues on the episodes with topic suggestions. You can do PRs. And now if you remember from last episode, I think I was talking about the OpenShift clients container, a PR against that would also be a PR in the level of our kind of vein. So our issues against that, stuff like that. So definitely let us know. I did call out, if my slides work, a couple of newish people. Trolling on chat does not gain points. There is too much self-benefit for the trolling that you are not. There's no external reward as well, I'm sorry to say. But we do have three new people that I saw. There may be others and I just didn't realize they were all new. But so thank you. And I hope you're back from last time. And I don't know, Walker. Triple Dub Walker maybe. And DJ Folo and Alex77G. Well, wait, seven would be Lee. Would be Elle. So no, still not pronounceable. So thanks so much for coming. I hope you are back this time. I hope you come and watch more shows. We have a pretty solid schedule, I think, out through May. We're doing credentials, not credentials, subscriptions and licensing, I think next time. Oh, actually, good note. We are dark next week because I will be in a weird time zone even though I'll be at home. So we're not going to do the show next week. But the week after that, we're going to, I think we're talking about subscriptions and, you know, license or whatever. We're bringing an expert on to talk about subscriptions. Yeah, exactly. And actually, we have a couple of experts and maybe some more because it takes a large number of experts. So we're going to talk about subscriptions next time. I think we're going to talk about Helm, the one after that. And then after that, we're going to have, that's when Brent is on. And that will be talking about Podman, V3 in particular, and maybe some deep dive stuff into the compose components. Then after that, we actually are doing a deep dive into UBI and what's involved in UBI. And then actually, right after that, we're going to be talking about Summit, and at least Summit, maybe KubeCon as well with Chris Wright, the Red Hat CTO. That's coming up in, let's see, what's the date for that? Looks like April 21st. So about a month from now. Or? No, two months from now. Oh my goodness. It's only February. It's only February. It's only February. See, when you start thinking it ahead, like I live in the future, right, when planning shows and everything, I forget which month it is sometimes. Yeah, we've, I've been talking to a bunch of people lately about, you know, like there's various kind of people involved in the Red Hat ecosystem, you know, so like partners, customers, casual users, you know, whatever. But a lot of them, they like feeling like insiders, you know, with kind of like insider knowledge and stuff like that. And I'm like, are they watching the OpenShift stream, you know, the OpenShift TV because that's where you get knowledge of the future. So definitely, thanks so much for coming. I hope you enjoy the show. I hope we weren't, you know, if, if that wasn't what you wanted to talk about with containers and security and that kind of stuff, let us know either on Discord or File and Issue or whatever and let us know what other aspects of security you'd like to talk about or hear about. And we can either invite Dan back or we could invite others to discuss those things. You know, we're, we're here for you. Yeah, indeed. And coming up next on the channel, at 11 a.m. Eastern, 1600, UTC is the OpenShift Administrator Office Hour with, we've got a guest, Christian Hernandez is going to come on and we're going to talk about Windows containers and getting those working in your environments because that has been a huge request of ours and Christian has been hard at work making it all work good. So yeah, yes. If, if you're interested in gaming and building games, co-worker, team lead, manager type, whatever you want to call them, Eric Jacobs is doing the scalable multiplayer game design with OpenShift. We've got C coder. Yeah, C coder or C++ even. The show is all about how to scale out like a multiplayer game. So just check it out. I always find it interesting, right? Like I actually tune in to watch it even though I have like zero involvement in it because it's just amazingly great. Cool. Yeah, thank you, Lange, for dropping that in there. Yeah, so then you just asked where they file an issue if they want to talk about like some future show or something. Yeah, if you file an issue on the episodes repo, I will be most likely to see it. You know, you can also, if we don't, for some reason, hassle us and discord. Yeah. So does Intar's just just recently failed to test molecule roles in Podman, which requires to set some SE Linux Booleans. I haven't seen that before. It happens. I mean, you know, like usually there's a, you have to set SE Linux Booleans sometimes. I certainly don't know how often. But it is, that is interesting. You know, feel free to bring that back for when Brent's on and maybe talk about it a little bit more. You know, but you know, let us know. Yeah. So yeah, let's wrap this thing up and send it home. Definitely check out the developer sandbox if you haven't used OpenShift 4 or if you have and you want to learn a little bit more, right? We've got free developer sandboxes we're handing out. It's in beta. So it's kind of resource constrained, but we have seen it stand up. What was it? We set up our sort of... I did the next cloud thing in there. Yeah, yeah, yeah. Let's put it on with it. The other thing to keep in mind, CRW are like, so the code ready workspaces is also in there. Yes. So you can do kind of, you know, like online IDE inside the same environment that you're actually going to deploy the thing that you're playing with or building directly into the same kind of environment. So it's kind of one big integrative environment of here is a text editor that looks like VS code and, you know, it will deploy an application directly into the containers running inside OpenShift. Yep. Yeah, it's pretty slick. That's awesome. So definitely check that out. And the amount of noise internally about trying to get more and more capability in there is very high. So expect more in the future, right? Like it just launched what last week. I mean, yeah. Yeah, pretty recent. Yeah. Yeah, so yeah. Regardless, check out the show later today. Subscribe to the calendar and we appreciate you tuning in and we want you to stay as safe as possible out there and... Drink more coffee. Drink more coffee because Lord knows I need it. I know, mine's gone. Sad. So sad. All right, folks. Thank you very much. Well, see you next time.