 This is a three o'clock talk on Sunday. Up next is a fresh look at SC Linux with Daniel Walsh. OK. OK, thanks for having me. I just wanted to, a lot of times, I'm going to go through SC Linux. I'm going to try to teach you. If you've had SC Linux taught you before, I often think it's taught it way too difficult to level. And so I'm going to try to go through it a much lower level. I'm going to use the coloring books. Sadly, I ran out of coloring books for everybody. But about a year, a year and a half ago, I heard about this bash exploit. And so I quickly wrote a blog, quickly did an investigation and see what SC Linux did about the bash exploit. And later that day, on the ride home, I was listening to NPR, and they came up and they talked about shell shock. And I said, what the hell is shell shock? I had no idea what it was. And after writing this article, so if you see this article, it's actually talking about shell shock, but I called it the bash exploit because whoever adds cool names to exploits had not reached me yet to tell me that they renamed it. So an interesting thing with shell shock. So shell shock instantaneously made every single web server in the universe that did bash, ran bash, or CGI scripts, was vulnerable. It could be totally taken over every single web server in the world. The only thing to stop that exploit was SC Linux. So the day it happened, any machine that was running with SC Linux in enforcing mode blocked it. How many people in this room were running the way? I'm not going to ask this. I hate to do the answers. But that's the power of SC Linux. SC Linux is not catching known problems, not necessarily catching known problems. It's catching the unknowns. So SC Linux puts everything into a box in control. So I'm going to talk about that a little bit later. I just wanted to point this out. We're going to talk a little bit about virtualization, multi-tenancy, and some of the stuff. So for those of you who don't know who I am, I used to call myself off in Mr. SC Linux. So I started at Red Hat 15 years ago. At the time, I have a security background. I work way back way on in Project Christina Kerberos. I worked on VPNs back when VPNs were just being created, firewalls. I work for Digital Equipment Corporation. Worked on the first firewall at the White House. WhiteHouse.com. WhiteHouse.gov. WhiteHouse.com was a porno site. OK? Yeah. I remember seeing Hillary and Bill on it one time. Anyway, so 15 years ago, NSI came to Red Hat. 15 years ago, I started working at Red Hat just before 9-11 to give you an idea of how long ago it was. And a couple months into my tour, about six or seven months into it, the NSI came to Red Hat and asked, this seats up front here if everybody wants to squeeze in. Came to Red Hat and said, we have this really cool new security mechanism that we really need to get mainstream. So we want to work with you guys to get it into the upstream kernel and everything else. People went around Red Hat and looked for anybody that had the word security in their resume. My name popped up and I became the SC Linux guy. So for the next 10, 11, 12 years, however it was, I worked on SC Linux. I also worked on a thing called namespaces. So in REL 5, we introduced a concept of namespaces, which was PAM Mount Namespace. The Mount Namespace was introduced for a thing called MLS. So SC Linux allowed us to get to MLS multi-level security. We're going to cover that a little later, but we actually introduced this concept of the Mount Namespace. Eventually, the namespaces grew into a bunch of other namespaces. I am now the head of container technology at Red Hat. So I have the Docker team underneath me. So we submit patches all the time to Docker, and I'm constantly fighting with them to get my patches accepted. But that's what I do now. I also did the Docker part of SC Linux. So let's get into it. But you're not here to hear my history. Let's go in and look at that. I call this topic. I actually combine two different talks here together. I'm not sure we run out of time. Are we the last talk of the day? Oh, shoot. OK. I was hoping to go over. I can go 30 minutes over. I can go 30 minutes over. They throw me out. OK, so SC Linux, when you get in SC Linux era, it's trying to tell you one of four things. And that's really what this talk's going to be covering. So you have something wrong with your labels. You change your file since it's fault and tell SC Linux about it. Applications or SC Linux has a bug, or you've been compromised. That's really the only thing it's going to tell you. The problem is trying to figure out which one of the four it is and how to react to it when you get one of those. So let's do it. So the first, the most common error for SC Linux is you have something wrong with your labels. So everyone please stand up. This is the interactive section. Everybody's ever seen me. They've had to do this before. I make everybody do this every single time. OK, you're going to read the screen in unison. This is the church of Dan Walsh. Ready? Yeah, you don't have to do that one. OK, that's it. Everybody knows what SC Linux is. OK, sit down. So SC Linux is a labeling system, OK? Simple as that. I don't want to talk about, we're going to cover some other deeper topics in it, but it's a labeling system. If you label some wrong, things are going to blow up. Anybody see the canonical talk yesterday keynote? OK, he put up a slide that I would love to steal, but he probably won't let me. And he put up the way the Linux operating systems evolved over the years, or actually UNIX has evolved since I was a young stuff. And then we have applications that look like amoebas. So you install them, and these applications can be configured 100 different ways. So you have the Apache website. How many different ways can you configure Apache? It's basically 1,000 millions of ways, right? And then you have databases. They can listen on network sockets. They can listen in local sockets. You can change the socket. So you can change all these configurations. And then we have to throw in other applications. MariaDB, slightly different than MySQL. So with this constant change in the operating system, and SC Linux wants to control all of that. So if you install and reconfigure anything in SC Linux, you have to fix the labels. OK, let me tell you about another labeling system that no one in this room probably thinks about. Discretionary access control. Does anybody even know what that term means, discretionary access control? OK, about half of you do. Now let me tell you what it is. Ownership, permission flags, read, write, execute. That's what's called discretionary access control. So every process on the system has what? An owner and a group. It actually has an owner. So there's an owner or a process. Every file system object has a label on it. The label is the ownership, the owner, the group, and the rewrite permission flags. So if you get permission denied anywhere on the system, the first thing you do is you go and fix the labeling system. It just happens to be the labeling system you are familiar with. So you're not going to go in and say, Apache got permission denied. Let me make sure the file is owned by the Apache owner. Let me make sure that the search path has proper rights the Apache owner can search through it. At which point, everything looks good. You run it again. It blows up. At which point, you say, SELinux sucks. And that's what you guys all do, because you don't think about fixing the labeling from an SELinux point of view. Tom Hayden. Everybody's gone to everybody's seen this before? This is actually a website, stopdissablingsselinux.com. It's done by Major Hayden. He's one of the top security guys that I'm at the blogging of the Denver's company, Bragg Space. So he put this up. He's a good guy. And he actually gave me the Seton Force one shirt. So here's the real world example. This actually happened. A guy in the next cube for me had an egg dispenser. What is this egg dispenser dispenser? M&M's. Someone put skittles in the M&M's dispenser. OK, talk about permission denied. That is, I hate skittles. And he wanted this slide going up. OK, so everybody get out there, SELinux textbook. You can start looking through it. This textbook, the SELinux coloring book, is probably the best document, I believe, ever written. So you can go out to Amazon on SELinux. You can go out to Amazon by those big thick textbooks, or you can read this. And you will have an understanding of SELinux. You can also, for those that are not in the room, it's available on the internet. One of the reasons people really like it is Maureen Duffy does her, she works for Red Hat, I'd say. Actually, all sorts of UI stuff. So you can get this and print it out. We've given out hundreds of thousands of these. I'm probably not hundreds of thousands of thousands. And actually, interesting story. Any Japanese in the room? OK, when we gave this out, Japanese Red Hat came to us and said, can you make it a comic book, because Japanese people don't know what coloring books are. Is that true? I said, that sounds like the stupidest thing I've ever heard of. Anyway, so I told them to color it in themselves and call it a comic book. OK, so SELinux is a labeling system. If the labels are wrong, SELinux will generate errors. Solution, picture, labels. I stopped myself. OK. Oop, that. So we're going to talk a little bit about the labels now. So we're going to go a little. This is where we're dropping down a level. And this is going to cover the coloring book. There's two types of enforcement that SELinux takes advantage of. The first one we're going to talk about is called type enforcement. And this is where we really define the type. You're in a patchy process, right? So if you look at the SELinux label, it's kind of ugly looking, but there's user, role, type, and level. And we're really talking about the third field after the colon. And this is the type field. But every process on the system has a type. When I log in to a system, most people log in at the ministry that run confined T. That's the unconfined type. You're able to do anything you want in the system. But if a process starts as a system process, like a patchy, that's going to run as the Apache type, which is HTTPD underscore T. When the shell shock happened, if the shell shock person logged into the system, he was running as HTTPD, he actually is a CGI script, which runs as HTTPD, this CGI script T. So when the CGI script was running bash, the way bash exploit worked is you were able to say something, you were able to pass something to the end of the bash command, or the end of the CGI script, that the bash command would then execute. So when SELinux R, the CGI script came up and said, hey, I want to look at user's home directories. And SELinux stepped in and said, no, you're a CGI script. You're only able to do CGI things. And said, hey, I want to try to run sudo. No, you're a CGI script. You're only able to do things with CGI scripts. Had nothing to do with permissions? Nothing. You've got a world readable files on the system. Anything, SELinux will block it. So there's the standard type for Apache web server. And here's the content type. An interesting thing about SELinux, that people that start to struggle with it often think is that if I have the type of the process, I want to assign it to the type of the file, right? So you don't assign process types to file types. So there's a separation. You have a process type, and you have data objects. So this is a type that Apache is able to write to. Most types are read-only. So most types that Apache is allowed to read to are read only. But let's take it back a step. So when we look at type enforcement, let's look at a simple example of a cat and a dog. So I have a type, and I say, this is a cat type. And then I say, this is a dog type. These are processes on your system. And then we have objects on the system. We have cat chow, and we have dog chow. So those are objects on the system. Then I write a rule to the policy engine. And this is exactly what the policy rules would look like. This policy rule says allow the cat process to eat cat chow food. So food is like file, directory, whatever. But basically, it's a category of the object. So basically, we write thousands of these rules. It says allow cat to eat cat chow. Allow dog to eat dog chow. Etsy Linux system, like all firewalls rules, is denied by default. If I don't have an allow rule on the system, you are not going to be allowed to do it. The process is not going to be allowed to do it. So the cat's able to eat the cat chow. The dog's able to eat the dog chow. The cat's able to ask the colonel for more cat chow. But when the dog tries to eat the cat chow, the colonel steps it. Any questions? That's what type enforcement is. It's that simple. It's type enforcement. We're going to get a little bit further into this. So Etsy Linux labels. Now we're going to cover. So that's the standard way we protect systems. Now I'm going to put on my other hat. So type enforcement works real well for use cases where you have an Apache server, and you have a Apache data, or you have a database serving, a database data, and you have home directory data. Where it doesn't work well is what I call multi-tenant systems. Multi-tenant systems are things like Docker, containers, VMs, OK? I'm going to show you OpenShift in a second. So there's lots of use cases where you're going to have hundreds of VMs running on your system, and you want them to vet the VM, and this is VM data. But what I want to do is I want to make sure VMA is not able to read VMB's data. Now I could create all these types, but all of a sudden I have thousands and thousands of types. So what we needed is the different types of security for the system. So this is back in Esper, probably around 2008 timeframe, when we invented the concept of what we called MCS separation R Esper, secure virtualization. Secure virtualization, by the way, I'll give you an example where Etsy Linux is ruled. There's been two breakouts at KVM on REL systems over the last six, seven, eight years. There's been two breakouts through the KVM. So if you have a process running in a VM, the process breaks through security in the VM, then it breaks through the hypervisor and actually gets control of the host system. So that's the object of any hacker that's running in EC2 right now is to get to that point, because then you can take over the universe. There's been two known breakouts, two known vulnerabilities in that. Both of those were blocked by Esper. Etsy Linux controlling what's going on. All right, but let me go a little further into what this type of protection is. So we can identify everything as a Docker type or as an OpenShift type or as a VM type, but we have to have some other piece of data that gives us separation. So an Etsy Linux system has this fourth field called the level field. Anybody in here ever do anything with trusted systems? All right, just one or two, three people. So things like Trusted Solaris or Etsy Linux all the way back since REL 5. The concept was this multi-level system where you would have a top secret process and a secret process. And there's rules that were written many, many years ago back in the 1970s that said a top secret process can read secret data, and a secret process can write top secret data, but a top secret process cannot write secret data. Pretty strange rules, right? But that's what MLS, standard MLS, and that's not really totally the way they work, but the idea was that if I have a secret process running a secret process reading secret data, I don't want that secret process to be able to write non-secret data. That is the WikiLeaks stuff, right? I don't want to take the top secret data and let it get down to secret. So that's why top secret processes can't write secret data. Similarly, the other one that's weird is that a secret process can write top secret data. And I always equate to the soldier who sees the tanks over the hill, and he's going to tell the commander that this tank's over the hill. So he's actually writing data that might be only secret, but when he tells the generals all of a sudden it becomes top secret data. That's the basic idea. So anyways, the problem with MLS, which we introduced in REL 5 back in 2006, is that almost nobody in the universe, other than the military, sets up their systems to run in that way. So we had this fourth fail that we weren't taking advantage of. So most targeted systems we've played with at the Linux has that as that's zero. So we decided to take advantage of that. We made a new thing called MCS, which is multi-category security. And what we did is we basically allow us to assign random labels here. And the rules basically say that that random label has to match. So if I have a process that has a label of S0, C1, C2, I have to have an object in the operating system that has the same label. And that's what S-Virt is. So we'll look at it. So S-Virt, if I'm running a virtual machine, it's going to run as the S-Virt type. And then LibVirt, the tool that launches virtualization in KVM, will pick out a random MCS label and assign it to it. Then it will pick out a file in the file system, assign the same label. And then basically the kernel will say that these two have to match, otherwise you're not going to be allowed to read or write that file. MCS, so it's multi-category security based on the multi-level security, which I went over pretty quickly. And now we're going to go through the next chapter of the SC Linux coloring book to cover MCS security. So in this case, we have two dogs. So they're both the dog type. But what we want to do is we're going to assign a random label, in this case it's not so random. We're picking a name, but it's Fido and Spot. We're going to now use our example of dog chow for both of them, but now we've added an MCS label part of the dog chow. So we've said that's going to be Fido, and this dog chow is going to be Spot. And if Spot's eating, in this case, the kernel must have been permissive, because Fido is eating Spot's dog food. So in enforcing mode, Fido would be allowed to eat his dog food, and then at the kernel, the kernel would stop Fido from eating Spot's dog food. So that's the basic idea of MCS. And now let's go, OK, MCS enforcement protects like processes from each other, VMs, OpenShift gears, SC Linux sandboxes, Docker containers. I figured this thing out. I've got a big hammer, and all I've been doing is looking for nails since then. So everything that is a multi-tenant environment needs this. Rocket now supports this. So if you're using Rocket Container System, the N spawn supports it. So all these environments support this type of security. The tools that launch the containers, that launch the object, picked out the random labels, signed the MCS label to all content, launched the process same label. And then the tools, the tool is in charge of guaranteeing the uniqueness. So Docker has to guarantee uniqueness. Libvert has to guarantee uniqueness. Rocket, any other tool that's going to do this has to guarantee the uniqueness. So let's take, hopefully this is not too, too loud. Today I'm going to demonstrate how SC Linux controls what an OpenShift gear is able to do on a system, even if the OpenShift gear becomes root. I start out by showing that my process on a system is running as unconfined T, and I'm about to use RunCon to become the OpenShift label. I also show that I'm running as root on the system. So if I run the RunCon command here, I'm going to switch over to a label of a statue on a system, our OpenShift T S0 colon C0 comma C1. When an OpenShift gear is launched on an OpenShift system, they are always launched with the OpenShift T type. But each OpenShift gear also gets attached to it, and a unique MCF label here, it's S0 colon C0 comma C1, is associated with the process. Also, the gear gets its content also labeled with the same MCF label. So now that I'm running as OpenShift T, I'm going to show you that I'm still running as root on the system, but now if I try to cat at the shadow, I'm going to get permission denied. If I try to create content on the system, I'm going to get permission denied. And even if I go into the BioLive OpenShift directory where the different gears store that content, in this case, I set up two directories for my demonstration, one called gear 1 and one called gear 2. If you notice, gear 1 has the MCF label of S0 colon C0 comma C1, and gear 2 has S0 colon C3 comma C4. So if I show an OpenShift gear trying to attack another gear, in this case representing myself as gear 1, I'm going to try to go into the gear 2 directory and look around. In this case, I could permission denied. But if I attempt to go into my own directory, I can see content in there. And if I want to create new content, I can create it. If I look at the labels on the content that I create, it's all labeled with the MCF label of S0 colon C0 comma C1. So now, if I get frustrated with SC Linux, I want to turn it off. I can attempt to turn it off with setting for S0. And of course, we're also blocked from being able to disable SC Linux on the system. So this demonstration, this little video demonstrates what SC Linux can control what an OpenShift instance is able to do, even if the OpenShift instance somehow became root on the system. To give you an idea of OpenShift, does everybody know what OpenShift is? OK, a few of you do. So OpenShift is a PAD server that Red Hat introduced, I think, four or five years ago now. Matter of fact, the guy that developed it was Mike McGrath, who now works with me and the Docker team. This is OpenShift v2 that we're demonstrating. OpenShift v3 is going to be based on top of Docker containers with Kubernetes and stuff. But basically, it's going to use the same technology. So OpenShift has sort of taken off. I think they have now calculated they have about 2 million users of it. OpenShift is based in EC2 cloud instances. Anybody in the world can get an account there. All you have to do is provide an email address and you're able to get an account. We also allow you to have shell access to OpenShift v2 machines. So we give you shell because we figure if we're giving you the ability to build web services on top of a PAD server, you're going to figure out how to get to shell on top of that system. So right out of the box, we give you shell. We take away no security, no tooling from you. So we allow you to use any tools you want. The only thing we do is we wrap you with SC Linux to make sure you break out. We have had over four years with 2 million users. We've had no known exploits. Maybe the NSA's done something. I don't know. But we've had nobody has broken through the security with a shell account on the system. We also monitor the heck out of this, make sure the kernel's totally up to date, things like that. So the attack vector here would be through the kernel. Yes. Because of the NSA, my answer to that is if I worked for the NSA and I wanted to exploit all machines in the world, I would not put an exploit into the thing that comes from the NSA. I would have someone get a random email account, give a patch to NVIDIA driver or something like that, and that would be my attack vector. This is open source. So it's open source. Everybody in the world has looked at the SC Linux code. It's gone through multiple review processes. It's far more likely that they haven't, if the NSA had an exploit, it's far more likely that it's going to do that. So turning off a security feature that has no effect on the kernel. That's just basically the kernel. You don't know what's going on in the kernel, whether SC Linux is enabled or disabled. The code is in the kernel. That to me, I always think is ridiculous. That's coming up in the set. Well, the labels are written in policy. So the policy writer's right the rules. I'm going to show you how the labels get on to processes and labels get on. OK, so when LibVirt launches the virtual machine, it picks out an MCS label to assign to the process. Then it assigns the image the same label. When LibVirt takes down the container, it basically kills the process and then relabels the label on the VM to something that no VMs are allowed to read. So it can change the labels all the time. It just doesn't leave permanent content on it. Yep. It's just that those sequence of letters, there's more to it than that, that those labels in an MLS system mean something. In an MCS system, we're just taking advantage that we have this construct of MCS labeling. And we're just using it. The reason we use two labels, so we use S0, S0 is a sensitivity level. So in an MLS system, you'd have S0 to S15. We only use one sensitivity on the system. And we have 1024 capabilities. So we can use 1024 two-character combinations. That gives us a half a million potential, unique containers versus SBIR. The uniqueness is basically 1024. It's a half a million unique VMs you could run on a system. So when we designed it in 2008, we thought that was pretty good that we wouldn't have a problem. That was before we got the containers. Now theoretically, containers are going to, as we see with OpenShift, they have 2 million users. All of a sudden, we have potential problems with that. But guess what? I can add a third label to this. So I could use S0, S1, S3, and all of a sudden I get another multiple of 1024 times that. Minus some numbers. But anyways, it can get huge if we really find that to be a problem. So most SE linux issues are caused by labeling issues. SE linux should work by default if everything is labeled correctly. Sadly, not everyone who wants to store their content the way I tell you to store your content. There's a problem. Every process and object of the machine has a label. If it files in that label correctly, access might be denied. Alternative path to confined domains, SE linux needs to know. So if you want to install your Apache server in some random directory on the system, you've got to tell SE linux about it. If you want to move your content, your Apache content to random directories, like slash u. I don't know why people like slash u, but it must be an old Solaris thing. Or you have slash server, slash web. There's certain directories that people put their content in. You have to tell SE linux about it. The way you do it is there's an SE manage command, which is the SE linux management command. And that's SE manage file context. So this is basically saying I want everything under this directory to be labeled as Apache content. When you use an SE manage command to do management, that just tells SE linux what the default should be. It does not change the iNodes on the system. I didn't mention earlier that SE linux is based on iNodes. So the same way discretionary access works, the label is associated with the iNode on the system. It's not path based, but it gets assigned from a path based system. It gets assigned to the labels, gets assigned via the file context file. After I set this change my default, I have to run a command called restorecon. If you play with SE linux for a while, you're going to get real familiar with restorecon. Because restorecon basically says, change the labels of this file or this directory back to the default. Go read the file context file and change the default. Lots and lots of tools on the system have file context built into them, things like RPM. So when I install a package on the system, RPM will automatically set the correct label. So when I install the default Apache server, it gets installed. So that's how the labeling initially gets created on the system. But if you want to put contents in random directories, you're going to tell SE linux about. So file labeling. So the file context file is stored into this directory. SE linux targeted context file context. So all the labels are stored there. If you want to go and look at that file, you're going to see thousands and thousands of reg-x's that way we've learned content gets stored. File labels are usually stored as x-hatters. Almost all we stored is x-hatters. Certain cases file systems don't support x-hatters. Yes. Yes, it performs perfectly zero performance hit. Is there a cost to discretionary access control? OK, so it's the same basic cost. Pretty much with the modern linux, they just did some studies on it. We published them. SE linux has negligible effect. You can set up horrible situations. It's not the labeling on the system that's going to cause it. It's basically the kernel checking access control on there. But the kernel's already going through lots of access checks every time you access an object. So for the most part, anybody that disables SE linux, because of performance, unless you're Wall Street trying to get an extra micro second, whatever, you're fooling yourself. There's so much CPU that it's not really a problem. Yes, I'm never going to get through this. It's going to go about three hours. So even in that situation, the one place to me you would have to really worry about is network throughput. So that's where you really need to check. Because again, for the situation where you want packets flying as fast as possible, the SE linux system is going to be checking every single packet that comes in. It potentially could check every single packet come in to make sure that it's a secret packet versus a top secret packet. So there are those checks built in the system. So match path cons. Another tool that you can use to figure out what the path is going to be on the SE linux system if you played with it, any core utilities, any commands you want to run, always look for the dash capital Z. So that's sort of the SE linux flag that tells you what the labels are associated with objects on the system. SE managed context file is used to change the default labeling, and restore can't command takes the default labeling and applies it to the X adders on the file system. I already showed that. So file labeling. So government entity uses Tivoli. I always change these to protect the innocent. Tivoli stores log files and via IBM, Tivoli, Common, Cod logs. Anybody have any idea why vialogs wasn't good enough for IBM? It's this type of stuff that drives me crazy over the years, OK? It's like, we'll just pick a minute. If you ever looked at an Oracle system, oh, yeah. OK, so we don't know where Tivoli stores its logs. So someone installs a log package in here. They put the logs in there. They tell the log rotate tools and the log analyze tools that this is there. They forget to tell SE linux about it. SE linux has a problem with logging tools because they run as root. They analyze random code that can be generated by anybody, and then they take actions based on that code. Well, that seems like a good exploit target vector, right? So we needed to have that. I see ABCs on confined domain saying, it's trying to access VAT, VAT is the default label of VAR. We don't have labels for IBM storing in random locations. I see labels. How do you figure out what the label of this be? It has the word logs in there. I instantaneously say, I wonder what the label on vialogs is. And I go and look at that. And it says, LS is vialog. And I see it's vialog T. And so I run an SE manage man that says they manage fcontext, add vialog T to IBM's weird location for its logs. And run RestoreCon to set it. I usually go at a fairly global level to run these, but you could have done it down to the path. Problem solved. Default T files. So SE linux knows about all the stuff that's in slash that comes from the distribution. Any other crap you put into slash, I have no ideas. And SE linux has no idea what it is. It could be top secret data. It could be credit card data. So we don't allow any confined domain on the system to look at anything that's in the slash directory that we don't know what it is. Similarly, we don't allow it to look at VAR-T. So VAR-T, we don't know what that content is. So you have to tell SE linux that you're going to store content there. So by default, all non-distribution directories in slash will be labeled default T. SE linux has no idea what content is. Yeah, I already said that. So a big retailer moves its home directories to slash-u slash-home. This is another Solaris thing, I think. I don't know why slash-home wasn't good enough for them, but they wanted it in slash-u slash-home. So slash-homes, guess what you're going to label slash-home as? Anybody know what directory we should look at for this? Slash-home. Very good. So this means all home directory content is labeled as default T. So we have confined domains like SSHD and Apache that want to look at content in your home directories, and I don't know if it's credit card data. So we block it. We probably want you home to be the same label as slash-home directory. So what do we do? Well, one of the problems with just doing what we did with SE Managed there is we could probably have multiple things labels underneath slash-home for differences. My home dot SSH directory is probably labeled differently than my home slash public HTML directory. So I don't want to go through and execute the SE Managed command 400 times. So what I can do is I can do an equivalency. So the equivalency label says fcontext add, I'm actually doing a regular label here. So I'm doing the same slash-u. I'm just going to name it the same as slash-home. But then I do an equivalency. And all it says is dash-e treat slash-u slash-home as if it was under slash-home. And when restock on u slash-home goes in and asks the system is actually asked, but slash-home-dwalsh set the labels instead of slash-u slash-home. So it's pretty simple. Labels under all files under u-home is if they were labeled under home slash u-home-dwalsh gets labeled as that. And public HTML gets labeled as that. So it'll use the same labels as if under home. And of course you have to run the store con on it. So common problem, creating content to my home directory and then I move it somewhere with the mv command. Everybody knows the rule about the mv command when you move content, right? It maintains the security attributes of the object that you're moving to the new location. When I move a file to a root system, it does not change from dwalsh to own a root. It stays as dwalsh. If I copy a file to the slash directory and that file does not pre-exist and I'm running his root when I copy it because I wouldn't be able to do it as dwalsh, what happens to the security attributes of the file I create? It's owned by root. So guess what? When I move an object on a system from, say I'm creating index.html on my home directory, which is use a home t, it's use a home content, probably where I store my credit card data. And I move that index.html to vww.html, it maintains the security attributes that it's dwalsh and it maintains the security attributes that it's use a home t. So now I go and run Apache. Apache gets permission denied. I go and say, oh, I forgot to change the ownership to Apache. I changed the ownership. I run Apache again. It gets permission denied. And what do I say? Now, SELinux sucks. Follow me. Follow me on this. OK. So this is the use case. And this is probably one of the biggest use cases for Apache content getting mislabeled. It's either created in the slash temp directory and slash root slash home directory. I'm happy with that. I run a Firefox. Now I want to move it into production. And then I forget to fix the SELinux labels. So I get that. So I check the permissions. I already went through this. SELinux sucks. OK. So what I can do after I move the object into the place is I can run RestoreCon. Because RestoreCon is going to look at the path and force everything to be the way SELinux likes underneath that directory. Yes. If a confined root user, first of all, SELinux knows nothing about the root user. It's a label on a process. If a label, as I showed you in the OpenShift example, that was the root user trying to catch Etsy shadow. But because it was labeled as OpenShiftT, it's not allowed. So root means nothing. Root is a discretionary access control non-SELinux issue. So move preserves labels. There's a move in REL7 now. There is a move capital Z. And what that does is everybody that uses move with SELinux expects it to do the RestoreCon for you. So now we have a move minus Z, which will relabel the content for you. So if you do this, which I have on my system, suddenly things start to work much better. The NSA hates the idea of this. The discretionary, but if you think about the top secret, if I move a file around the operating system, and all sudden it goes from top secret to unclassified, that's a problem. But for targeted systems, for the systems that most everybody uses, it makes a lot of sense. That's why we can never change the default to be minus Z, but you can change it yourself. Not a common problem. My machine's labeling is so wrong, it will not boot. OK. So there's a kernel parameter you can use called enforcing equals zero. It's different than SELinux equals zero. If you boot a machine with SELinux equals zero, because you think SELinux is screwing up your system, that will disable SELinux. But SELinux, the operating system, is smart enough to know that you booted without SELinux. So what it's going to do, it's going to put a little file, slash dot auto-relabel, and the next time you boot with SELinux, it's going to relabel everything, because it has no idea what you created while you were logged in without SELinux. So it forces the relabel. If you boot with enforcing equals zero, that means that the system is running in permissive mode, equivalent of set and force zero. So the labels will continue to get created correctly, but SELinux will not block anything. Touch auto-reboot will re-boot label the entire system. And if you have the example you were using where you have 8 million files there, a mail server type thing, go get lunch, because it's going to walk through the entire file system, looking at every single I-node, and checking to fix the I-nodes. Usually on a modern system, it probably takes five to 10 minutes on a normal laptop, but on massive systems, it can take massive amount of time. So brace yourselves. How do labels processes get on labels? SELinux implements process transitions. I can write a rule and policy that says, when process labeled A executes a file labeled B exec T, the kernel will transition the process to B underscore T. So I'll give you an example. When the unit system, which is running as a knit T, runs Apache, which is labeled as HDPD exec T, it will run it as HDPDT underscore T. So that's how transitions work. That's how processes get their label. So it's a rule that looks like that. Look what happens at boot. OK, so this is a fairly old slide. It doesn't even cover system D. But when a system boots up with SELinux, the first thing the system does when it comes up, well, this is somewhat of a lie because there's no SELinux when a machine boots up. So the machine's booting up, it's going to read in the SELinux policy off the file system. And what the policy says, any process on the system, by default, at the point that it reads in the policy, it's going to be kernel underscore T. The policy has a rule that says, if kernel T executes an init program called label the knit exec T, it will run that process as a knit T. So kernel T runs system D. System D is labeled this. It runs as a knit T. Now we go down and it runs an init script, which is labeled initRT exec T, which transitions to knit RTT. That goes down and runs K log exec T, which transitions to K log exec T, or init name D exec T. So obviously this is a RHEL 6 system because system D works a little bit differently, but not too much differently. Similarly, you get down to KETI running. It gets down to log in. At which point the log in stuff works a little weird. So log in program has SELinux knowledge in it. So that log in program is going to figure out what the log in the user in. So it has SELinux. It picks the labels that log in it. But this basically is how all the processes get on the system. Most processes you exec stay in your context. So if you're running as unconfined T and you execute a program, it's probably going to stay as unconfined T. But we've written policy that you can be running as a user T. You execute Firefox and it'll go run in Firefox T. So we have that type of transitions for user space. How files get created on creation. So two things that happen. We've told you how they get initially labeled via RPM in the file context files in RestoreCon. And we told you how processes get labels. Now we're talking about an administrator or some object on the system is creating new content. How does that get labeled? Multiple ways. The first way is a file gets created based on its parent directory. So if you create a file inside of a directory labeled SET by default, it will get created as SET. If a tool like SELinux, is that SELinux where it can override that. So RPM, for example, tells the kernel, I'm about to create an object, please create it with this label. RestoreCon changes it. For example, password creates two contents when it creates objects on the system. It creates SET password and SET shadow. So it's going to create these with different labels. So password has to have SELinux awareness in it. Policy writers can write rules to what we call file creation transitions. And this basically means when a process is labeled network manager T and it creates a file in a directory labeled SET, we want it to create it as netconfT. That means when network manager creates at the resolve.conf, it creates it with a label of netconfT instead of being SET. So we can write all these rules into policy. And we can do, this is what the policy will look like, network manager T creating SET, we'll label it as netconfT and create it as a file. So there's a fourth one that just came, this is only in rel7, not in rel6, and this is actually a huge advantage to usability in rel7 versus rel6. We can write rules that says when you create a file, you can do a transition based on the name of the file. Not the path of the file. Path means nothing to Linux kernel. Names of file objects mean nothing. So I can write a rule that says when an unconfined T process creates a directory and a user homeDirtT, which is the home directory label, I want to create it as SSH homeT only if it's named .ssh. So one of the biggest bugs that we get for rel6 and rel5 is admins set up a brand new SC Linux system. They go into root slash .ssh, they create the root at .ssh directory, and all of a sudden they try to set up public key automatic logins, and it blows up because this is Hdemons not able to read content in the root home directory because they have to run a stock con. This actually fixes that problem. So we have hundreds of these rules. So if this happens to go on creating random content that we know about, we're going to label it. So if he creates that to resolve .conf, we label it correctly so we can set up all these rules. It also, we actually have patents on this to improve security of the system because we can actually tighten up some of these file transition rules. OK. I've just beaten the living hell out of labeling. But there's a hell of a lot more. If we really wanted to go through it, we could go over three days. But basically, all this stuff should work. And everything I've talked about up to this point is if you pick up one thing, it's for stock con and the SC Manage command. Those are the two most important things we just covered. We gave you a little more insight of what's going on in the system. Section two is you change the system to false, but you do not tell SC Linux about it. SC Linux needs to know. These are, when we write policy, if I write policy to just allow Apache to work, I end up writing unconfined T. Because Apache can be configured in 4,000 different ways. Things like FTP. I set up an FTP server. I want FTP server to share our FTP to the internet. OK, someone else wants to use FTP to be able to share home directory content to the internet. Someone else uses FTP, and they want to share everything on the system to the internet. So how should I set up the default policy for the FTP demand? So we introduced many, many years ago the idea of Booleans. So that allows a policy writer to write these Booleans or if then else rules into policy and allows an administrator to toggle it a little bit. So Booleans are if then else rules written into policy. If you want Apache to send mail, you can do that command. How many people here set up their Apache service to send mail? Good. Very few, yeah. The default is that it's not allowed to. Why is that bash exploit happens all of a sudden? You become a spam bot. So out of the box, you would expect your Apache service not to be able to send mail, except for the two guys in the room who wanted to send mail. And they're going to say, SELinux sucks, OK? All disabled, right? They're going to do 7-4-0 and disable SELinux, get the hell out of my way. So we have the Booleans to do that. If you want to use it to use FTP to access this home directory, you can set a Boolean that says FTP Home Dirty. How do I know about the Booleans that are available in the file context? So SCManageBooleanList will show you all the Booleans. We actually have in rel 7, there's over 1,000 man pages. Every single confined domain in the system has an underscore SELinux after it. So it's not bad to do man-k, SELinux, and then grep for some strength. So if you wanted to look up a man-htp-d underscore SELinux. By the way, I didn't write 1,000 man pages. So these are all auto-generated, but it does an incredibly good job of that. And this thing will cover all the stuff like, what are the Booleans available for the Apache Deam, and what are the types available for the Apache Deam. So there are lots and lots of man pages to help you out. There's a tool called SC Troubleshoot, which usually runs on desktops. Anybody that's played with the Linux desktop, Fedora desktop, has seen SC Troubleshoot. SC Troubleshoot, I used to say, is Dan's brain in a box. So it basically looks at an AVC and tries to analyze it and then gives you a response that says, hey, it looks like Apache's trying to send mail. You might want to turn this Boolean on, if it wants to send mail. If you don't want Apache to send mail on, you're being hacked. I don't put that pot in, so I hope people just don't read this and say, oh, I didn't know my Apache wanted to send mail. I mean, still, it's sent mail. There's a tool, Auto to Allow, that we're going to cover in a little bit. Auto to Allow basically can analyze your log error. So SC Linux Error is going to the audit log system by default. They're also now in REL7, they're going into, I think in REL7, or at least in Fedora now, they go into the journal as well as the auto log. So if you run Auto to Allow to look at what happened on your system, Auto to Allow will give you a message that says, here's the rule, you'd want to add to your policy, but hey, there's two Booleans available to allow you to allow this access. Using Auto to Allow to diagnose issues when Apache is running passenger, you end up with something that looks like this. So if you wanted to run passenger, you can get these Booleans and turn them on. Problem solved. You didn't change the system to false, but you changed the system to false, but you didn't tell SC Linux about it. SC Linux needs to know. SC Linux also controls port types. So we basically say Apache is only able to listen to port 80. SSHD is only allowed to listen to port 23, 22. I never remember which. So out of the box, if I see an Apache web server trying to listen to port 492 or SSHD and trying to listen to some random port, SC Linux is going to block it, going to report it to an error and say, hey, something strange going on in the system. So I want SSHD to listen to port 55. SC Managed Port is the tool you're allowed to change that. And this basically says treat port 55 as the SSH port. How do I tell what ports are available? Well, man, SSHD and underscore SC Linux will tell you all the port types are available. There's also an SC Managed Port list, which will tell you a huge amount of ports. Port prefix usually matches the type associated with SSHD, so anybody guess what HTTPD underscore T's port type is? HTTPD underscore port underscore T. So in row seven, SC Policy Network is very handy. That shows you stuff like this. SC Policy Network is basically tools to analyze the policy. It tells me which port is assigned to SSHD's port T. And if I want to know all the stuff about the SSHD man T as far as networks are concerned, these are all the ports that are allowed, it shows you some Boolean information, so there's lots of information available. So those are the two things. So two major causes, SC Linux labeling issues. The second one is you change some configs, change some normal config, and SC Linux needs to be told about it. The next one is SC Linux errors, SC Linux and application errors. Over the years, as we've evolved, row four, we had a ton of errors. Row five, we had less errors. Row six, we got better. Row seven, we got better. We're always getting better at figuring out these, because we've been running SC Linux since Fedora 3, and now we're on Fedora 23. So we've been doing this for 20 versions of Fedora. So SC Linux policy can have bugs, unusual code path configurations that we never tried, redirection of standard out. So all we can do with SC Linux is run an application under a confined domain, and then wait to see what blows up. So we learn over time. It's like doing static code analysis. We can do a little bit of that. But you really need to run these things in the wild to figure out all the things in the applications they have to do in the system. And then we write rules about that. Applications are SC Linux bugs that have been not fixed yet. Applications have bugs. Everybody know what a leaked file descriptor is. So 30 years ago, when they were developing the C language, they decided that when I did a fork exec, I want standard out, standard error, and standard end to be leaked to the new process. So when I do a fork, all the open file descriptors in my original process get inherited by the child process. Do you know that it's not just 0, 1, and 2 that get inherited, it's so, you got a network socket open, it'll get inherited by the child process. Every single process, and that now becomes a child, grandchild on down, has full access to that process. When we first started using SC Linux many years ago, we found the init system was leaking dev init control. That's the socket that you're able to send a packet to to change the run level of the system. So every process on the system had full access to that, unless an intervening process went through and closed all file descriptors. So we were finding all these domains that basically could change the run level on the system. No other system in the world would figure that out, except for SC Linux. But anybody that writes code, every single developer in this room that writes code, probably never thinks to close the file descriptor before an exec. So there are calls that you can basically say, this file descriptor, I want it closed when anybody execs out of here. But you have to code that way. So we found hundreds of those. Executable memory, this is basically buffer overflow attacks, SC Linux will block those. But a lot of code is badly ridden. Badly built libraries cause these problems. Report bugs to bugzilla. If you find them, we will fix them. Applications or SC Linux have bugs. You can tell SC Linux to just allow those using to audit to allow. So SC Linux is blocking Postgres SQL. Did you make the labeling correct? Yes? Did you make sure there were no appropriate booleans? Yes? Most important thing you're going to learn today are the most abused thing you're going to learn today. How do you use audit to allow to build policy? So use audit to allow to build your own custom policy module. If SC Linux blocks something on Postgres SQL, you can go through your audit logs, pipe it to audit to allow, give it a dash M, and tell it to create my Postgres SQL.pp policy package. Then you can install the policy package. This basically says SC Linux, shut the hell up and let me do what I want to do. OK? The problem with this, if you do this all the time, you're allowing the hacker to do anything you want. So you really should look at what you're doing here, understand it. But if you just want to get, if you don't want to send, this is better than sending for a zero. It's much better than disable SC Linux. But it allows you to get by. But when this happens, you should really look at what you're adding. Try to figure out what you're adding for rules. Make sure you're not allowing too much, and you can always ask for help. There's lots and lots of people on the internet that will help you with your SC Linux issues. You can even send me email. Report the bug to your team, your support, your bug to Zilla, mailing list somewhere. Report it so we can find and fix it if it becomes a bug. So the big one. So all of a sudden, we have Shell Shock. We have a Apache web service trying to send for zero on the system. OK? Might be something we want to investigate. You probably don't want to do a send for zero audit to allow, oh, let the dash CGI script turn SC Linux off. So if you have a confined domain that tries to load a kernel module, turn off SC Linux in forcing mode. Right to Etsy, or right to ShadowT, modify IP tables rules. You might be compromised. The problem is the tooling really can't tell the difference. So Etsy Troubleshoot tries to do a good job of looking for these known signatures of attack vector, like disable SC Linux, and figure out no confined domains ever going to be allowed to disable SC Linux. So those are good things to watch for. So Etsy Troubleshoot has a few diagnosis tools, but Etsy Troubleshoot and SC Linux is not an intrusion detection tool. All it is is protecting you in case you had a vulnerability. Questions? By the way, there's more slides after this, if we have time. Yes? So like console-based access to the box, and then just set it for SOV that I can load on? OK, so the question is, he's asking about confined users. So I didn't really cover confined users here. On most systems, by default, the user that logs onto the system is on confined T. OK? That basically means he's able to, he basically has allow star, star, colon, star, star. OK, that's the rules for the unconfined user. So if you have a user that logs into the system, his root is full access to the system. If you log into your home directory, you have full access to the system. You go through sudo, become root, you can do anything you want. That is the default unconfined user on the system. I think we have a couple slides in the next section that might cover confined users. We can control the user process. The problem is, if I controlled the user process out of the box, we would have everybody disabling SELinux, because nobody as an administrator wants to be confined. I actually run confined. So I run a staff T. And the staff T, the only way a staff T user can become root, is actually through sudo. And I have a transition rule through sudo that can allow me to become administrator, at which point I become unconfined T. There's actually even tighter things called sysadmin T that are used in like MLS systems. I have policy, remember I showed a picture, I confined my wife with SELinux, she runs the thing called ex-guest. She has no reason to ever connect to random ports. All she does is use a web browser to get onto the internet, so I can actually control that she can only use connect to the Firefox type ports. She can't run su, sudo, there's no way. So if she downloads an application that tries to run su with sudo, SELinux will block it. Same thing with my parents. My parents run in the ex-guest. Basically it's a user that can log into a box and do very little. There's even a tighter policy called guest T, which is basically for SSH accounts. If you have an SSH account in your system that you allow hundreds of users to log into, you really want to put those guys in a box so you don't want them leaving the network. When I do the longer talks, I call it the Hotel California. You can come in any time you want, but you can't leave. So it basically controls who's able to log in the box, but you can't SSH off of the box. And these are for web services. So my content, ftp.redhat.com, fedora.people.redhat.com, they give me an SSH account so I can update my public HTML. But they don't want me running su, sudo, and all these tools to try to become root on the system. So if I could lock those down with SELinux. So I have the capabilities to do that. But SELinux is more about, by default, is more about controlling network-facing services rather than users that log into the system. You can enable the log is used in a system, but that's more of an advanced SELinux thing. You can do a really nice job, too, of controlling sudo. So you can actually have a confined staff T user, and you can set up a user that can execute a shell script as root, and then write policy on that process to control it. Yes. Yes. I mean, you could build some kind of system if you cut off certain categories and treated those differently. But yeah, pretty much no one to my knowledge is combined. Yes. How do I handle this diversely? Well, SELinux can run in the container, well, switch not containers, it can run in the VMs at separate policy. So you can run a REL6 box with a REL6 policy on top of a REL7, as long as you're using KVM for virtualization. If you're running inside of containers, we don't disable SELinux inside. We lie inside of the container that SELinux is enabled, because we don't want any process inside of the container accidentally attempting to do SELinux actions. So if you're inside of a container, if you did it again in force, it'll say disabled. If you get outside of the container, same process in the system, it will be in a forcing mode. So in containers, we only have one policy. And my rule about containers is what happens in Vegas stays in Vegas. So we allow you to do anything you want inside of your container, but you're not allowed to break out of the container. And we use the SBIRT MCS. That's basically how OpenShift works, too. But yeah, as long as you can use full KVM virtualization, separate kernels, then you can have separate policies for each object. Anybody else? Yep. Yeah, there has been some effort to get Fuse to work over the years. And there is effort like NFS. So Fuse file systems, the certain file systems that don't support X adders properly to be able to handle it. So NFS for years didn't support it. And basically, with an SC Linux system, you basically label the entire mount point with a single label. So NFS by default, NFST. NFS now, if you use REL7 clients and servers and hopefully soon NetApp and other services will support. NFS protocol now supports it, just a matter of getting the services, all the clients and services to start supporting X adders. Fuse has the same type of problem. And Fuse has some weird situations that have never quite been fixed. Anybody using a Samsung phone? Anybody using the Nexus? This is the Nexus phone, Nexus phone. You guys disable SC Linux on this? Did you know SC Linux is on it? So SC Android, why is SC Android easier than SC Linux on a REL box? Why is it able to confine it easier? Anybody have any idea? Remember the canonical talk yesterday? The amoebas, the amoeba applications crawling all over each other, right? FTP being able to do 15 different configurations and all this weird configuration stuff that's happened. When I'm using an app on top of one of these phones, what is it? It's a box, as the space man was saying yesterday. In our terms, we call them containers. So when we move to containers, SC Linux becomes a lot better because I can treat every single container as the same object. I no longer have to worry about these random directories and random way you set up things. So SC Linux, on top of virtual machines, on top of containers, almost becomes second age. Anybody that disables SC Linux on Docker, you are out of your frigging mind, OK? Especially if you're running a fellow co-worker of mine called Docker is all about downloading random shit off the internet and running his root on your system, OK? And that's really what it is, right? Because you're just going and grabbing random content, and you put it in a container, and you're giving it root. And there's very little separation. There's some separation. I do a much longer talk on that, which we checked it for here, on Docker security. But SC Linux provides the best file system separation. Again, because it can basically say you are a container process running on a system. If you try to read database data, you're not going to be allowed. You try to read SC Shadow. You're not going to be allowed. Basically, what happens in Vegas stays in Vegas. You try to break out of Vegas. SC Linux is going to shut you down. And because it's all built into the system, it's built into Docker. The only time you have to worry about labeling is if you volume mount into it. And there's a colon Z, low KZ and a colon uppercase Z, which actually will fix the labels of whatever you stick. If you volume mount into a Docker container, it will fix the SC Linux label of whatever you volume mount in. Someone back in the room? Yep. No. Because SC Linux is not a mechanism for getting around discretionary access control or other types. Security's always in depth. You always want, when I run Docker, I have read-only file mounts. I have discretionary access control drops. I have capability drops. I have SC Linux drops. I have Sec Comp. I want all these things. What I don't want is one of them installed and bypasses the other ones. So you have to configure everything. So we're not giving you backdoors with SC Linux. So in order to become root on the system, if you're a confined application, you have to not only change your label, SC Linux label to do stuff, but you also have to change through sudo to become root, UID zero. So that's different. There's no secret SC Linux bypass mode to bypass things like discretionary access control. Are we running out of time? Yeah. No. They use live fuse, libfuse? I don't know. I don't know what that does. Yeah. I don't know fuse, so. No. I just worry about it. OK, I'm going to upload all these slides. I think we're totally out of time now. The second part of this talk covers how to manage SC Linux in a distributed environment, which is really critical. Again, this talk could take three days if we wanted to go through it all. But the main idea is you configure, a lot of people go on and you configure one machine the way you want it. All right, so you go on and you do all your SC managed commands, you do all that. SC managed commands also have the ability to extract your configuration, all your SC Linux configuration, into a flat file. You can then move that flat file to another machine and load it. And what that will do is basically take apart all your booleans, all your file context changes, any port changes, things like that. So if you configured one Apache server the way you want it and you want to duplicate that Apache server on 100 machines, you can do that with these tools. Lastly, about SC Linux, again, this covers it. The question we've often gotten asked about SC Linux, don't we have some magical tool that makes managing SC Linux easy? And I always say yes, we do. We have lots of them. They're called Puppet, Ansible, CF Engine. Whatever SC Linux's configuration data happens to be security configuration data, but it's configuration data on your system. If you manage your systems using the satellite with RPMs, you can package SC Linux and satellites on RPMs. If you manage using Ansible, great idea. We now own them. You can do it with Puppet. You can do it with other tools. So managing SC Linux in the enterprise, the way that Fedora does it, the way that OpenShift does it, they use Ansible to manage all hundreds to thousands of machines using standard tools. Thank you for coming. And did the Patriots lose? Thank you.