 All right, so this is a question that many people have posed. Is cockpit secure? A more fundamental question actually is, what is cockpit? But if you're asking that question, then you should try cockpit out. It's super easy to install in your system. We have a really good, you go to cockpitproject.org, and it doesn't load. But at cockpitproject.org, there is a really easy way in one or two commands to run on almost any distribution. It's distributed and tested all over the place. So I'm going to show you a bit of cockpit here and there, but I'm not going to go into the details of it. I'm Steph, and I work at Red Hat. And some of the other members of the cockpit team are here. And we've been building a UI for Linux that lets you log into the system and perform all sorts of tasks on the system. And the question of whether it's secure is a very variable one. It depends on what your risk mitigation expects, what kind of security you expect, what your security model is. And as always, there's an XKCD for this. We like to focus engineers very closely on the implementation details of security. Is it theoretically possible to break into this thing? And in this case, the laptop is locked with 4K, bit, RSA, and boom, suddenly we stop in our tracks. In theory, it's impossible to break in. Whereas in practice, we have another threat with a different kind of impact. Someone coming in and beating you up, and that gets you into the laptop. And you'll see this in many security discussions. You'll see this displayed like this. We have threats. These are usually people or companies or different parties. We have vulnerabilities. We often see these as CVEs. We have impacts. What happens if someone exploits it? And that represents the risk. So these are a lot of variables here. Threats are how likely someone cares about you. For example, vulnerabilities, how easy to break in. Is it remote, exploitable? Is it local? Does someone have to come into your location to exploit it? And the impact, what do they get in the end? So what I'm going to present in this talk is for cockpit, some things that help you make the decisions of what to place in these variables for your deployment, for your systems that you use cockpit on. But first, let's stack the deck against cockpit. Let's show why we're even asking this question. Cockpit is actually a Linux session in your web browser. I should do a Dan Walsh thing here where I make everyone stand up and say this, but maybe for the next talk. And here's a good example of it. I've opened a JavaScript console here at the bottom. And I can literally run commands. In fact, let's just do that live. So here's cockpit. And I'm logged into the system. I can, of course, do all sorts of things, but I'm not going to. I'm going to pop open a JavaScript console. And I can just run things on the local system. So I'm going to run cat etsy shadow. In this case, it responded with a failure. Let's do something a little simpler because I don't have access. And I'll talk about that later. But let's run a ping command. We can see that it's pinging the system. And this is actually a real process running on the system. The JavaScript code is interacting with the system as you would from a shell script or from other tooling. So here's another example that's going on where we're accessing divas properties from JavaScript, where, in this case, printing out the pretty host name from system D and changing the host name, we can just run these commands directly. And all of the cockpit UI is implemented in these terms, the fact that it's really running a Linux session in the system. In the talk that we had last year here, we showed how it was a PAM session, an SA Linux session, an auditing session, all of those things. So we can access files. We can do all of these kinds of things from this session. Let's just see. Here we go. We're actually in the OS release file. There you go. So the first question we have, the first variable that we're going to place in our algorithm there, is the web browser. Is it really plausible with all the vulnerabilities that web browsers have to run a Linux session in your web browser? So that goes into the vulnerability column of the threat vulnerability impact. It turns out that browsers are a lot, a lot of code. This is the code for Firefox. You can see just in C++, there's over 15 million lines of code up there at the top. That is a lot of code to make secure, a lot of code that could have a security impact. If you look at this link here, you can see the security vulnerabilities for Firefox. And there's a lot of them in a lot of different versions. Nevertheless, it's really hard to make this claim. Imagine in your security, you wanted to make this claim. No browsers were involved in the deployment of this system, much like no animals were involved in the creation of this film. So we have a lot of config management systems. We have Puppet, we have Foreman, we have Satellite, we have Landscape, we have all of these tower, all of these different systems run in web browsers and essentially let you execute code on the target system. That means if we were to make the claim that browsers are an unacceptable vulnerability for our systems, we would have to exclude all of those things. So when you're calculating the risk profile for involving browsers and accessing a Linux session on your system, make sure to account for that. In addition, this is even harder to make this claim about your systems. No browsers were involved in the building of this system. Every Linux distribution that we looked at today, or I've heard about here, sorry, today, has browsers involved in the creation of that, of the packages in there, of the config in there in some way. If you could subvert the browsers, subvert a browser and use a vulnerability there, you would have access to place content in RPMs. So it's really important that browsers are secure. And nevertheless, we cannot dismiss them out of hand since they are so bound into our ecosystem today. And if we are to solve this problem, we have to solve it as a community rather than just targeting a project like Cockpit. So actually I don't have time to get into this topic so much. We have a lot of stuff to go through today, but within the browser, there's a security policy. It's called content security policy, and it's much like SC Linux running in your browser. That only specific things that you allow the page that the policy describes the page can do can be done. Specific URLs can be accessed or objects could be loaded or where the JavaScript comes from that is executed and so on. This is an interesting topic, but we'll have to skip it. But it's one of the mitigations that Cockpit uses a lot. It locks down the security policy of the code that's loaded that powers that Linux session to make sure that the code comes from the system and only accesses that system doesn't leak information anywhere else, so on. But let's skip over that. Here's a fundamental angle. When we go away from browsers and we talk about the fact that Cockpit has, it has no special privileges on the system. When you log into Cockpit, you're logging into a Linux session just like when you're logging into an SSH session. You log in and the, where the actual commands that you run inside of Cockpit are running as your user. Cockpit is not somehow proxiesing, sort of making access or privileged decisions based on who it thinks you are. The Linux system itself is making those decisions. So let's look at this in live. We'll go into a terminal inside of Cockpit. And much like you saw, if we run ID inside of here, we're running as stuff, because I logged in as stuff. When I log into Cockpit here, I'm logging in with my username and password, just like logging into the local system. Here at the bottom, I'm gonna talk about this checkbox later, so I'm gonna leave it unchecked for now. When I go in here, again, you see I just have, I'm logged in as stuff. This is my expected SE Linux context, and that's the group that I'm logged in as. And when I try to access something that I don't have access to, I'm just locked out as if I would do this from an SSH terminal. And the same applies to, of course, the Cockpit code itself. Here it says the user stuff is not permitted to modify host names. There's various access permissions like that. And like you saw before, when I tried to access, let's see, Shadow, just from JavaScript, it wouldn't let me. Let's get a clearer message here by putting that little argument on. Basically here, permission denied. So, yeah, we showed this. So how does Cockpit actually then work on the system? It would be pretty useless if you couldn't configure a system from Cockpit given that it's supposed to do those tasks, right? Well, it doesn't exactly the same way you do in a terminal. It escalates provisions via pseudo or via policy kit. And that's what this little checkbox here is about. It would be kind of silly if you had to log into Cockpit and then immediately type your password again in order to do the thing that you logged into Cockpit to do. So we make this checkbox available to you. And if you check this checkbox, it'll log into pseudo for you. Essentially, if you can, if that username and password is allowed to escalate privileges, it will do so. So here's a good example. Let me log out. So if I check this checkbox and then I go to the system page, I can start to do things. Look, I can change the hosting and so on. And in fact, if you look here at the terminal, you'll see I'm still logged in as stuff. This terminal hasn't escalated privileges, but if I go and use policy kit, for example, I can escalate whereas before I couldn't. Again, if I now, if I go to the terminal, oh, let's go, let's use the, I think I might be running out of time, so I'm gonna use the display here. But if I try to exit the shadow pass, the shadow file, I get access denied. But then if I, when I'm logged in like this and I use the super user true argument, which basically says use pseudo to do this command, I get access to the file. You can track that state up in the corner here of Cockpit. It says that I'm privileged here. This is an old screenshot, right? Now it's changed, but it says that I'm privileged. And if I log, if I don't log in that way, or if I log in using account that can't be privileged, I won't have that here. And again, if I, now that I didn't check that, this won't work. There we go. Let's turn that off. All right, exactly. It works. All right, so, in addition to all, in addition to that, there's no big fat demon that has all these privileges, somehow dispenses them to different people and makes decisions in any way, shape or form. Cockpit is very separated as far as each piece doing one job and having the right security for missions to do that job. When I log into Cockpit, I see that there's various processes started and they scroll off the screen. That's pretty, pretty dumb. Okay, let's do it from the terminal then. So, I'm logged into Cockpit here and let's make this bigger. Are we on screen? Yep. And we need to, just a sec. We need to make this terminal the right size. Well, why don't I do it from in here? Ha ha, Cockpit. So, you can see that there's various pieces of Cockpit running. Here, there's Cockpit.ws. Cockpit.ws is the thing that's listening and talking to your, listening to the, on the HTTP socket, talking to your web browser. And it's running in a SE Linux session that's very, sorry, SE Linux context, that's very restricted, and it's running as a specific user. The user is actually called Cockpit-ws, but somehow PS insists that it's still DOS and doesn't display more than eight characters here. Once you log in, and it can't really do any, you can't do any authentication, it can't do those kinds of things. It doesn't have access to any of the, of the password files, it doesn't have access to run PAM or any of that kind of stuff. So, what it does is it spawns a process called Cockpit Session, which is here. Cockpit Session runs again in its special SE Linux context, and it is privileged to do these things. It's a very small C program that loads PAM, which is actually a non-secret or small C program that somehow has a benefit that loads PAM, which is like millions of things, but that's the state of Linux today. And it authenticates the user, and when it's satisfied that PAM has told it everything's worked out, it launches another process called the Cockpit Bridge. Cockpit Bridge is launched as a login session, much like Bash, so this, in this case, it's unconfined. It's running as me, not as root, and this is how the interaction with the system happens. Cockpit Bridge interacts with the system as me, and none of these other processes are involved in the actual interaction with the system. If I log in, again, with privileged access, you'll see an additional process where there's a bridge that's privileged, and you can see that it's running as root. And that's, Pseudo has run that bridge for us, and because I have access to escalate privileges. And again, there's all sorts of different measures here. This particular file can only be executed by the user that's running this, and so on, so we make sure that as many things are tied off as possible. All right, so what about listening another port open on your network? Cockpit, by default, listens on port 9090. 9090 is the WebSM or Web Systems Manager port. This was in existence long before Cockpit existed, and, but in order to access it from your web browser, you obviously have to open that port, or do you? You can actually use Cockpit over SSH. It would be a pretty cool, amazing party trick to actually make your browser talk to SSH, maybe one day, but not today. What we do, instead, is we have your web browser, talk to CockpitWS, on one machine, machine A. Oftentimes this is your local machine, or bastion host, or something like that. And it can talk via SSH to machine B and C. And obviously there, you see the Cockpit bridge running as your user, and so on. The same stuff happens. We're really, instead of CockpitWS via Cockpit session launching the bridge, we're using CockpitWS via SSHD to launch the bridge somewhere else. And in other talks, we've talked about the protocol that these things talk. It's a very simple standard in, standard out protocol. And for that reason, it's really easy to nest and extend this over SSH. And when you log in, you can just click down on the login page, where are we? And you can, let's find another server to connect to. So really even, you don't even have to log into the local system to do this. In addition, when I log in on the local system, I can go to a dashboard and add other systems here. So I can have them show up here, and I can click between them, and so on. And it's just doing all of that via SSH. Oh, I can just show you on my local system. So I'll add local host, sorry that I don't have a VM or anything running. Oh, it understands that I'm local host. Let me try via, let's see if that works. I think I have a video of this. So let's look at the video. Here we are, and the video doesn't display because it's too big, is that better? There we go. So I'm adding another system, and it's SSH, so it's showing me the fingerprints since this is the first time I'm accessing it. And there it is in the list, and I can just click over to it, so on. The display is slightly different because there's a much nicer UI design now in Cockpit, but the implementation is basically the same. Thank you. We have five more minutes. All right. So lastly, what about the web server? Is this like a big Apache stack or Tomcat running in order to power all of this? No, it's not. There's Cockpit WS. It's a small socket-activated process written in C with only enough HTTP to load Cockpit into your browser. It doesn't even support post requests, just get requests. Now, so some of the security characteristics of how do you secure, let's say Nginx, or how do you secure other web servers don't apply, different ones apply here, and it's important to bring those into your risk matrix, your calculations there. Again, if you're uncomfortable with this, or you need certain systems to not have this process running, then it's simple to use Cockpit over either, over SSH, or you can proxy it behind another web server that you feel more comfortable with, or you already have security worked out for it, and here's an example at that link of how to proxy Cockpit over Nginx, but people have done this with Apache, I think 2.4 and later, and so this is a possible solution to mitigate the security risk. I don't, there's certain older web servers that don't support proxy and Cockpit because Cockpit uses web sockets and they don't know how to do that, but in general, modern ones do. So, and there's other aspects to Cockpit security, but we cannot get into them. It can do Kerberos authentication, for example, the whole way it uses certificates, and how you place your own certificates in there, how you can plug your own authentication instead of using PAM and various other things like this, but we don't have time for those today. So, and lastly, we are looking for someone to join the Cockpit team, so if you're interested in this project and you've played with it, come and reach out to Lars and it'd be cool to talk. Any questions? Yeah? Since the process is running using an unprivileged port, have you considered one local user attacking another administrator by opening a fake Cockpit instance on this port? Interesting. And so the question, oh, we already have it there. That is an interesting attack angle. I believe one of the principles that we follow in Cockpit is we are neither more nor less secure as Linux in general, and previously we tried to make ourselves more secure than Linux in general, it didn't work out. So to answer that question, you can do that already by faking pseudo inside of a local user session by faking the screen in X, by showing something that looks like a terminal, one process could do that for another. It's a slightly different vulnerability profile because of different users, and Penny wants to answer as well. So let's get the mic to him. Yeah, so that question actually has been discussed on the Futuramening list, particularly when we were discussing enabling it by default, and there was a proposal to just use SLInux to only allow the user bin Cockpit WS process to access that port. Hasn't been implemented, but it would be possible. Right, okay, yeah, that makes sense. There are other security vulnerabilities such as we, in order to use pseudo, we pass the user's password to pseudo while we're logging in, and previously we tried to avoid that, for example, but we realized that's what everyone does on their terminals anyway, they pass their own password through their user session, and so it's another example of where we realized, hey, this is how SLInux security works, let's reflect it, and let's all fix it together rather than try and just change it better just for Cockpit. So this is class of vulnerabilities in that area that are worth thinking about in that way. Any other questions? Yeah, why don't you have a turn and then go back. So I was wondering if within the web interface if you can, almost like using a bash history, and to replay commands, if you can then go from machine to machine from within the cockpit front end and kind of have a history of what commands you ran and then go to the next machine and replay them, or would you just, because you have full access to JavaScript, so would you just expect someone to write a program to do that, for example, if that's what they wanted to do? And a lot of people ask for this, basically the ability to take what you did on one machine and then make it happen all over. So what we don't want to do is turn cockpit into configuration management because there's plenty of those and people like the whole space is strewn with bodies and it's just like all sorts of stuff going on. But one of the things we're trying is in certain parts of cockpit you can extract a script or an ansible script to then use to then deploy that to a bunch of systems. So helping people get involved in the right way to do that rather than inventing another one. But do you have a security question back there? Yes. Let's switch back to security. So have you, I mean, just an idea that instead of passing the password around, why don't you install a PAM module in the PAM configuration that would just authorize actions if there is this flag set by the cockpit login program that is privileged. Yeah. And very good. So we did that up until like for the first two, three years of cockpit we did exactly that and it involved the kernel key ring and it involved passing tokens. It involved inserting PAM modules into stack and screwing around with SSH and all sorts of things. And it turned out it was incredibly hard to both explain and validate the security of complex. And really we need to fix this more broadly. We need to fix it on our own terminals. We need to fix it everywhere in Linux. And so we decided to just use the, like go back to the security we already have in Linux rather than trying to marginally increase it. And I'd love to, we can discuss whether that was the right call or not, but we definitely tried both of those options and we had for a long time that code sitting around. It's still in the get history. Maybe we can bring it back one day. Let's see. Any other questions? All right, cool.