 All right, let's get started. Welcome to SL Linux Fun with MySQL and friends. As you can see, I'm missing a co-speaker. Yvonne unfortunately couldn't make it today. He's in South America, and he couldn't get approval to come here. But he did help me prepare these slides, so we left his name on there. So quickly, who we are. My name is Matias Krawals. I live here in Belgium. I live in Ghent. I've been a Bachelor in Computer Science, been a Linux user, and I've been for over 20 years. Then I've been a PHP developer in the start of my career for about 10 years. And since 80 years, I'm MySQL DBA. Currently, I'm a Lead Database Consultant at Pythian. Yvonne lives in Argentina in South America as a Systems Engineer. Unfortunately, he also has left Pythian. He's now a Senior Consultant at Percona. This is something about Pythian, yeah. One disclaimer. I'm not going to claim here that I'm the SL Linux guru. I'm just a DBA having to deal with SL Linux on systems that are managed by other people. And this presentation goes about how we go about that, and what we did, and what we learned about dealing with those kind of situations. So let's get started with some introduction to what SL Linux is. So this is the definition you can find on Wikipedia. It's something because someone thought that the normal Linux security system wasn't good enough. So we actually have the privileges for user groups and others. But it wasn't granular enough. It's originally developed by the NSA in America and Red Hat. And it's distributed to your kernel as a set of kernel modules for enhanced security or for bothering DBAs. There are three modes to SL Linux. And it's very similar to what Nick earlier said in his presentation about proxy SQL. The default mode is set to enforcing in Red Hat's Enterprise Linux and Fanto S. You can set it to permissive or disabled. Permissive does what it says. It will track all the settings for SL Linux, but it will allow them, but it will log them in the log file. And disabled will just completely disable SL Linux. And a wise man once said, every time you disable SL Linux, a kitten dies. Think of the kittens. The truth is mostly the compliance or security teams will bite you if you disable it. It's often a part of their audit trails. They want the logging at least, or they want to block everything they don't allow, which is also the default policy. So there's a deny policy for anything you don't specifically allow. A useful tool on Red Hat Enterprise Linux-based system or CentOS-based system is policy core-use tools in Python. It provides a lot of tools to manage and define your SL Linux policies if you want to go a little bit deeper and really start writing policies. The devil package gives you a lot more tools also. And how you can check it? So there's a tool, SL status, and you can run that. And it will show you the current modes. Like in this case, it was enforcing, it's enabled, and these are the config directories. You can get enforced to see what the status is of the enforcement. And remember the kittens here, because I made this screenshot. I killed the kitten. So SL Linux is defined by users policy context. There's another one-on-one mapping between Linux users and SL Linux users. So the SL Linux users, by default here in this system, you can get by sa-user-l, and then you get all the SL Linux users. You have guests, routes, stuff, a regular user. And you can then assign a login. So in this case, I assigned user John to my S-user, S-Linux user. And then you can see that this John user is there. And the default is that they are unconfined. So it's very straightforward, very easy to track or something. And also S-Linux adds a dash capital Z option to commands like LS or PS. So you can see what SL Linux users and roles and types it uses. And also for a process, you can see MySQLD is running as the type MySQLD underscore t. And MySQLD save is running as SQLD save underscore t. And so the context are defined as user role, type, and then a level. The most user that I have seen from it is we're using user and type. A role is usually like a system object or system R or something like that. A level is if you want to go even more granular, you can start doing like this user has access level to these SL Linux levels and you can go up to more and more levels. But that takes us a bit too far if we. So MySQL and SL Linux out of the box experience is that so that everything works. So if you just install, you have install MySQL server, which gives you MariaDB server. Or if you do MySQL community server, everything will work. So there's a module, a policy predefined. You can do the SA manage module L and then you can grab for MySQL. So you see that the MySQL object is there by default. And it's very granular. So you have a lot of different types that are defined for different locations. So in the Etsy MySQL, it's MySQLD Etsy. You have the log directory, the data directory has its own. The MySQLD process has its own context and everything. So it's very granularly defined and you can then start playing with it. So in this case, in this example, we wanted to change the data directory. So the default is far like MySQL and somehow we were requested to put it on data MySQL. And so we made the directory, we gave it the permissions. And you can see here by default, it's unconfined. And we started MariaDB service and it didn't start. So what we used to do was turn the selenix off and then it worked. But remember the kittens. So there's an audit log. And if you install that Python package that I referred to, you can do audit to allow and w-w-a will give you the list of things it's denied for you. So in this case, it says, I denied write to the location. I tried writing with MySQLD context and you are writing to default D so it's not allowed. And it's true because this is by default default type. And so you cannot write to that with selenix enforced. So we were then finding the right data directory context. So on varlib MySQL, this is the default context. So we were trying to just set the context to the directory context. And then we did it now as again, and hey, wait, we changed this, but it didn't change anything. So you still have to apply that if you change that. Something we learned. If you do that restore.con and dash v gives you useful output. So it shows you what it's actually changing. Then it will start changing the types. And now you can see that it has the right type. And if you now start to server or get the status, you can see that it's active running with the right data directory as you were expecting to. So this is one way of dealing with it. Another thing is custom ports. So we had a running MySQL instance and we just wanted to change this from 3.2.6 to 3.2.7. So what we did, added sports equals 3.2.7 to the MySQLD section of the config, restarted and boom, it didn't start. Why didn't it start? So you check journal CTL, it says it doesn't start, it has a failure, but it doesn't say why. The error log says permission denied. Do you have another process running on that port? No, we don't. So obviously as the Linux was bothering us there. So again, if you do the audit log, you can see that it is trying to use an unreserved port and it gives you a possible solution. So it says you allow NIS enabled, and NIS enabled will allow a process to bind to any port. And so if you set that as say Linux Boolean, it starts and it binds to the port, it works. But the compliance team may or may not like that level of freedom to give it to you. So another way you can go about it is as the Linux manages the port. So if you do, if you grab the MySQL port from port dash L, in essay manage, you see that it has a number of ports predefined, three to a six is part of it. And you can just add three to a seven to that list by doing essay manage port dash A for add to the type MySQL D, the T port T, and then you give it TCP three to a seven and then it adds it to that list and then you can just restart the database server and now it starts with Portrait Viewer seven. So that is just examples of things we run into while dealing with it and how we try to work around it without getting the compliance teams on our necks or getting us in trouble for disabling things or changing Booleans and all those kind of things. So another friend of MySQL is Proxy SQL. Nick just presented about 2.0 and we also use Proxy SQL quite often and also by default, it works. So if you have, as a Linux state that's enabled and forcing everything, it works, it runs, but it's running as an unconfined service and it's running with the default privileges for VARLIP T, where the default type of directory for Proxy SQL is. So why should we bother? It works out of the box. In our case, the log rotation failed. So we have log rotate rotating the Proxy SQL logs and if I run this as root, it worked. I defined my Proxy SQL log rotate config and I tested it and everything worked until Krone started running it. Krone runs and it says it starts, I start log rotate Proxy SQL and it exited abnormally with status one. So why? If we do the, this default system has, it's mailing to VARLOG mail root and so it says error renaming my .2 log file to .3. Why? It says permission denied. If we try it as root, it works, but if we try it in KroneTap, it doesn't. So it has root, we can just run the log rotate, we can get all the output, it does everything, everything works, but Krone doesn't let us. So if we go back to this audit log, we see it tries to write this file, but it tries to name this file, so it's not allowed. So in this case, you can use also the audit to allow tool to generate a policy for that. So if you do audit to allow on the log file, so here is the same command we did for that audit log. This just prints the output on the screen. If you do dash and Proxy SQL into a file, it generates this file for you. The problem is in this case, it only generated for the one error it saw. So it saw one error, the rename, but there's multiple things that log rotate will do, it will create new files, it will rename files, it will delete files if they are expired. So it needs more privileges. So these are the commands you can do then to compile this .te file into an actual module file that you can load and package before it gets loaded into the kernel. So at the end, you have the module loaded, version 1.0, which is the version we defined here. So if you define a different version there, it gets a different version number down here. So as I said, this only just allows for a rename. So more iterations of the same process are required to make this work. An easier way to get to this is to set SLS temporarily to permissive. So you can see what it will do, it will allow, it will log everything, and then you can run the process. Obviously, it doesn't do all the operations it might do in a single run. So in my case, I was only at my second or third iteration and I wanted to keep five log files. So it only, when it reaches the fifth log files, it will start deleting. So you have to run, make sure that you have the entire process that it will do. No. Oh yeah, the question was, can you set SLS permissive only for a specific process? The answer is no. Not as far as I know, but. So if you then have the entire list of things it can do in the audit log, then you can generate the full file that you end up with, and then it looks something like this. So it has access to those types. So the varlip, dev, log rotate, unreserved ports, tcp connects, and then all those file actions. So it needs access to the admin port for ProxySQL in order to flush the logs, because otherwise ProxySQL would just keep writing to the old log file. And it needs all those privileges on the varlip ProxySQL directory to make sure that it can rotate the file. In the end, it was a great success. We did have no more errors. So our log rotates in chron was running fine. We started rotating and everything started working. Thank you Dave. Is this the best solution? Probably not. What would be a better solution, I think, would start to define ProxySQL's own SELinux policy. So the varlip ProxySQL directory gets its own policy, gets the right privileges for log rotate. But as I said, I'm not the big SELinux expert. So I didn't get that far yet. Maybe in some next project I will have to and maybe next year I can come demonstrate how you generate the SELinux policy for ProxySQL. But in this case, this was enough for us. So we got log rotate to rotate. And in this case now it has permission to do so on all the files in varlip, which is probably not what you want. But for this project, it was okay. Alternatives for SELinux, most well known is probably AppArmor, which is the default in Ubuntu-based systems, I think. And SELinux. And then the key difference is this, yeah, you can see here, SELinux supports a remote policy server. If you define it centrally, then all the servers get it, which AppArmor doesn't. And then there's some other tools which are less known, but when looking up alternatives, I found them, so I put them here. So, any questions? You said that SELinux has some good defaults for my SELinux. Not yet, I think. Not yet, I think. So does the default policy include a port for MySQL X? I don't think it does yet. It might, and there might be a feature request for that, but I don't know. And I think the policy is not managed by Oracle itself or by MariaDB, but I think it's managed by the kernel. Yeah, it's got to be a surprise if you... Sure. Show me. So the question was, when you specified the port to access, is there an option to include SELinux? I don't know. So the question was, can you export the database after you set SELinux? I think you can, it's just exported to anywhere you want. I think as a route, you will be able to write anywhere. So, okay, thank you.