 I think it sounds quite the same but... I think it's better. And if they don't buy it, it's best practice. It's one of the best. The print is really... It's a little bit smaller. It's a little bit smaller. It's a little bit smaller. It's a little bit smaller. Ah, ok. The other print is quite the same issue. It's a little bit bigger. It's a little bit smaller. It's a little bit smaller. It's a little bit bigger. It's a little bit smaller. It's a little bit smaller. Right. Yeah, 4-3. Hello. My name is Milosh Malik and I've worked with Redhead for several years. And I would like you to... I would like to introduce you to something I call the BIC, a Selenium Stable Shooting Chart. Please, if there are some problems with the resolution of this display, it's better for you when you open the chart. This one. And it's located on this address. If you want to follow me on the slides, the slides are available at this address. Both the picture and the slides are available through Dropbox. I hope that nobody has something against Dropbox. So, why this presentation? Why this workshop? It seems through the years that a lot of Selenux bugs have the same cause. If I take the amount of bugs which are reported each year, they can be divided into few groups and the groups can be solved quite easily. Some of them not. And we will discuss, I hope, the majority of bugs and how to solve them, how to avoid filing bugs in easy cases. Basically, this workshop is oriented to a Selenux policy. If you, to avoid your confusion or if you are interested in a Selenux, the parts of kernel, this workshop is not about it. This workshop is just about a Selenux policy, major problems which are filed as Basilas, and there is, I will present you, a workflow, something like a mind process, how to go through these problems and how to solve them. It also seems that a Selenux documentation lacks some important info because people still think that troubleshooting of a Selenux policy problems is difficult. Some of the issues are difficult to solve, but I think that majority of them are easy to solve or maybe with help of our Selenux policy developers, they are solvable in a short time. Who is the targeted audience in this case? Usually system administrators, developers, quality assurance, and also people working in global support, which means with customers. What I would like you to get from this workshop is that you will be more confident when solving a Selenux issues. I hope that your troubleshooting skills will improve and what I will present here is something which I use for several years and I think that it's a big help even for skilled system administrators. What's the structure of our workshop? Fortunately, I found out that this workshop is twice as long as I thought, so we will have more time for the discussion in the end. Before we get to the discussion, we will go through the chart. I will talk about the chart in parts then about the whole chart. I would like you to ask questions immediately. Once you find something confusing or you want to get some answers, please ask immediately. Then I would like to show you that the chart is worth while working with. We will apply the flow, which is described in the chart on some bugs. I hope we will find solutions. I prepared a list of bugs, which are definitely solvable. I would like you to pick a favorite back of your something you reported either through Baxilla or a customer portal or you just see on your computer and we will apply the chart on the bug you propose. It is some kind of proof, I'm confident that we will prove the chart usable. I see that a lot of you have your notebooks. If you are watching the chart right now it may be confusing or too difficult but basically the chart has several nodes and the nodes are identified by numbers and colors and the colors mean in which part of the troubleshooting process we are. There are some colors, like light brown, we are identifying the problem, then these orange ones, we are analyzing the problem and then there are the blue ones which mean conservative solutions then the red ones. These are radical solutions and basically we would like to go to the green one which means that we solve the problem. The yellow ones we need some kind of work around. So the first part is problem identification. First we need to find out if the problem PC is really a selenux related. It usually means there can be different kinds of problems and a selenux problems usually manifest themselves by an AVC, a selenux denial. Either it is a denial in the user level or a user level on the kernel level and so on. For finding a selenux denial the best tool I think is whole search but it also means that the audit demon must be running. If audit demon is not running then the AVCs end up in log files or in journal, we have to find them or better way is to start the audit demon and repeat our scenario which caused a selenux denial. Sometimes who of you are familiar with don't audit rules? So these rules are a bit tricky because your scenario may not work and you still don't see AVCs because there are or maybe there are some don't audit rules which hide it. This problem is again illustrated in the chart and we will get to that. Other possibility when you run X-Session applications you can of course use a SETrubble should apply. I think that you know that. It's a small icon. It usually pops up when there is an AVC. Once we are sure that the problem is a selenux related then we can go to the next part of the troubleshooting process which means we have to analyze the selenux denial and find out where is the problem. If the process who caused the AVC is running in correct context or if the object which was accessed has wrong context or is there a rule missing that could all of these causes are quite common because a selenux policy in comparison let's say we have a program and the program once the developer adds a new feature to the program a selenux policy usually does not know about it. So once the program is executed in a way that the feature is started there may be AVCs. This is okay and again it is worth of reporting because without these AVCs a selenux policy developers basically they cannot know that a program X was improved by a feature Y. So there may be other problems in a selenux policy for example you know file context patterns. These patterns basically they match a path a regular expression with a context. Once these patterns are wrong you from time to time can encounter a problem like yeah, AVC and the AVC says there is a context and a selenux policy expects other context. This is also quite common and again can be solved and it's part of the chart. You should always ask why it happened and if you see a selenux denial do you see the consequences of some forbidden access or do you see the root cause? It's always better to solve the root cause because the consequences there can be many consequences and if you try to solve consequences you usually end up in the same situation another time when the AVC appears again because the scenario triggered the problem again. So once we analyze the problem we should take into account conservative solutions. Some of you are familiar with local policy modules. Who of you knows about audit to allow creating a policy module? So, please try conservative solutions first before you end up with a local policy module. Because local policy module you can do a lot of things with local policy modules but you can also make policy on your system more benevolent which means you are basically you are weakening your system policy instead of making a scenario work. So, what are these conservative solutions? First, there are some best practices how to do stuff for example running services running demons. In like a few years ago people were used to run demons just by init script like I mean slash atc slash initd slash for example SNMPD space and start. This makes the new process does not run in the correct context. So, I advise you to follow the best practices and in this case of services and demons please use the tool service or system CTL which was introduced in system B of course. Other conservative solutions are Booleans. Booleans are designed to switch on or switch off a set of rules and there are more than 250 maybe 300 Booleans right now so a lot of possible combinations are already I must admit poorly documented but are available. Other conservative solutions they use an SEManage tool who of you used at least once SEManage? Okay, so this tool gives you some opportunities how to solve problems and I consider SEManage the conservative solution. So SEManage enables you to redefine a network port redefine file context patterns and also to make a domain which was before enforcing now make it permissive. This kind of work or this command allows you to make just one domain permissive while the rest of the system remains in enforcing mode. If you know the setEnforce command I think that you don't need to run it with zero anymore and also remember that each execution of setEnforce zero makes then worship. Okay, next there are radical solutions. These radical solutions usually involve local policy modules and tools like audit to allow which is able to generate policy module on demand. It usually does create rules you can also define types in policy module and of course you can also change rules and other stuff which needs some advanced reading about it. I recommend Denmark's blog and I have to repeat it again that please consider conservative solutions first. It may happen that even the radical solutions are not usable meaning they does not help you solve the problem. In this case you need some work around and of course you should file the problem, report the problem either through our customer portal or through Red Hat Buxila depending on what's your favorite kind of party 14. Yeah, I have to say that even if I talk a lot about Assyllium's policy the fact that you see an Assyllium's denial does not mean that the problem is always in Assyllium's policy. Sometimes the problem is in the application who triggered the Assyllium's denial or sometimes it's the problem in kernel. But this is out of topic for the chart right now and I recommend contacting our developers through bug reports or on IRC channels and so on. So before yeah, I think that the chart is explanatory but you usually need two iterations to go before you find some kind of solution. Why two iterations? Because many of you know that there is difference between enforcing and permissive mode. In permissive mode when you imagine a running program permissive mode means that the program is allowed to do whatever it wants but in enforcing mode some of those actions will be denied when the Assyllium's policy does not expect such behavior. It means that in both modes you get a different list of Assyllium's denials. It usually means that in both modes you get a different list of Assyllium's denials. That's why we need two iterations. I have to admit that not all Assyllium's denials that you see are really necessary to fix. Some of them are redundant. The program works fine even if there are additional Assyllium's denials. And the rest of the denials are the excess which trigger the Assyllium's denials if necessary. In that case that must be fixed in Assyllium's policy. Do you have questions right now? If you have questions please ask right now and then we will go through a list of bugs which I preferred. Will you be talking more about the Assyllium's domain? I could stop because the workshop is longer than I thought. In that case I have an example and we can show it's too small right? I think that yesterday a guy called Akien filed a bug which is related to prosody. My knowledge about prosody is very shallow but I know how to find Assyllium's denials and if you present if you give me the scenario I can repeat it. I basically did what Akien this is the bug report. Akien filed a bug where he speaks about HTTP versus HTTPS version of the prosody demon has some kind of embedded HTTP server inside and the HTTP server can serve HTTP and HTTPS requests and what he found out that there was an AVC or two AVCs on his machine and the AVCs basically tell us that a process running in this context prosody underscore T wanted to bind it wanted to bind to a port 5281 on TCP because Assyllium's policy nobody before tried that or didn't before that Assyllium's policy does not have such a rule which would allow it but we can for example of course we can solve this problem this is one of the most frequent ones and basically what we need is here there is a nice tool one of the most useful ones and the tool says that this port 5281 the only Assyllium's label for this port is unreserved port T if the port is a default port for the application there should be a better labeling than just unreserved port T let's look at the other port which Etienne wrote about and we see that port 5288 is called also a jabber inter-server port T so it seems that the HTTP connections are okay and HTTPS connections are not so I prepared basically what you see here is a virtual machine with rel 7.2 and basically we try to first you see that we have enforcing mode and we also have there is a tricky boolean called NIST Enabled it was before it was called LLIP bind and this boolean is one of the mighty ones that enables a lot of applications to bind to a lot of ports but our scenario does not have anything to do with NIST or yellow pages so this boolean should be off and let's check the status of the service so the service is stopped right now we change the configuration in that way Etienne proposed he basically oh it's visible, sorry just VI so basically Etienne advised us to enable HTTP so we do that now let's say first we want to see if there are any AVCs or user AVCs or in the last let's say 10 minutes so there are none let's start the service the service starts it seems but when I run the all search again I see these two basically they are the same they just differ by the time but I see this a series denied as you can see that's the same Etienne was reporting and again here is the port and the operation name now we have if you went through the chart on your own then you would see basically two solutions either you can redefine the port number or you can create a policy module I think that we should follow the creative solution first which means to the number 5281 to the definition of Jeber server port which was the same port as which works basically so but I'm running the system in an enforcement mode and I'm not sure if I see all of these services maybe there is some operation which is denied but the program because the demon was not able to bind to that port he gave up and we don't see the other accesses so now let's come to the permissive domains basically in rail 7 there are several permissive domains these ones so meanwhile the rest of the system is running in enforcing mode these processes running under these contexts run as if the system was in permissive mode so they are allowed to do anything so now we can we switch the prosody domain to permissive and we will check if there are additional AVCs which need to be solved so once the command succeeds we will see that the prosody domain was added to the list of permissive domains so you can see here that these domains are defined in a Selenux policy as permissive and these domains are defined as local customizations so will it stay after the reboot? yeah they will survive so now let's repeat the scenario so let's stop the service so these are the two AVCs we saw before let's start the service again and there are more AVCs so this user AVC is basically the execution of Selenux permissive so we changed the policy in such a way that prosodyty from now on is a permissive domain and this is basically the change and here we started the demon again and because the demon before it was running as enforcing domain we can see success no and exit code minus 13 now the domain is a permissive one so we see success yes, exit code 0 and we are pretty sure right now that there are no additional AVCs which need to be solved so other possibility basically this helped us to find out which rules are needed and if the set of rules is complete I would say and from now on we can either create a policy module allowing that kind of access or redefine the port definition let's go for the conservative solution so here is this is port 5,280 which is allowed and we need to add the number 5,281 basically we do that by we use again a Siemenage command and we instruct the policy to add another number to the port definition so right now you can see that the Jeber inter server port T is defined as these three port numbers so from now on we can the scenario should not generate any AVCs so let's make the domain enforcing again so again we use a Siemenage permissive command but now we use minus D and minus D means that we remove the prosody underscore T from the set of permissive domains so after this command the prosody domain is enforcing again it takes time but it will be faster in Rails 7.3 because a lot of optimizations were done by a Selenux developers those just who left ok so the port number is redefined and we will rerun the scenario again we are in enforcing mode the prosody domain is enforcing also the prosody domain is not in the list of permissive domains anymore so we will stop the service let's see there are as you can see there are another two user AVCs one of them is we switched the prosody domain to enforcing again and the second one is we redefined the port number so let's start the server it's running and is there another AVC it's not so this we basically solved the efficiency in a Selenux policy right now and we used the right solution and we just need to wait until a Selenux policy developers make the port 5281 a default port in the voice till that time let's say there is an example I installed some other package which also uses this port and how it's solved then like it's my package or the new package so how to solve this okay may I ask which package it is it's just an engineering okay so either let's see what's set in policy policy basically expects that so basically a Selenux policy expects that the prosody server is able to bind to port sorry so in this case we see that there is an allowable if you install some other application which will use the same port so you have to check if there are other rules which allow named bind to that port of course we can use again this SE search tool is very handy and I cannot imagine my life without it I mean my professional life okay so you can see that there is about 39 rules which allow different processes to bind to a port which is defined as Jabber inter-server port T so but it may happen your application is not able to bind to that port and you are not able to find a proper context for the application in that case you would need a local policy module only one not policy it's no as you can see here that the port number 5,280 is basically in two intervals like first there is this one interval present in just one number and there is another interval so yes the number the port the number of the port can be from Seljuk's point of view visible under different names but it brings some problems in the future but a Seljuk's policy defines some or these ports so yeah it's common so it's better to allow the application just to use Jabber inter-server port than creating new label for the port yes but these network ports the name basically these are you cannot define a new port in a local policy module it's one of the constraints you can define a port name in policy but in local policy module you can just use it or via SEManage you can add or remove the numbers that's one of the limitations local policy modules are a mighty tool but they you cannot do everything with them so are there any questions so let's go to the list I think that Zdenek sits here lately filed a bug which basically demonstrates the problem that S-Tunnel, who of you know S-Tunnel or use S-Tunnel so it is it is how to say it a way how to secure a connection using NSSL and the scenario which was at the beginning is that S-Tunnel was not able to write any messages it means that either nobody used it before or used it but didn't report it or they were running the system without a syrinx a syrinx in permissive mode and they didn't bother filing the reports so I again created a small reproducer for this but before we get to the reproducer you can see that the ABC again it says that S-Tunnel is not able and here is operation search search operation usually means that you are trying to find a file by its name and it's not S-Tunnel is not able to find a file and in this case before you can write to a log file you should be able to find that log file or create it so the bug report after I used the scenario mentioned by Zdeniek I executed the same scenario but after switching the S-Tunnel domain to permissive so I get other a syrinx denials and now when we see the a syrinx denials that there is something interesting that it is always the ABCs are always talking about var underscore log underscore t that seems strange because usually if there is a log file I mean not log for logging basically logging messages then such files for such directories a special context let's see so this tool matchpadcon tells us what's the expected context on directory slash var slash log let's say that we create a log file for the S-Tunnel and what should be the context of that file and a syrinx policy or matchpadcon tells us that again it's var log t but var log t in the sense of there are some general contexts and some specific contexts and even more specific contexts like directory slash var has a context var t when we go inside and ask for the context of var log we get another context if we try for example what's here pcp again has even more specific context so we expect that the S-Tunnel log file should have more specific context again we can use this extremely useful tool seinfo and we just grab for s-tunnel s-info minus t it means that it prints all defined types and we can see here that here are some defined types but type for a log file is not among them so basically the bug which was reported by s-tunnel tells us two things that even that we don't have a special syrinx type of files created by s-tunnel and then even if we had such a type there would be an ABC because we were not able or the s-tunnel process was not able to get to that file because the process was not able to search through var slash var slash log directory to find it now we have again we can go through the same scenario as before we can switch the s-tunnel to permissive maybe you saw that the s-tunnel t is in the list of permissive domains that's of course my change because when I reproduce the scenario I followed the chart I switched it to s-tunnel s-tunnel is by default not a permissive domain and now once let's say I have a reproducer so I'll be there with me for a while I will run a test case and this test case triggers the ADCs which are mentioned in the bug report as you can see here are a few fails I already added a special part that test case is about the existence of the type we spoke about and which check for the existence of allow rules and the result of the test case should be some ADCs and we will transform those ADCs into a local policy module we will load it and on that time it should work without ADCs so let's see so the test case basically triggered following ADCs and now this procedure is again described in the slides there are more slides than I will present here because we don't have enough time and I think that those slides are for those who are interested because the Cellulink troubleshooting chart has about 40 nodes some of them are self-explanatory some of them are difficult to understand and that's why you can see the description of each node in the slides basically each node has a unique number and in the slides you can find an appropriate slide for the node so we have a list of Cellulink's denials we create a policy module out of it let's see what's in the policy module to generate the policy module a very handy tool is Spodix to allow it basically creates a TE file which is a text file containing the definition of types and rules and then there are some .fc files for file contexts and it also compiles the policy into binary form so if we just compiled this module and loaded it into memory the ADCs would stop appearing let's see so to load a compiled policy module we have a tool called SEModule as you can see here audit to allow advices us how to do that so the system is in enforcing the STUNNEL domain let's see STUNNEL domain so let's remove the STUNNEL domain from the list of permissive domains and let's run the test case again now even if the policy module is not written as it should be but for the purpose of hiding advices or adding allow rules it will do just fine so we started I think that here you can see there are different advices that they are related to snout because they are not okay here are the last last STUNNEL denials and since that time there are just snout denials so our test case passed there are no additional denials it seems to work but there is still something to improve and the improvement should be here in the local policy module because we should define a special type for log files accessible for STUNNEL let's say that we will create we will create a type called STUNNEL and we will define it I think it's this way maybe you wonder why there are two lines this first line speaks about the slash var slash log directory when you create a file in a directory it means that first you have to write to the directory you have to be able to write to the directory to add an i-node and then you can create the file if you cannot write to that directory you cannot create a file inside I made a typo end of the line I think that the problem is when this line is files type I'm using macros and when using macros there are no semicolons but I think counter unknown type this require section basically says that I would like to use definitions which are already in the policy and what else we need we need a type transition rule type transition rule but we will see that no we won't is anybody of you familiar with type transition rules basically type transition rule says that either when a process executes a file what should be the context of the newly created process when a process creates a file what should be the context of the newly created file or newly created directory so these type transition rules basically say what happens when a situation when a process does something so we will create a type transition rule which says that if a process running under a standard T creates a file then the file should get an S standard log T context and where so the last line basically tells us that if a standard wants to create a file in a directory which is labeled while log T then the file would get labeled a standard log T a standard exec T is for a standard T is context of the process a standard exec T is context of the file which is on the file system and you can see basically here I can illustrate it let's say that Estano we will see if Estano is if initD or systemD is able to run Estano it is we can see here that initD is context for systemD or initDemon but in Rails 7 and Federal Ratset is systemD so if systemD starts a process which lying on the file system has this context then the newly created process has this context the policy we created is we are able to compile it it seems that some of it is syntactically correct now let's see if we are able to load it because there can be other problems previously you installed my policy as well the different types of these commands just replaced yes, that's correct if the important thing is that the policy definition file I mean the .te file it has a line and the line says that I am a policy module with this name and with this version if you compile it and you leave the name the same then new module inserted will replace the old one with the same name it's not necessary to remove the module and load it again, we just replace it if I change the version it should replace it also we had the policy module now we can search for the let's search for the rule we added so the module we inserted, added this rule and now let's see if it's enough first I would like to remove the file the file is not there it didn't work because I forgot to add some rule and basically the rule was as you can see here the best practice for writing log files is that you always append log files should not be writable anywhere the purpose of log file is to keep the history unchanged so you should always use append and Estonal does it correctly and basically we forgot to add the append permission into the policy module so now I will add it and so here we can see that we can create the file we can open it but we cannot append these things I mean the editing of the policy file could be of course done by the audit to allow in the slides there are two ways the first to allow can generate rules without using of macros or with the usage of macros macros have one significant benefit is that the macro can expand to more rules and the defined macros usually take into account several operations which are needed for accomplishing some goal when creating a log file means you have to write to the director you should be able to create the file and other things you can compile it now let's load it I hope that it will be all from this test case as you can see the checks are here green all the rules which seem necessary are present and also you can see that the STUN process is running so our local policy module was successful of course it can happen that there are additional ABCs but not in this case so all the ABCs we saw we transformed them into rules then we somehow improved the rules by defining a new type for STUN log file and that was all necessary it failed but yes the only thing that failed is here so we have the rules but the file context pattern for the file is still incorrect I will show you so we can again use this matchpad and we will ask what's the correct context for the file it says this now we have a discrepancy between file context patterns and the policy because the policy says when a log file is created by a STUNL give it this label to the file but file context patterns will give something else another context so let's use again Semenage to improve the situation or maybe I can show you what happens if we don't do it so I will just touch the test case cleans up after itself that's why we don't see the STUN but I can of course change it I will remove it so now the test case does not clean up on itself so there will be the STUNL log file on the file system and we will see that the file has a different label than what's policy thinks about it and we will run a restorecon which is again one of the most useful tools to correcting context on file system and basically the tool will screw up so the file was created correctly with the correct context as you can see STUNL log but once I start restorecon I feel that recursive is not necessary right now so you can see that restorecon changes context of the file but to worse so if I run the test case again we will see again a solution again a conservative one is to change or to define a context for the path so we do that by again a Siemanage tool so this command this command will change the file context patterns database and it will add a connection between STUNL logT context and this path from now on the matchpathcon will say that the file context for our file is this so this is the situation we wanted now we can run restorecon again and it will it changes the context to the more specific one which is correct it is of course it is possible you can see that when I executed audit to allow it basically created several files this for one and here my policy FC means file context and if you want the context to be part of the module you edit this file and you basically adjust the definition so you can do that by editing this FC file do you have other questions? you can basically ask anything as serinux related and I hope I can answer do you have some recommendation for creating possibly serinux like what to put in the description what information to contain of course some NEC denial message probably serinux policy version but definitely I usually recommend put there the serinux policy package version like which version of serinux policy was installed sometimes the version of kernel is also handy and when don't use grep for finding ABCs please use all search because when you use grep it does not catch all the lines so yes sorry so we are looking for for example type ABC and type C score lines and a lot of them and so so this is a recommendation which the serinux policy developer needs to see is both these lines but when you use all search it will provide even more information for example I can show you what's the difference let's see so the difference between just greping the audit file is that you see some cryptic I would say cryptic hashes like this for example when you use all search it has a feature or a parameter which is able to interpret those hashes and the output looks in a different way so now the hashes the cryptic hashes are way and you can see that here are like paths complete paths and here also so please use all search a serinux policy version if you gather all ABCs in enforcing mode a serinux policy developer will most likely ask you for a list of ABCs gathered in permissive mode so if you provide both lists you save one round trip time and when filing back or customer case if you describe the scenario what happened it's even more informative for the developer and again for me because it will most likely end up in my queue you mentioned that module can contain file context definitions so that I don't have to label the files or folders manually using the semanage tool okay using semanage you can use a regular expression like all download folders in all home folders for various users so you use regular expression I was able to get absolute path working as file context definition using the gen context function in a module it worked I was able to use semanage with the regular expression it worked but when I use the same regular expression in the same module instead of the absolute path it stopped working is there some good job using regular expressions in a seal? I think that the usual problem is here this specification I mean that did you use this dot asterisk? yeah it looks exactly the same okay so maybe after the workshop I will go and we will see the problem so no generic advice? no even for example if you specify these tokens like twice in the pattern then it could be a problem when interpreting it and so on usually when I try to understand the ABCs I would like to know where to look for more explanation like what is the type like what are similar types if there are any Booleans not only just one line but one paragraph of explanation what it does and so on what page how to move forward with the ABC? yeah that was the poor a series of documentation I was talking about yes these Booleans are documented but there are yeah that's the one line unfortunately there are HTML pages that are part of a Selenux policy doc package and if you don't find the detail description in those HTML files I think that we don't have a better definition I think there is a lot of advice to enable Booleans yes yes but if you go through the chart you will see that the recommendations given by audits to allow are not always the best ones Is there any other helper for CETRABOShoot? Yes SE Alert basically this SE Alert is a tool which goes through the reported Selenux denials there is a database and it tries to interpret what happened and gives you some advice and you also but you asked about the Booleans and I'm afraid Selenux documentation lacks a lot of documentation sorry we are running out of time if you have other questions please come to me we will talk personally yeah we will talk about it yeah we will also ask so here I think that here was the question I don't know I think we have to do it Cometa Cometa Cometa Cometa Cometa Cometa Cometa Cometa Cometa Cometa Cometa Cometa Cometa Cometa Cometa Cometa You can show it to the camera. Good. They are here, but I don't know what they are. I don't know where they are. I don't know where they are. I don't know where they are. I don't know where they are.