 Okay, so let's start on time. Hello and welcome to my talk about application-specific hardening. Application-specific hardening is a bit of a, well, not really clearly structured topic. We want to share or want to share in this talk a bit the experiences which we had with application-specific hardening. So going into like configurational settings, which we apply to applications for them to be more secure. And especially why do we do this? So obviously we want to have a secure system that is clear. So what are the aspects that we can enforce? What are the things that we can do so that our system gets secure? At first, normally we will try to have a secure system architecture. So we will somehow isolate components, we will somehow segregate components from each other, so that we have either physically or logically separated the core components with fitting security measures on top. So secure key storage, HSM or trust zone, secure key management on top of this so that we can introduce new keys easily and flash data security, obviously, so that we have the right code running on our device and secure boot stuff like this authenticated boot so that nothing started which we didn't want to. But then we have then the separation of which I talked. So we have like a networking stack, which is separated from the center gateway so that our core components are not directly connected to the outside world. Here again, obviously, certificate and key management. And here we're getting now a bit more into the hardening direction because these services which are running then on this gateway, on this networking stack, need to be more secure and more hardened because there's an interface, obviously. So here we need to have component-wise software with small dedicated functionality that would be best. And we have to harden this as well, but we're getting to that. This gateway here, I said this is gateway functionality so this will be well enforced by like some routing functionality, some IP filtering, some packet filtering and so on. And we only want to have few open interfaces, obviously, so that we can secure them. So this is perhaps the first aspect that we have like a secure general architecture, which can be deployed either on large scale or on small devices like IoT devices and is obviously changing a bit depending on our system. But we as well have then now our components there, our software components, and they should be secure as well, obviously. So we have different possibilities there. So we need to have like a secure code development. We need to have developer which have a mind security and good coding practices. We can have on top of the static analysis tools, we can have code reviews, etc, etc. We can look at the programming language which we are using. So this depends now if we are using Rust or if we are using like C, C++ as a big difference then in C, C++ we would have not a real memory safety. We can have problems with pointers and so on. So here it is sensible to introduce like binary exploit mitigations so that we have like an address based layout randomization buffer overflow checking and further which are, which can be activated then in the compiler and linker options. But another thing then is which is getting now to the hardening section more other secure configuration settings. As I said we want to have software with small dedicated functionality. So we would be good now if we would have like our secure architecture and our secure components themselves. But to have really secure components is not generally speaking given and we cannot ensure this completely. So there will be always coding errors. There will be always new technologies arising as threads. If for instance for the coding errors just they are different estimates for vulnerabilities existing in the code base. This depends now obviously on the complexity of the code on the coding language and so on and so on. But this is a bit high estimate here of this source but obviously there will be some vulnerabilities in every code. So this could be for instance validation errors this could be value overflows. This can be like unauthorized access to key material or in the cc++ cases would be point-to-g referencing for instance as well. But now coming to the hardening point at least we can have as well configuration errors vulnerabilities which are not directly coming from our code base. So we can have for instance a service that is simply running as root which is well a basic problem then. We can have the set UID or set GID functionality enabled or we can have capabilities here kept this admin is again a bit like root. And we can have write access to system directories like boot like further system directories. Other things would be write and execute of any program of any library of any best script whatsoever which we have there. Further points which are perhaps to keep in mind obviously in larger projects we will have perhaps proprietary libraries we have third-party libraries we have libraries from suppliers perhaps we have legacy software which is grown over years where we cannot ensure that there are no coding errors or less coding errors where we cannot really do a good code review. So all in all our applications will have some vulnerabilities and if those vulnerabilities can be exploited by a hacker then it is possible that he can afterwards escalate privileges to gain access on the whole system or control over the whole system and that is something which we want to avoid. So here to make the point a bit clearer we had I mean in the last years there were several occurrences of vulnerabilities in different services of Linux and this will not stop. So security in depth will now describe that we have more layers of security and one of this is that we even that we try to isolate our services and try to make it difficult for an attacker to make a privilege escalation once the service is compromised. So what are now our measures which we can take for application hardening? So I want to give a brief overview over the different Linux security features which we have existing. Here in the top bar you can see the different features which are like intersecting having intersecting scopes which are partially depending on each other and I think this landscape of Linux security features is not that easy and not that clear. Let us shortly or briefly go through them. We have a discretionary access control because it's pretty basic everybody knows this is like ownership and permissions but it's not always that easy nevertheless to deploy so it would change like the service which is running as route to a dedicated users would not always be very easy to do this. Next we have the capabilities which are like checks in the kernel which are specific permissions granted per user and then as well per service because every service is then had started as user. This can be for instance kept as time like the possibility to set the system clock can be kept as admin which would be basically route capabilities or kept net admin kept net raw here in the networking section so kept net admin would be setting for instance IP configurations and so on IP filtering kept net raw would be access to sockets either write or read. We can have mandatory access control as contrary parts to or the different parts to discretionary access control this cannot be given to other subjects this is configured per service and cannot be passed like discretionary access control and this enforces as well file access resource control different capabilities can be set here as well and this is provided via an extension to the linux kernel via the linux security module and it's called via hooks that you can see you have a security policy which you can set via for instance se linux or abama profile or other and this is then enforced when the kernel is working we have the namespaces as well namespace provides a kind of an isolation and so we have a different on distinct view on resources and depending on our namespace I think that's pretty much clear and can be inherited we have different namespaces most prominent perhaps is username space but as well amount namespace networking namespace for instance PID namespace and here you can see I hope it's I hope everybody can see this even from the back you have the init for instance username space so this would be the root the first one and if then a child process is being created and here the namespace is changed then we have a new new capabilities for instance in the user root and the user namespace and the the user inside the namespace can be for instance zero even if outside it isn't zero so access to every resource in this namespace is working with the rules from inside this namespace but outside the namespace the resources cannot be that easily accessed this holds as well for the amount namespace here we can have different virtual file system overlays resulting from this the next point are c groups which are resource restrictions which refer more to like really hardware resources those are set with respect to different different subsystems of the linux kernel so you can have here for instance CPU memory or network so a process can have a restriction that he has only limited share of the CPU so that he cannot use more CPU share than is set here this helps us against the denial of service attacks for instance and last we have system core filter which reduces again the interface between services and the kernel so that different system calls are blocked or this has to be clearly adapted on every service that we use because otherwise the service would simply not run anymore we need kept this admin capabilities at least to set this filter and last one which is not really a basic linux security feature but obviously rather service the system d which is present in most linux distributions and running as process id one and orchestrates background processes and here we can have different handling parameters as well different security settings as well and they are kind of the other linux security features are abstracted and being used throughout this system d service so all in all it's good to have most of them activated or many of them activated so that it's more difficult for an attacker to gain access and control over the system so that you have more isolation of the services but you have to consider different dependencies between those linux security features for instance the way if you like first use the username space and later change capabilities and so on so we have to keep in mind they hear the order in which they are activated and this has to be obviously adapted one time on the system but on the other hand as well on the service so generally speaking we want to like tailor our capabilities and our permissions of the application which we which we are axon mining so that it can only do that stuff which it's made for coming now to to the hands-on part where we look into what we did here and what we how we deployed those services or how we deployed this hardening on the services so as axon mining system we had a software system package for IoT devices which is based on Debian and here system D was present and we had as Mac implementation app armor we did review three applications which was Mosquito which is an MQTT broker obviously Conman a networking service and Blue Z a Bluetooth search as the name suggests so we concentrated mainly on those three security features but we have to keep in mind obviously that system D incorporates many of the other security features Mac incorporates to some extent as well other security features namespaces and cgroups we did leave a bit out at the beginning now because they could be provided yeah as a basic version pretty good beer containers to have some isolation here which could be hardened afterwards so how did we do this now how did we get going starting with discretionary access control we looked obviously which which users or as by which users application started which user owns the directories which the application is using what are the permissions here and if the application was run by route then we did change this setting and set up a dedicated user for the mandatory access control we looked into if there's even a armor profile existing if it was existing then we looked if it could be enhanced if it was not existing then we had to create it from scratch or search the web validate it and possibly refine it system D as last point here we looked into mainly into the capabilities this was the capabilities as I said so some D abstracts many of the other security features and we looked into what other hardening measures could be deployed here so as I said we wanted to tailor our permissions and access as far as possible onto the service possibly one other point which is to keep in mind and one could forget possibly about this is what I said in the beginning is that we wanted to have small dedicated functionality of applications and this is something which is good because then we have little attack surface on the one hand and on the other hand we can limit the capabilities the permissions and so on further so this is another thing which we looked upon are all the features of a service needed can we perhaps limit the service a bit more can we disable the feature because it's existing on the system in another form and if that's the case if we could limit this then we could reduce here the capabilities and permissions as well okay so I talked now how did we do this and so on stuff let's look into a service getting to our first one Mosquito first how did Mosquito go what was the status as we looked at it so Mosquito was being started as route and then dropped its privilege its privileges afterwards using the set UID call configuration directory was owned by route which is okay manager access control we didn't have any app armor for profiles so far and system d there were some minor hardening settings but this was not really yeah there was not really much done so getting a bit further what we did now obviously for discretionary access control we started now Mosquito not anymore as route but made a distinct user which we named Mosquito so that it's now not running privileged anymore we had now to change we had to add some access permissions for the user Mosquito or the other users here in this case because as I mentioned ownership of the configuration files was route and perhaps one thing which one could think about our certificates and key how are they handled who owns them rather root or rather Mosquito user this depends on the use case for the mandatory access control we found actually a Mosquito base file on GitHub which was pretty nice we had to add their more permissions because we changed our user from route to to a dedicated user and we we use here error messages which were generated when we started the service by app armor because to do like a trial and error approach and to find out what was missing there coming to system d now this is perhaps a bit more like the core of it what we did now but as I mentioned system d abstract this kind of okay the first point is obvious we again we changed to an unprivileged user but perhaps here an interesting thing is the known new privileges are set to true this blocks set uid and set gid calls so this can be dangerous if a service can change its uid so this is something that we don't want and we don't need anymore because we are starting as non-route so we can leave it as it is then the app armor profile is loaded sure this is basic again we did have here a very strict IP setting so only local IP addresses were allowed this is because we did start Mosquito in a virtual machine but this is not always the use case so this is here a specific thing we did sets memory deny write execute which I mentioned earlier so that we protect against malicious execution of code and memory pages cannot be written and executed this is obviously not possible if we would have like interpreter based languages here we set restrict real time this is something which is in the IoT space perhaps not the common thing but this would be just for dust protection so here C groups are used again in the background and here we added some further restrictions restrict namespaces some protect home protect system private temp private devices protect control groups it's all like some file and directory protection that are that they are not accessible or not writable by the service and this is all realized via namespaces the amount namespace via username space here C group namespace yeah and the last thing here which we use is protect kernel modules so that no additional kernel modules are loadable this is for instance implemented via capability since this call for it just in the background so you can see here this abstraction a bit so this was our first service which we contemplated this was the mosquito service coming out of the con man service which was much more extensive to to look upon so common again and it was started as route here and at armor profile was existing in system D again we had not very much hardening measures but another interesting thing is that con man is running as well dns resolver and an ntp client which are not running in sub processes and this dns resolver was already exploited in the past so we had some motivation to to limit the functionality of of con man given that dns resolver and ntp client were existing on our on our system otherwise so this was one thing that we did we did disable this functionality in con man and therefore we could apply some other measures as well so we could reduce the permissions in the mandatory access control file and further we did some hardening measures in in system d as well like in in mosquito and we had to add cabinet admin and cabinet raw to the system d file here because we changed from user route to a dedicated user and we did need now the capabilities which we didn't have anymore for networking ups now this doesn't seem perhaps so much work but some challenges here for instance for the discretionary access control as we changed from route user to a dedicated user some things didn't go anymore so we had for instance to add debuts communication permissions and we had to to change access to temporary directories like run and def so you can see where to change the debuts configurations in for for con man for the con man configuration and debuts and for the vpa supplicant and add the user con man there as well obviously for the for the temporary file systems we had to enable access via temp files d service and via udef rules so that on every startup the access is set up again here perhaps so this is like if you do know the functionality of of your service very well and you know very well which which files are accessed which files are manipulated which files he does need then it's possibly good to or relatively fast to to deploy these changes if you don't know it that well this takes a bit of time getting now to blue blue zed blue zed was pretty much the same like con man so here again we changed the user from from route to a specific user enhanced the app armor profile and system d for only the usage between only to enable the usage of bluetooth we could deny many things like ip usage like ppp usage and socket access we could deny many networking things but now if we use if we want to use bluetooth tethering and use con man and blue zed combined the situation changes a bit because now we have to use we have to connect to sockets for ip p for ppp and we had to allow some networking here another thing which is which was to do then is to add in the in the debath bluetooth policy not only blue zed now but con man as well as they were communicating with with with each other so what we didn't this extensively so here more changes might be needed and what was the outcome of those two services that now we have to consider a bit more cases we have more functionality possibly which we didn't see beforehand we have more we have more communication between the services and perhaps we have like files which needs to be accessed via those two services or more services even um which needs to have need to be in the same namespace for instance and and stuff like this which has to be kept in mind so this doesn't scale like linearly if you add a service which is hardened as well but if you want to harden all your services on the system then it's can get much more complicated and time consuming i was right now going a bit into the conclusion part as well but we're getting now into the rear conclusion generally a detailed knowledge of the application and the interaction with other applications is always helpful here this makes sense but we do not always have this knowledge either on the other hand if we would have this knowledge we could change perhaps the applications in the first way or we would be like the developer the security feature landscape is quite complex so there are many security features and they are interacting this is not that easy if you are not familiar with it obviously we do not know the vulnerabilities beforehand that's why we have to tailor the service the most that we can do um so that it's not easy for an attacker if he exploits the service to um then um exploit the whole system afterwards and when we now tailor the the hardening to the application the problem is that if we do not really know to the whole extent every interaction or every system call which the service is doing then we can run into the problem that some case of the some edge case for instance of the application or or some functionality is not working anymore and the application crashes so our hardening efforts have to be um paralleled with a very with an extensive testing as well and this extensive testing has not only to be done and this hardening as well but as well the extensive testing has not only to be done when we are hardening but as well if some service updates are being enrolled so we have some changes in the service and some of our hardening configuration might not work anymore so this is not always leading us to the effectiveness that we would like to have this is like perhaps the drawback of this service and it's obviously um taking some effort but nevertheless so we want to have less attack surface which is to some extent provided via this because we have a general um uh smaller interaction with our system um component separation is something which we can realize by our hardening efforts as well I mean you can use containers but then you can go further steps and and harden those as well so it's an even more separated system um and then the the next point was if we had like a compromised um application that we secure our system from this application which is now the the reduce of the risk of privilege escalation which is leading to our secure and security and depth approach so those are very important points if you have the time if you have like the energy and the manpower in your project and this is very good to do this the backside as I already said this is there is some effort for the hardening you have to have knowledge of the application and knowledge of the security feature landscape there is some effort on the on the testing side there is some effort on the deployment side if the service changes um so yeah there are like two very important parties or parts here so you have the very important part and very um yeah yeah very important part on the on the positive side but as well a big drawback on the on the negative side of this so I'm finished right now here so we're coming to the question part um it's not only about questions from your side but as well about the question from my side do you have any experiences with application hardening or what were you uh what were your favorite um or your best working tools for that yeah so that you mean that we are a bit okay um he said if we could look a bit more into the service like right that I did I understand this correct that we look a bit more into the service like right I didn't understand everything as well yeah that's right so looking at the application and and what is perhaps the threat there and and doing our hardening on this makes sense obviously but we had we wanted to try out at first just like hardening measures how they are working how easy we can deploy them this was not like getting into a product um state but this was more like trial um like uh yeah like like an X mining phase but it makes sense to um look for instance as applications on the edge and what are perhaps their interactions and so on and deploy the the the fitting hardening measures there yeah yes so did you try to do some extent yeah yeah yeah yeah specific to your deployment with the package maintainers or even with the upstream maintenance regarding your cookbooks that I would like to do this by default it doesn't make sense would it be generally useful or am I missing the corner case you were talking about um we did only do this to a limited extent and this is something which would make sense obviously so if you have deployed this upstream then your deployment efforts wouldn't would go away sure but this is like yeah depends on how much your upstream vendor is going with you then or not I mean this depends then but for this case this would be possibly um a thing to do right yes you mean it you mean for mosquito right yeah so we did add then for instance read access access to the other users for all other other users in the cage and the the problem where for instance now the certificates right because not only mosquito could could read the certificates but other users as well there's that's why I said this is the point which one has to think think about possibly um but if we would not take root as a user but for instance mosquito as a user then then you would have if mosquito would be exploited mosquito could change its own settings possibly in a in a bad way or stuff like this so this was like not really clear and depending on the on the different files this is answer your question or no we did use standard pipe emissions for this further questions of out of or further answers to my question yes things yeah this is something obviously which we did as well okay okay okay yes so the threat based thing is one okay this makes obviously sense right okay further questions so far otherwise yeah we actually didn't try those but this makes sense vulnerability scan tools are available and this goes into the threat threat direction as well um yeah so we we did more now look on to the measures and not on to the analysis but normally you should have obviously a risk analysis and the threat scanning and so on or threat analysis and um vulnerability scanning um but this was like more as I said like more and yeah trial phase to to look how these how these measures could could work together but this is not for products yeah I do see there's some ah I have to come to an end right okay and thank you so far um yeah and see you