 Hi. Welcome all to this talk. So today, we'll be presenting how to run secure and isolated workloads via Bosch. So I have with me Subankar, my colleague, and myself. I'm Shashank. So we'll run you over how we have done and run secure workloads of the Bosch-based deployments. So just to set the agenda at a high level, so when we talk of Bosch-based deployment, we'll take an example of service fabric broker, which is deployed as a Bosch deployment today. And we give you an overview on what is service fabric. Then there are certain requirements for service fabric where we need to deploy certain extensions to it, which means there will be third party code or scripts which will be running as part of service fabric code execution. So this is where the challenges are and how do we secure those kind of workloads. What are the challenges we see is what we cover in this talk. We will show you a demo on what kind of possible attack vectors we see when we allow custom code to be executed on the Bosch-based virtual machine. And then what is the mitigation for the same? How do we mitigate this? And what are the best practices around doing these or mitigating these attack vectors? And then we finally talk about what is the roadmap and the future for these security extensions we are trying to develop. Over and above all, we also demonstrate the actual attack vectors, what we saw, and then also the mitigations for the same. So at a high level, this is how service fabric looks like. So let me try this. So you see on the top the consuming environment, which is Cloud Foundry, Kats, and other platforms. And this is the piece of service fabric. We say service fabric broker plus plus because there are OSB and extension APIs on top like Backup and Restore. So the way it works is basically, so you have an API server sitting here. So service fabric, when you trigger a deployment or create CF service, create as an example. So service fabric puts a custom resource definition into the API server. So here we use the Kubernetes API server standalone mode, not the Kubernetes itself. So it talks to the CD where the CRD is persisted. It also generates an event back where you see these components like the Docker operator, the Bosch operator, and the Kats operator. So there's a specific reason why we did it this way. So in the initial release of service fabric, what we were supporting is only these two kind of deployments, the Docker and the Bosch. But with this framework in place, we are able to support much more flexible provisioning mechanisms, like you can add a Kats operator and deploy into Kubernetes. But the talk is not about that. Talk is about primarily the Bosch side of it, where how do we secure the deployments done via Bosch? How do we secure the service fabric deployment itself? So this is the whole user flow I'm going to talk about today. So why is the need of doing this? So when we were building service fabric and after a certain point in time, people asked, how do I extend service fabric? So in the deployment lifecycle, is there an ability to introduce some code which I want to get executed? As an example, you have a deployment manifest. Can I modify the manifest a little bit? Can I inject some credentials into it? So these are the kind of use cases which emerged. And this is where we brought in the concept of saying the third party extensions or the plugins. We call them the deployment hooks, where we say that you have the ability to introduce a deployment hook, execute a deployment hook, as part of the lifecycle of service fabric deployment workflow. So typical use cases, as I said, center around the manifest manipulation, but there are many things you can do if you are not checked or controlled rather. So what can happen possibly? So if you have the challenges of introducing a third party code running on the control plane, which is service fabric, what a rogue code or a rogue agent or a rogue plugin rather can do? So you can see he can inject a malware, can cause denial of service on the service fabric component. So he can launch a denial of service like a slow low risk on the HTTP endpoint on the service fabric broker itself, which means he can cause an outage of the whole deployment chain. Then there are components which service fabric talks to. If you have an agent which is uncontrolled and unchecked, he can also bring down the other components which have interfacing with service fabric itself. Then there are mechanisms like Petris where you can change the binary or the executable which is running. So a malicious code can just do a process following or change the binary of the service fabric itself, and it can go totally unaware. So what service fabric doesn't control? So we don't control what are the qualities of these extensions. I mean, you can have a limit to checking the static code checks and other things. Some things we will show in the demo can just pass through. So how do I check what kind of resource usage a specific hook is doing? What kind of system calls it is making? Is it trying to load a root kit and trying to elevate its privileges? We have no clue. There are mechanisms like using LDP preload to divert the system calls itself. So all these mechanisms can totally go unchecked. So basically the idea is introduce a small code, can just bring down service fabric or the Bosch deployment which you are running. So now I go over to a small demo on what are the possible attack vectors we see here. So over to Subankar. Thanks, Shashank. Am I audible? OK. So as Shashank mentioned, so we had this use case of running third party extension code as part of the creation lifecycle. But we also wanted to make sure that nothing can go wrong and somebody cannot introduce a raw code that can bring down the whole control plane of service fabric broker. So we would show a couple of demo how it can basically bring down. And then we will show what were the solutions that we brought in and how we solve the problem. So the first demo will be about a reverse shell. So just let me explain the pain over here. So I have the local machine from where I will create the service instances. And in the middle I have the rogue machine or the attacker machine where I will basically open up the reverse shell. And I have the third pain where I have locked into the broker VM, the service fabric broker VM. And I will show that what the attacker could do in the broker VM. So let us see what can happen in this scenario. So in the beginning, we are just doing an IP config, IF config, just to show that all these machines have a different IP address. As you can see, the first one has 172.17.0.1. The second one has a different IP address, which is a private IP, which is the attacker machine in this case. And again, we go to the broker VM and we do an IF config. And here the IP address is 10.11.252.10. So the idea of this attack is that this is the attacker machine. But executing a deployment hook here on this side, he will be able to obtain a shell here, which is the start of the attack. And basically then you can do much more on this. So I will just create a watch on the monit process that is running. So in this case, the monit process, the Bosch monit process that is running is service fabric broker. So over here, you can see that the process is running. And it also shows the IP address, just to show that we can actually locked into this system. On the middle pane, what we are doing is we are opening a netcat server. And we are just listening to this particular port. Now what we have is we have some rogue script, which is part of the service fabric deployment hook. So the moment somebody creates a service, this script will be invoked. So let us see what it can do. So we will just do a CF marketplace to see the services. So we have a blueprint service, which is a demo service. And we have a small plan. We will be creating a service instance for that plan. So I am doing a create service to create a service instance of blueprint of this specific plan. So as soon as I do this, the script will be executed. And you would be able to see that the reverse shell is. So in the second pane, you could see that the reverse shell has obtained connections. And it has basically obtained a shell of broker. So if I do a who am I, I get a vcap user because the process is running as vcap. And I do an if config. And that's why I had the IP address on the third pane also. So you could see that the if config actually shows the IP address of the broker. So this is the start of the attack, as Shashank said. Now what it could do? So it could do process tracing. So you can do simple ps minus ef. And you could probably guess that the process would have something like service fabric broker. So you could grep with that. You get a list of processes. And you could do a kill minus nine of all these processes. Because these are running with vcap privileges. So you could actually kill all these processes. So in this demo, we are killing one of these processes. And you could see that the Bosch process on the Monit process on the other side will go down. So this is basically you caused the outage. Yeah. So as soon as the Monit process goes down, what it means is your Bosch deployment of service fabric broker goes down. That means your control plane for creating a service goes down, which is an outage for your platform. So this is what you could do with a river shell, with a rogue code. We just to show what kind of code you can write to just obtain a reaction here. I mean, if you are thinking that writing this kind of code is a very big thing, it is not actually. So this is what I actually wrote in the script. It's a very small Python code, which runs as a shell script, basically. So what it is doing is it is just creating a socket. And after that, it is just redirecting the stream. And with that, it opens up a river shell on the attacker machine. So this is the kind of attack that can happen. So one more example that we want to show is the root kit example. So we found out one of the example root kit. And this is the code that you could run to basically gain root access. So you could be any user if you have a privilege to load a kernel module. So using INS mod, you can load any root kit. And after that, you have the root access. So you could do anything. You could do monet stop all and destroy all the processes. And that's probably the least you could do. So we will show a demo for that also. So again, we have the local machine. And we have the broker process running on the right-hand side. So you could see this is the script that I showed that will be executed as part of the lifecycle of the create service. And again, I would run a watch on the broker process. And again, similarly how we did for the other previous demo, we will try to create a service instance for the same blueprint instance. That will, again, deploy the deployment hook. So that will, again, call the deployment hook and will execute that piece of code loading the root kit and getting the access, the root access. And it will do a monet stop all. And as soon as it does a monet stop all, it brings down all the Bosch processes, all the monet processes. You can see the service fabric broker process has become not monitored. And again, it will cause outage. And your control plane will be down. So basically, what you are trying to tell is that there's an ability to install a kernel module via the hook. And then that kernel module itself is a root kit which allows a user to elevate his privileges to do root and then do a monet stop all on the other processes running on that virtual machine. Exactly. And also one more thing that we could do is a fork boom. I mean, that is a very common thing that we have probably done during our childhood coding. So here, what we are doing, we have the broker VM. And we are checking the number of processes that it is running. So currently, if you see, overall in this VM, there are like 149 processes running. So if you don't know about fork boom, what it does is it exponentially creates processes using the fork process system call. And then as soon as the number of process that it creates grows so high that your resource consumption becomes so high, your VM could crash and it could become unresponsive and eventually it will crash. So let's try to do that. So again, through the CF create service, it will invoke this particular code. Again, this code also writing this code is very easy. You could find it on internet. It's just a one line of script that you could run which brings down the whole virtual machine. Yeah, so you could see now on the right-hand side, it is the number of processes increasing at a rapid rate and at one point in time, even the watch process itself will be killed and eventually the virtual machine will be killed. It will exhaust the PIDs here. Yeah, okay. So we saw the problems that we have, right? So our intention was to still provide this flexibility for people or for the service owners to add this capability of writing custom code before the actual Bosch deployment happened. So what are the solutions? So the solution is to sandbox these processes so that it cannot affect the other process. So what you could do, you could apply resource limits in terms of memory or CPU or network or you could restrict the system calls. So for that, you could use Seccom and you could also then filter some certain system calls and you can disable some certain system calls like a loading root kit or probably opening up a TCP connection, something like that. Or disabling clone to be done from a script to launch such kind of an attack like the fork bomb. So also what you could do is you can have fine grained mandatory access control like AC Linux which is a Linux utility and you could also have it as part of it so that you could define fine grained user controls. And of course, Linux capabilities are one of those things that you could use. So in our case, there was a case where we had to deploy many service instances and these service instances should not look at each other. They should be restricted so that they can only see each other and not others. So we had to manipulate the IP tables for that but you don't want to be root and manipulate the IP table. So for that, we use the Linux capabilities and only enable manipulating the IP tables capability and not the entire root access. So this is something that you could do. So for setting up resource limits, so in the Bosch Monet script that we have, you can actually define U limits. So it's a bash utility that you have and you can define different limits like memory, CPU, number of open files, number of connections, et cetera. Which ultimately uses the R limit which is a system call. And also, of course, you have the C groups which are a Linux kernel feature to basically limit and also isolate the resources between the set of processes. So coming back to SecComp, so as I said, you could actually filter the system calls that you want to allow and this is what we used for our use case. So SecComp is basically a very efficient way of filtering the list of system calls that you want to allow. You can just configure it and you can provide the list of system calls that you want to allow and others will be denied automatically. And it's especially useful if you're running untrusted third party programs. We have used LibsecComp which is a C library that you can use and you can filter unwanted system calls. And in service fabric, we have a configurable mechanism of basically allowing the system calls. So our approach is to deny all and then whitelist certain system calls which are needed for a specific process to be done. So this SecComp actually is coming from a long time since it was introduced with Linux 2.6.12 and it was part of the CPU sharing program and it's a very, very nice tool and along with SecComp, if you use features like namespace and capabilities, you could actually restrict your process so that it does not cause havoc to the whole ecosystem. So we will show the exact attacks demo that we did and we will show that how it was solved using SecComp. So we will show the first one which was the reversal attack. So again, the same pain and I will open up the netcat process on the attacker VM in the middle and then we will see what happens. So on the third pain, we will show the settings for SecComp. So as I said, it's configurable and you can list the list of system calls you want to. Sorry. I'll get it? Yeah. No, this is not though, sorry. Yep, sorry. Yeah, so let's come back. Yeah, so as I said, we have a setting which is configurable and you can list down the list of system calls that you want to enable. So as you can see here, you have a flag enable syscott filters which is if you make it true, then it will be in effect and then you have some whitelisted syscall. You can list all the syscalls that you want to whitelist and those will be allowed and anything else other than that will not be allowed. So in this case, we have the reversal where you would need a TCP connections to be open, right? So we don't allow this and we will see that how Service Fabric Deployment Hook process blocks that. So we open up the Netcat process on the attacker VM and then we try to create the service instance. And these policies can be applied per process. Yeah. So as you can see, as soon as I created the service instance, previously you could see there was a reversal already created on the second VM. Now it's not there, right? So what happened? Something must have happened which has blocked it. So let us check the Service Fabric log for the deployment hook process. So we could go to the log and in the log we would see there is a bad system call exception that has got from the filter and it has basically blocked from executing that particular script. So this is how you could block a specific script that is not supposed to run. In the similar scenario earlier where we showed that the root kit was, it was able to load the root kit. So in this case, again, we show that the settings is enabled and we have some white listed, limited set of SysCalls, which is only allowed. And when we try to create the service instance. So basically you block the init module or the loading of the kernel module. Yeah, exactly. So the INS mod will be blocked and then when we try to create the service instance, previously it was able to bring down the process. Now that would not happen. So only the creation probably would ultimately fail. So it would later say that the creation failed. But then you would still not be able to acquire the root access. And similarly here also we can see that in the log you will have certain bad system called log, which basically restricts the process from getting executed. Yeah, so this is about the demo that we have. Now where do we see this is getting used and we have sort of other use cases that is happening in the CF community. So you probably would know there's a Bosch BPM release, which is also targeted towards process isolation and making sure that the process runs in its own name space. And it uses C groups for achieving the resource isolation. And also it has the ability to define a resource limit like open files or the memory that you could use. So over the period of time where we were developing this second based approach, we also saw that the BPM also was doing in parallel something similar. So we could also use some of the capabilities that BPM has. So running a Bosch job on BPM also is very easy and you could get things like the resource isolation and applying the resource limit. What BPM also provides is through Linux capabilities, you can also configure the grant for a certain user or a certain process. So you could also have Linux capabilities and the list of capabilities defined and the process will have those capabilities only enabled. So we see that along with BPM and along with Seccom, you could also have the list of system calls that you can block and that could be an efficient way to run third party extension code or third party extension. Or even the normal Bosch deployments in a hardened way. Yeah, so this is all we wanted to discuss and talk. So if you have some questions, we can take up. Anybody have any questions? So let's give them a hand. Thank you so much. But also I can take anybody if you have any questions. So if you want to have some discussion, we have all the contacts given here. Service Fabric is also an open source incubation project. We have given the links for all the GitHub repositories and our Slack channel. So you could also talk to us over there. I didn't get it maybe. So would it be possible to reach the same level of isolation with Bosch BPM? Or? Yes. So when you say same level, you mean the work we have done? So what you have done? Yes, yes. Okay, thank you. So at the time when we were doing it was in a alpha stage. So that's the reason. I think now it's in a stage where you could also use BPM. So for some of our Bosch jobs, we were also using it and we saw it's quite good. So along with that, I think for sub, I mean if you have use case of restricting the system calls, so we could also use something like set comp or lip set comp to basically restrict certain. So I think one thing we are not sure of is the Linux capability party yet. We have not checked on the Bosch BPM site. So how do you, let's say he talked about the IP tables use case where you want a process just to apply IP tables. We cannot do more than that. So you have fine-grained security applied via the Linux capabilities feature. So we are not sure whether Bosch BPM and which way it does it. So maybe there may not be there. Great, thanks a lot. Thank you. Thanks.