 Hello everyone, I think it's time to start our talk anyone is here Thank you all of you for coming here or Listeners in this talk. It's really nice to see faces not in virtual And thank you the organizer on the CNCF or a 13 hour or talk to be To be here. We are to talk about how attackers use exposed Prometheus And server to exploit coon at this class there My name is Miguel and it I work as a security researcher for the last five years in different topics like cloud security or Sintending content to social engineer or also for detection and I work with David a Torres is manager of Prometheus team at 60 and we're here to try to explain this man It's like we put more and more layers more and more software But if you fail in one point everything is Broken and with that is the sports matrix. Okay, it's a typical Insecurity is very typical to to talk about this, but I think repeat and again and again and again in the next in the next time well I Lost The first thing to mention here is the disclaimer We don't like to release any new vulnerability or CVE to Kubernetes or Prometheus here is for common sense to educational purpose and we don't like to Provoke any attack for real customers and also we Recommend to follow always the secretive practices in Kubernetes and also to use Prometheus to monitor everything But please don't open the door to attackers or expose your Credential for free because it's very easy. You are an attacker to get to get access Why it's important or why someone likes to do Kubernetes fingerprinting. Well, if you are a Pentesting or ethical hacking or you're working security The first thing to do is when you have a target is to gather all information as you can Okay, you need this information to choose the best tool the best technique to get access or exploit this these things and it's normal to match with CVE or Non-vulnerabilities with the exactly tool and you get access is the first step to in detail access You need to gather information and we try to explain in this talk is possible with the metrics of Prometheus to gather this information and Create the plan to to attack a cluster of coronaries And this is not it's not something new if in the past this problem with Tesla is Forced because expose our Kubernetes dashboard. It's something I don't know if you you like to protect everything and you follow the secretive practices and you I don't know we use a lot of security techniques or Application if you expose your credential in clear text doesn't matter everything you Don't you access for free to your cloud provider and start to Crypto mining for example, but this one example of the things you can do and I like to remark this because in Spain No, but on the part that said this aquel que no conoce la historia está condenado repetir like you you work in the Incident response team or digital forensics the most important thing to do when you create the report is to to know what happened with this In an incident and you need to learn from the past to don't repeat and again in the future And this is important because what happened when Prometheus you expose again and Like the credential or not? well Also, we like to mention that this is not the first thing on call the first guys that Talk about this you do a quick search in Google you can see an issue from this shop that discuss this the same topic It's like if you use Prometheus and expose the the dashboard is Like your credential or you have a problem in the future or not? Also J frog Write this amazing Article about this problem, but more focus on the secrets not in the pivoting on gathering information, but this is really nice and Finally, we are not the first a speaker talking about hacking monitoring. This is something that is cool, but As I mentioned before The history repeat and repeat and they doesn't matter if you explain once or two. That's a matter The first thing to do is it's a real scenario to Access to a Prometheus exposed to the internet well, you need to configure is not by default the dashboard open to the world But the configuration is not enabled and if you configure the dashboard and sure Joe Configure the authentication or is free to the attackers to make queries or access to the Prometheus metrics and if you do a simple to Google dorking you can see some several results with real Prometheus service post And also if you are in different security conferences, this is very typical use other sessions engines like soda and then she's off To see the impact is a real impact because there are a lot of servers exposed or not With sudden is most simple because you use to filter the Fabricon but as you can see a lot of server are exposed with this introduction I like to share the hands to my friend and I OK? There we go. OK. We are under, and you guys know the ways of open web service, which is published, and we will get that way from there. But in a way, we don't know when we will be able to do it in time. So, the ground are not on the basis of this. But I'm not on the basis of this. So, let's start looking for where to use the code. This is not a database caused by an old spoiler. It is not a spoiler. It is not. It is not. It is not. It gives us information about even the chassis sometimes, the board vendor, but most important, it is giving us information about the system vendor or the product name. Cloud provider, it is. So, we are starting to have some information. We know that after behind this example.com web page, we know that maybe it is AWS or iShare, we know already what is the provider. And we are starting to add things to our map. This will be our map that we will use later on to make a route for attacking. We can have more information. This is another node explorer metric called node underscore network underscore info. And we can filter by all the labeled device like F or net. So, we have all the physical interfaces of the machine. And this is giving us some important information, like not only the physical address of the node, but also the availability zone of the service provider on the VPC. So, we will have also information. Also, these are from the cube state metrics supporter. We have also information about the hostname with the cube underscore node underscore info. We can even make some black magic from QL to filter all the service of the kind load balancer. As you know, these load balancer services can be found at the service provider implement behind something to expose to the public. And we can also have information about the ingressers that we have. This is interesting because the ingressers are telling us which reverse proxy we are attacking. If we are attacking slash api slash cartdb slash product, it goes to one service in the cluster and if we are going to make a slash catalog slash id, we are going to another service in the cluster. So, we are starting to make a map of networking inside the cluster. We are starting to have more information now. We are actually starting to make the most important thing is the root of attack. We know that behind these example.com slash api slash whatever, it goes to as ingress and that ingress go to a service provider. And we can do this with different points of the api. We have also information about the nodes and even the name spaces. And we are starting to add new things to our map. But let's go further. With KSM, we can have the name spaces, we can have the workloads, the deployment, the stable sets, even the pods, the container inside the pods. We are starting to have real information of what are the names of the applications, what are the name of the workloads that we have, what are the name of the containers, what are the names of the cron jobs that are being executed. And this is giving us a real map of what is behind. But not only of the user workloads. As we can see in this metric, this is from the Kubernetes components metrics that they expose natively. We have the Kubernetes underscore build underscore info. And this is giving us information on the version, except version, not only major, minor, even the git commit and the build date of each of the components on the Kubernetes control plane. And this is important because now we are having more information, but we can start to add vulnerabilities for the components of the control plane. We cannot access that now, but we are starting to add more things to the map. Let's change the dimension. We are talking about Kubernetes, but let's talk what is behind the machine. We can have information about both the image that the node is running and the kernel version. And this is important because we are adding more known vulnerabilities, not only of the Kubernetes cluster, but also from both the kernel version and the image of the operating system that it's using. Let's go further. Let's go for applications. We are having Kubernetes metrics is giving us more information about the image and the tag of the containers running in the cluster. So we know which image is running, which tag they are using, which hash it is, and even we know also the registry that they are using. If we even want more information some applications are even exposing the more specific information about the version or even the configuration options in some custom metrics like Prometheus underscore build underscore info. And now it is when it is starting to get interesting because we are adding even more known vulnerabilities for each application that we can find in the cluster. And last but not least, we are also having information about the secrets. The name of the secret. Yes, but we are also having information about the annotations and the content of the annotations of the secrets. Why this is important? I don't know if you know that in some versions of Kube CTL Kube CTL was adding an annotation of the current applied configuration that was exposing in plain text directly the content of the secret. So all the roles and all the things to service accounts no sense because the content of the secrets was being exposed directly in your KSM Prometheus metric. So we already have a lot of information and we already have a nice map of the cluster that we want to attack. Do you want to know something funny? We gather all this information most probably without anyone noticing because Prometheus has the capacity of logging all the queries made by the API on the frontend but unfortunately this is disabled by default and you can even check if this is disabled or enabled with this metric like Prometheus underscore engine underscore query underscore lock underscore enable. If this is a zero we are in Ninja mode and no one realized anyone locked anything that we did to gather all this information but I'm not the hacker one I'm just the Prometheus guy and I will call now my friend from the secure part Miguel you drive from here okay nothing to do okay we have all this information it's quite simple to make queries, gather all the information map everything you don't need to initialize to know if this component is vulnerable or not, if you can escape or not because you have the information we like to expose three different examples on how this is possible or this fictitious but not it's possible to achieve the goal of the attacker it's like I like to leak data or I like to crypto mining or I like ransomware it depends on the goal of the techniques or you can perform different paths to get the access and I think it's interesting to see what happened with all this information the first scenario is with like data okay this is an example of a real scenario everything is fine we follow the secretive practices we don't have any CVS or known CVS in any of this part the components are fine the nodes are fine the applications are fine and we have an IPI to make queries but I like to leak the data inside without authorization the first thing to do is to use for example a SQL map to try to do some injection in the IPI and it's possible to do or not it depends but in case you can't what is the registry if it's a third party or private I know the images and I can perform some social engineer to get access to a third party registry for example because I like to upload a malicious images in the continued delivery to get access because the other way is impossible to initial access in some cases in some scenarios we craft the imaging with a vulnerability to get access and perform the other steps to do a privilege or something like that but in this case you need to focus on the supplication because the other way you need to 0 days or something like that but in this case we need to do by supplication attacks and you can use beef to create a phishing attack or you can do something similar to tipo squatting to upload a new container with similar names I don't know but this is the first scenario the next one is possible to be a register or not you have an application and you know we are promising you that the ingress send the command to an application that is vulnerable to love4j and you have initial access and you know you are inside a container but this container also have vulnerability in the node that allow the elevate privilege to what is the goal in this scenario the crypto mining you like the AWS credential and you need to do these steps to get access to the node and following or seeing the config files or in the environmental variables if the AWS key or the cloud provider key are in this place like the example of Kubernetes dashboard it's a clear test in this case we need to do these steps this is a long path if you like to do very short you can do the same example with Kubernetes it's possible if you use AWS and you configure your secrets it's exposed by default in the dashboard and you make the query and get access to the credential and it's fine you don't need to perform any need to access or any other step because it's unclear ok this is example but is in the documentation please be careful if you expose the Prometheus it's possible that expose your secrets and it's something known for the for the community and again don't open the door ok and finally another scenario run somewhere it's a bit strange in Kubernetes cluster it's much for traditional environmental database that you cipher the data and ask for a rescue and in this example is a bit different you know an application have a vulnerability in the initial access via spring cloud you can inject a command via htb and access to the to the container and also you detect with Prometheus that one component are vulnerable at one tv and you can use to change the name space or get access to the secrets and access to the data in the database the problem here is to ask for the money ok I don't like to give any advice or anything to know who is possible to ask for the money in this case because you need something to I don't know deploy a pod with user interface or something like that but it's something similar you get the information from Prometheus and prepare the journey to get access ok this is a summary of these three attacks or three scenarios and I like to summary that is for security it's important to protect all the information you are exposed ok that doesn't matter if it's a metric or it's a credential or it's a server application I don't know but it's important because in security it's a long battle it's red team versus blue team and all the time we are fighting or cat and dogs it's a different idea but a long battle if you expose your credential or you expose your Prometheus or expose the dashboard you change to a speedrun it's very simple and we like to finish with the normal recommendation in each step ok all this talk is about follow the security practices in Kubernetes follow the security practices in class in the code, in the deployment you have these layers and sometimes we forget the metrics and the security metrics in this talk we don't like to explain any vulnerability to Prometheus itself because it's not the scope but it's also something to keep in mind and we like to share this recommendation and I think that's all thank you so much if you have any questions