 All right, I still see a lot of people joining which is great, but we'll we'll go ahead and get started Welcome to week nine of our 10 week CubeSec Enterprise online webinar series we've had just tremendous interest in this program and You know, we're we will do it again, but we're exceeding It'll be a few more months before we kick it off again, but we've had some great sessions in today's session is no different We'll I think you're gonna really enjoy it So let me before we get started just walk you through a little bit of the logistics We are have everybody on mute. We'll be taking questions if you do have any questions Please you to use the go-to webinar questions panel Over there on the side so if we can advance to the next slide I'll walk through today's The the other elements here So go ahead and use your questions panel and We'll you know, we'll we'll we'll take them that way We'll try to handle them You know in real time if they're relevant to what our speakers talking about Otherwise, we'll save it for Q&A towards the end. We are recording today's session You'll get a copy or you'll get a link to that recording you know after today's event and you can Share it with some others on your team if you find it interesting and we're very We're also very interested in your feedback towards the end So we will be sending out a survey along with that Confirmation after the event we'd love to hear what you think and if you'd like to be part of our next program You know, let us know that too. We'll keep track of it If you if you feel like you have a topic that would be of interest to the audience for our future series So let me turn to today's event now. We're excited to have Riaz from cloud will hear with us and the topic that he'll be discussing today is is really interesting in that You know a lot of times as security people. I think we Take the perspective of let's Our own perspective. How do we lock things down? But one of the best ways to understand what's happening is to look at, you know, how are these people? the bad guys, you know aiming at your your Your environment, how are they getting in and what does it look like when they're in? Riaz will be walking you through, you know, exactly what people look at when they're targeting your environment and And how does it look in your systems from your perspective? What can you see and and of course, you know, beginning to think about what do you do as a reaction? So with that, let me turn it over to Riaz today's presenter and look forward to you to any of your questions We'll keep an eye on that as we go All right Thank you so much Andy for that Okay, good morning and good evening folks for people joining in joining us from different parts of the planning The idea of today's presentation stemmed from an internal discussion we had right where we were trying to understand From an attacker's perspective, what is it that is visible to an attacker and how does an attacker traverse all the way from obtaining initial access right and all the way to the impact that the attacker wants to cause within the cluster infrastructure that the attacker is targeting The question that we would then pose to Customers would be are you certain right that you know who exactly is in your part To rephrase it who else is in your part, right? And the whole idea was to get an attacker's approach to commit a security Right. I'd like to shout out to my co-founder Akash and my team here as well as Matt if you've joined us thumbs up Thank you for joining this session as well I'm one of the co-founders of Cloud Link. I my work primarily deals with offensive security at Cloud I have over a decade of experience with the breaking applications Which has truly transitioned into now focusing on container and security security On throughout my entire real bad looked at Applications or any infrastructure that is given to us, right? Regardless of whether it's any new tech the questions that I ask the approach that I have always taken is to You know align myself with what important attacker do and ask relevant questions around that so that you could apply that Defensively and then say oh if you are an attacker I have this in place, right? So the defensive parts of what We ask also become clear I also have multiple certifications Which I've gathered over the years as part of my learning curve seek and seek out being the latest of them because covenants I have a technology itself is really new to me and My entire approach to learning that as well has been through an offensive site Well, let's jump into what we're going to be covering today the The entire presentation is kind of divided into three subsections I would say with with us starting and asking the question Are you or rather? rephrasing that Do you know what your organization's footprint looks like on the internet, right? Are you aware of what your organization exposes your employees expose your? You know infrastructure administrators expose the second part of today today's presentation will deal with actually asking relevant questions around what would your What would an attacker do to get inside of infrastructure or attack of gubernator's cluster, right? What are the tactics and techniques they're using and we're heavily going to map all the Tactics and techniques that we're going to look at with examples taken from real-world Cases as well as our pen testing engagements that we've done around covenators will map them to the mitre attack You know the threat matrix at Microsoft has created and we'll spend some time with that as well Like going through all the tactics will finally any presentation with Last section being around what will you do? to minimize the spread of an attack or to prevent an attack in the first place and all of these recommendations that we will talk about at the end of the presentation come from What I have seen as an offensive and security specialist, right? Let's jump straight in Oh, we have a poll here And you know that they go in Yeah, definitely. So before Riaz gets into his talk, we'd be we're interested in You know what you've all seen so go ahead and select one of the answers Have you faced any internal or external breaches? You know essentially have you been hacked? You know specifically in your your Kubernetes world here I mean obviously if your insecurity go back 20 years to to a network access different But you know have you been hacked in your Kubernetes environments and you know the answers multiple breaches? You don't have to tell us the details you've had a breach, but you feel like you dealt with it. Well You know you you feel like there were attacks, but you actually were able to prevent them And then wow, you've never been breached. We're so good, you know Or maybe you just don't know and so go ahead and pick one and you know We'll give you another 20 or 30 seconds here just to pick your answers and then we'll share the results And and see how we're doing so I see a lot of the answers coming in here it's interesting and because when when we try and phrase this question and in a public forum, right and The answer changes based on anonymity You wouldn't want a lot of folks to figure out the day You know what I've been pretty sure they are kind of embarrassed about it because we left the keys out in the open and get up for example, right? and essentially accepting the idea behind Hey, I've been pretty sure this is what I've done to change security helps in learning for the larger community as well Yeah, so interestingly, you know about half the audience is confident that you know so far at least in their Kubernetes world They haven't been breached You know or you know are unaware of it if it certainly hasn't had a major impact or they would know But you know and a number of people are pretty confident that they even if an attempt was made they were able to thwart it This is what do you think about these results Riaz? But then that's the the last option that the never breached that's what we know for an infrastructure is Is You know a little misleading because if you're all familiar with what happened in the latest the solar winds attack that happened Organizations are affected and they did not know for long So that's that's always going to be the case, right? All right Close as it was. Yeah, sure The Essential idea is always being that When you ask especially then during conferences or during interactions is when they're imposed as well the standard response has always been the more common one has been that we don't have a lot of You know stuff that we expose on the internet. We're careful about what what we do, right? But let's take a look at what as an attacker over the years I have seen people exposed right and attackers tend to use these to then focus channel their attacks to specific regions of their infrastructure Trying to gain access to a bigger set of the pipe that exists on your network Hey, and a lot of these a lot of this information Occurs to you as an organization occurs to you as something that's required as part of you being For your existence right but for an attacker all of these tend to become pieces of information that they can chain together right The most common information pieces I've seen come from employee information Which exist on forums or late in right the profiles contain what technology that you're using within your organization Internet accounting details that you know a lot of companies post Detailed job postings and public forums are responses to stack overflow depth of responses the blog post that companies write all of these leak some kind of technological information attackers a common Social information for attackers by bounty hunters, you know, whatever your motive is a DNS records The DNS IP history IP ranges all of these tend to become a lot of good sources of information These petty services that get detected by internet wide port scans An attacker need not even run a port scan against your interest There are services that provide you provide details of what's running one on your network simply by querying for your IP addresses That are mentioned in shodan being one of them right Attract the OS server version information by HP headers page responses a very common piece Recurring team. We've seen is for a lot of folks the embed developers tend to embed keys and secrets inside javascript and then they Assume the the minification or the Compressed format of JavaScript to become an offer station technique I but that I can stand can easily reverse engineers or see that in the job in the browser a very interesting source of information and bunch of you can try this out Look at the PDF file properties that exist for file uploads on your own website, right? As an organization, we produce a lot of content. We may have ebooks. We may have spec documents We may have you know job postings documents that are hosted on the websites If you download any of these PDF files and look at file properties You'd be able to see the software that was used to create the document potentially the username if it has created on the windows or a mac machine and All sorts of interesting things including file path sometimes if you look into the PDF You know, we we've had some data from our own threat research team at aqua and you know when Fascinating is when you know They'll do honeypots and they'll put up a new Kubernetes cluster and it's a matter of minutes before people are attacking it like like you said Yes, those ports again and everything else the DNS and you know, I'd be ranging all that stuff They've automated the process of finding new places to go. So it doesn't take much Yes Hey, a lot of organizations tend to You know have staging and dev environment that I'm is configured because they don't restrict the kind of traffic or sources of traffic That are going to be coming to those endpoints, right? So that's that's a particularly interesting case Default credentials for third-party apps a lot of large companies You know take to pay out Large amounts of money to bug bump your hunters because they found an endpoint which you could log in using an admin and admin credential And I thought this has happened in in the real world as well The tendency to use old and vulnerable software packages that don't you don't update periodically, right? That also becomes an attack footprint for your organization hidden files configuration finds Repositories on GitHub Bitbucket right all of these comments containing secrets all of these tend to become your source for The attack right the attackers tend to gather information about your organization using these and this is not a complete set This is this is the tip of the iceberg Right, but do folks really exposed of online? Right, that's that's the question regardless of the claim that I'm making and when I was creating this presentation. I tend to Sat out to see Let's actually try and Answer the claim rather prove the claim that I'm trying to make What is it that I can see by doing a quick search for the sources that I mentioned and what I noticed was attack Organizations tend to leave information on the internet all the time right these three screenshots show the first one shows the shows a bunch of Kubernetes clusters that are listening to the internet and If a Kubernetes cluster has been started the API server has been started with the priority and fairness features You would have the X Kubernetes pf flow schema hyphen you ID and that's a unique header Fingerprint that Kubernetes will expose to the internet, right? So simply searching that I'm showing shows you that it results This is a search that I did for deployment. Yeah, I'm hoping to find the Kubernetes related emin files for deployments on Greyhatte warfare a pretty cool search engine for buckets and storage on Azure And this is a Vim underscore setting so jet brings the file that contains project information sometimes the history that Of the project itself and sometimes confidential information can be on GitHub This is a peer search and guitar that was done And then just to just in an essence to figure out if what is it that attackers tend to see and is How much is this close to the real world like the game that I made across all the different sources of information? All right, great. So You know, I think as we as moves into the next section of his talk, you know The question comes up of you know, where are you our audience in terms of your own adoption of gubernettis and You know, I think you know the question is you know Some of you might already be there you've all of your production workloads are running Kubernetes I'll be surprised but we'll see You're drawing up plans and you're testing some apps in Kubernetes you're thinking about you know what your infrastructure looks like and could be moved there or I Love this one Riaz. I don't need no stinking gubernettis, right? You know I don't know why you'd be on today's webinar if you felt that way, but let's see People a few seconds to sign to pull their answers Yeah, I see them coming in and we'll give you about ten more seconds and share them there you go So you're kind of a good mix. I mean people are either in production already and with a you know bulk of their workloads there or You know, just just drawing up plans, you know, they're doing some apps maybe in pilot modes and but you know, definitely You're strong strong in the first two Let's then jump into the meat of today's presentation Yeah taking in from what we've learned that organizations tend to expose a lot of information I don't attack us when they want to focus on figuring out Kubernetes clusters I didn't targeting companies that run Kubernetes on production or by the testing it as long as you can get into the cluster you would begin with a Structure, okay, and this is this is this is something I've seen over the last decade Working with a lot of other security folks, especially in the offensive security domain All attackers irrespective of how unstructured they are follow a set of tactics to reach their end I'm extremely unstructured in my real life I think in the real world I tend to have windows all over the place and they extremely With a very short attention span the regardless of what your structure around Whether you have a structure or not Attackers will follow a set of tactics to be stable, right? And each of these tactics you could consider them as milestones They tend to have techniques by which the attack is fulfilled I don't I'm trying to slowly build into what the mitre attack framework looks like right and depending on the technology that Is being targeted the tactic might differ but often alliance with you the attack will perform initial discovery There would be some amount of code execution. There would be an escalation of privileges You want more right from your access that you've obtained There's going to be persistence in some form or the other There's going to be evasion or evidence removal in some cases that the attackers don't want to know that They're there and eventually ending with an abuse or impact of whatever access you've obtained, right? All of this maps to what the mitre attack threat matrix is In a sense, this is a globally accessible knowledge base of adversary tactics explaining What an attacker would do across different technologies and the techniques that they would that that they would use For attacking those technologies, right? And all of these are based on real world observations and the link is right there if you want to take a look Several matrices have been created for various Tech verticals like windows Linux being the most popular ones. You have a place GCP that have come out right and bunch of mobile OS tactics and techniques as well These can be used by both offensive security folks as well as defenders like offensive security folks Obviously the information is straightforward. You can pick up a technique and make oh, this is what I need to do You know complete this tactic within the within the framework and for different as the information is equally useful Identify the attack path that an attacker would take the back as we take so that I can build my defense in depth based on what I see right Microsoft created a similar threat matrix for cobalt is and that's going to be our focus for today's Presentation you're going to take a look at what the mitre This is not exactly the mitre attack Matrix essentially, this is Microsoft's version of using the threat matrix as a framework You know in the framework Creating one for covenants and attack like that matrix for covenants and most of the techniques that are going to be mentioned Within the tactics right bunch of them are repeatable to achieve different tactical milestones Okay, so a simple way to read this if you're looking at this for the first time a simple way to read this Table is the ones in blue the initial access execution persistence are tactics Each tactic has techniques that attackers use right and this particular table is focused on covenants Okay, let's jump into taking a look at what and I'm going to cover examples from I'm not going to cover all techniques That's just going to be a whole day session if I jump into each one of them I even in the time that we have let's take a look at some examples from each tactic I take me from each tactic and see what it looks like in the real world. Okay from initial access Let's take a look at one of the most common ones, which is the exposed dashboard right in initial access for the tactic to become a plugin you have An attacker figuring out an exposed dashboard right for the covenants I mean you say expose dashboard we're talking about the covenants dashboard right the screenshot on the screen right now that you can see was Taken by the amazing folks at redlock.io right when they were researching What kind of dashboards of the covenants dashboards are visible on the internet and they ended up on this particular Dashboard this dashboard belongs to Tesla right and this does not have any Authentication authorization simply browsing to that end point allowed them to browse the Kubernetes Cluster configuration right and from the secret they were able to pick out AWS SDA credentials, and this was pretty popular back in I think this was last year right Covered by a lot of major news agencies where Tesla's Kubernetes dashboard was exposed and that was the news that came out and I'm going to come back to this dashboard again at the end with an interesting story with the impact that they were trying to discover Things around crypto mining and we'll touch upon that when we reach the last tactic in the thread matrix But this is what an expert dashboard looks like on the internet and you could reach this either by scanning the entire internet and Tools available to do that or there are services like showed in for example And if you craft a search query properly you'll be able to find a bunch of these on the internet right This is another example of initial access where an attacker was able to gain access to You know the the fines required to create your own to config Okay, using an SSR and in the bug bounty community. This is a really popular bug because yes, this was an exit The SSRF was used to extract, you know image by image from the image itself The attacker was able to extract client certificate, the private key, right? They took in all of that then became part of the query that he made using Q-cuttle and was able to then gain access to the cluster At least to specific means spaces and if you look at the bottom most screenshot You can see that the attacker was able to get a shell into one of the parts that were running Right So application vulnerability definitely is a way to get initial access inside your Kubernetes cluster And a multiple large companies have in the past. We've seen this have exposed your communities dashboard without authentication to the internet right The Qubilator SAPI ports 10 to 50 which is read right you can make you can use a current request to start Upon in the name space in human space using if to 10 to 50 is open and then 10 to 55 From older Kubernetes, this still exposed 10 to 55 allows you to read communities configuration That's a really important right and both of these are widely still accessible for older versions of communities running up there One of the most common ways by which attackers can gain access to your cluster You're looking at vulnerabilities like any one of these are especially can make an approach request because you then use that to read configuration data and You know essentially you need either the access to a dashboard or you need access to the group configuration file So one of these like SSRF your LFI is part traversal arbitrary file downloads Right all of these vulnerabilities that allow you access to the back end would give you You know some level of access to the cluster. So that's what your initial access is about right Moving on to the next tactic that the backers use and execution being The eventual outcome of this and you've gained some level of initial access now You want to move to executing your custom code and the screenshot that we have That's visible on screen. The left screenshot is is a yaml description of a pod that when You know apply to your cluster would give you access to the underlying board itself Right and because the post the ID is shared Right You can actually access the process is running on the underlying right and new container being the Technique technique that I'm talking about under execution Once the ability to interact with the clusters within right regardless of how dashboard keep one thing five and this is a book hilarious in common one Simply pulling them off a developer's laptop using a client-side vulnerability, right? Or secretly quiet application vulnerability. I can't just be letting to gain a shell access to the cluster Sometimes the tactic can also lead to a privilege creation like in the screenshot that we saw You then from using a single execution you'd be able to gain access to the underlying node itself Right so from the node then you could move on and use Docker for example to if that's a container run time People who regardless of what the content momentum is actually Use that to see what other pods and containers are running access logs. You want to gain those privileges inside the cluster right so execution can overlap with privilege less creation Right next we look at persistence. That's the next obvious technique according to the threat matrix and The example that I have is of a Kubernetes cron job The screenshot on the left shows a program to scorn job that runs every minute and as soon as it runs It's going to get your river shell on the IP address that is specified within the unit Right, and this is a standard bashing if you folks, you know the standard river shell that we have One liner bashing that allows you to gain the shell execution capabilities to an endpoint This is because it's a cron job cron jobs create pods when trying to complete the class that is supposed to do The you what you will have if you look at the right side of the screen The screen shot there shows you have a river shell that has come in from the pod on to the attack on mission Right the attack of missions IP isn't what is specified as the cron job, and this is persistence You would have Even if somebody was sweeping through the cluster tension look at namespaces and pods the cron job Even if they disconnect Any outcome traffic the cron job ensures that the river shell jumps back to life every minute Right, so if there's some level of persistence that is created to the cron job for the attacker Riaz in your experience, you know, what are people doing when they get in I mean Do you have a sense of the kinds of of malicious activity is it? You know, I know there's been a lot of crypto mining and things like that But any other types of acts or what do you see? sick because the the feature of the capability of a cluster the most important capability of a cluster that is You know that attracts attackers is the ability to scale right so if Be just like a mining is such a popular outcome of an attack on on your cluster Because when an attacker runs their deployment that is actually going to do the crypto mining based on resources Automatically going to scale and that's profitable to the attacker apart from that. You also You know if you if you have if you have any jump boxes connected to the cluster for example An attacker might want to jump back into a different section of the network, right? And data is definitely something that attackers go after that's been forever Since the day the first launch Something that obviously Mining is kind of unique to Kubernetes like you say because you know, you've got these almost unlimited resources and the first time you find out is when you start seeing your Your costs of your AWS or Azure environment going crazy, right? Yeah The interesting thing is that For extremely large enterprise environments, right where they are AWS and Azure gc builds or any ways in the tens of thousands of dollars Starting a deployment that does crypto mining would you know would not raise a lot of eyebrows because the The builds that would come out would essentially be it looked like it's part of the workload build that you would get in every month Hiding inside So yes, the Kubernetes con job is Very nice example of how assistance could be maintained and other ways that attackers do but they're not limited again This is from experience. I'm mapping them back from the infra base days back to a computer cluster I mean SSH keys to be underlying node once it gained access. That's that's an easy win for an attacker Creating jobs con jobs again That's what we saw shadow admins definitely an attacker would create a cluster role or role that has the ability to administer the entire cluster And they would use names that you know would mimic something that the organization's infrastructure team has created Right, and we'll see a couple examples as well user right table most part bound to gain access to code to config file that the application is possibly hosting Reuse secrets and a lot of organizations trying to get to this a lot of organizations individual organizations tend to reuse secrets You know depending on what kind of workload there and I could be an app would be Keys that they're You know, I'm going to have across different staging environments or you know, I'm a dev environment But key reuse and secret reuse is is a definite thing that a lot of our functions still deal with right? Storage of secrets inside config maps, for example, which is not recommended but it's config maps are supposed to show configuration But some examples on the internet definitely show secrets being stored inside config maps So developers tend to do that as well and persistence from there is also is a possibility adding a backdoor binary To a running container. Yes, that's that's also definite technique that attackers do try I believe that the back is like to persist in most networks for the thrill of seeing how long it takes for the network admins to detect them and boot them out Right and I'm sure there is like a attacker forum where Like your Linux uptime, you know separated Essentially talking about oh, I've been inside this network for over a year and nobody's seen me yet Right. So that that's something that possibly attackers do or use the network as a means to you know further compromise Whatever is available for the organization and we're going to see a couple of examples of lateral escalation, right later on in the slides Or it could be simply symbolic of a profiler collection of networks that they've compromised Do you see people actually jaring any of this on the dark web? I mean is it has it reached a point where people will say hey, I got into this one and You know letting other other other hackers have that same way in There have been cases in the past and then you know Leaks that have occurred on Which accidentally other attackers have stumbled upon because they found it or pasted on Pastebin or something like that with the active light backdoor credentials to something to a possibly large You know network that was there Facebook had a bounty that they paid out for An application that was compromised by a bug bounty hunter But what the bug bounty hunter realized was post him compromising the application He saw a lot of shells that are already planted inside the application, right? And this this was also covered in the media I think it's a it's a definite possibility that attackers do share Trusted attackers, especially they shared, uh, you know their the trophies Are there already need some kind of access to a different network environment? Hey, here's one like take this off my shoulders. So that that's possibly that's That's something possibly is happening out there All right. So, uh, we have the next tactic which is privilege escalation, which is The example that I have here is of a cluster admin binding, right? and this is on my home clusters, uh, that is running and I created like a bunch of cluster roles That have the cluster admin binding to them, right? And the steps to create that are then the screenshot But essentially what this would allow is if an attacker is able to gain access to a cluster role All right, that has the ability or rather is bound to the cluster admin cluster binding The token that is generated from the secret the token of the extractor from the secret Would give them complete access to the entire Kubernetes cluster And you could use this token inside your code request that could be a piece of it You could create a cube unfit file, right? But essentially giving you access to the entire cluster and a lot of you know, third party Software that install any workloads that you deploy Right, um, cube flow being one of them for example, um, which is a standard ML AI workflow based Product on top of Kubernetes When you deploy that onto your cluster, and if you've not You know counter the number of cluster roles and roles it creates inside the namespaces that it creates But four or five namespaces the default installation path and it creates over about 70 to 80 different cluster routes And it's understandable because it does a lot of activity and it requires this kind of credential But a lot of those cluster roles have privileges that they ideally should not be having Right, so that's just an example in the wild and the dashboard Kubernetes dashboard being another one of them. It's required the community dashboard admin cluster role requires that kind of access But if an attacker is able to gain access to that token game over that they have access to the entire cluster from there And Privileges is an important step right for an attacker to gain access to additional resources or If the attacker wants to be in step mode, this is where they would sit around Right a couple of examples where these attackers can escalate privileges reading secrets tokens credentials to pull them out to write out On a different cluster that you've obtained And the token might possibly work the user keys might work The managed Kubernetes on awgcp. They have a service token mapped to The iam platform, right? This is accessible at the mouth point inside the pod If you've gained access to a pod that's an attacker inside you would then be able to access to the service account and the token itself Detecting with pods are running as privileged pods and then using an accept to gain access to those pods from there Moving down to the node right and then from there moving using a docker Come on to move to a different section of the cluster. That's again escalating your privileges and the more interesting one that a lot of attackers tend to do is Especially for managed clusters if you're if they've gained access to a pod access to the instance metadata can allow Reading of iam credentials if they've been attached to the instance for example, right? if you're running aws now if you've hosted it Your Kubernetes on aws not on eks, but on You know instances that are running inside aws and in potassium iam thread using a pod you could jump to The instance metadata extract credentials then move to the care right so that that's an escalation of privilege right there Defense evasion is something that attackers would definitely want to try once they've gained some kind of privilege escalation because hey now I have access to the cluster and I'm kind of administrator inside the cluster, but I want to ensure that I remain hidden or My traces of all the attacks the commands that have been trying they're erased right the example that I have the screenshot here again the amulet on the left is of Pod that has been created but with the name That maps or matches something else that's running inside kubes system. If you have code DNS, right? uh Inside kubes system if you do a couple get pods you'd see a bunch of code DNS pods running Right, but if an attacker creates which is what the right side of the screenshot is if an attacker Uses this yaml file and applies this to your cluster A pod is going to be created with the same name as a similar name as other core dns pods running And so going to be very difficult to figure out which ones The actual and which is the attacker sitting in playing hiding in plain site free Right, so for pod and similarity is a definite defensive vision that would act as to try the others being The ability to delete logs that pods create by gaining access to the underlying node And all pod have all pods generate logs that are stored in bad log pods and the same link there The underlying if you have access to the underlying node, you'll be able to take this part from Log files from from the director Right the kubectl delete events hyphen all is very dangerous to mention and never been unimproved But uh, this it kind of erases the entire event log of covenetus and across all namespaces Uh, the attacker could also launch pods and reserve namespaces to match up the workloads that that's what we saw with code dns And uh, the technique that uh, the treatment which also talks about is to hide the original IP addresses using prox right Moving on to credential access and coming to the uh, you know, the final parts of what the attackers tend to do They've now gone from obtaining access executed their code. They persisted using a backdoor Using a cron job. They've escalated the privileges using a cluster admin Binding they've eroded defenses by starting a pod with a similar name Right now they want to essentially try and take comfortable inside the cluster and the cluster that they've compromised They're going to now try and see what else is visible Inside the cluster in terms of secrets, right credentials are extremely beneficial to attackers From the viewpoint of that you can reuse credits if required And also they allow access to regions that you might not have access regardless of whether your status is the cluster admin or not Right applications, for example, if you want to access the application Uh, and from their access data, right? So, uh, what you have on the left is I mean trying to access the Kubernetes API endpoint without any token and then on the right I am using the service account mount point to gain access to the token and passing that through a code request Right and using that to try See what is visible to me as as an attacker The container service account is what I've used here to leak the token and this is mounted by default in every pod Unless you specifically have it written in your email to not mount the service account And the default service account of the namespace is mounted by default Right so Default service account is mounted by default that that's the correct way to say it Right so the token is extracted and then If the token is privileged you have access to the code and the API from there Most attackers will try and identify, uh, alternate regions once they've gained some level of access They would look at resources what is visible what else can you see from your vantage point inside the network, right? They try and reach protected regions which may require credentials that You know, otherwise they would they would hit a 401 unauthorized, right? Uh, some places where attackers tend to find secrets and credentials the Kubernetes secret store itself Using the service principle mounted within the underlying node, which is what the example is So, uh, the pod upon mounted credentials account token After within config maps pod descriptions hard coded within application sites, right? Any of these places where secrets can reside would be a good place for credential access to the technical company Discovery is a little I mean in my personal opinion discovery comes a little too late inside the threat matrix, right? And I say this because most of the techniques that this tactic has Are pretty much Useful to an attacker if they're done at the beginning of the threat matrix, right? And for the sake of completion, let's cover this as well. Uh, what I have here is a screenshot of and A batch prompt inside A pod running on a case, right? And Because the the pod has privileged. Uh, it's running in the privileged container I now have access to the underlying node, right? And if you can see the host name case agent pool, right? I am from there using, uh, I actually didn't need to do this because the attacker pod itself would be able to access the, uh Metadata instance endpoint address, which for AWS Azure and gcp is the same 169 to 54 169 to 54 And then extract metadata instance information the metadata instance information is extremely well documented by all three cloud providers, right? And, uh, you know, uh, a WC simply implemented, uh, the version two, right? That requires a header that has to be passed with a token But as long as you have a shell access to a place where the 169 to 54 address is going to be accessible, right? It doesn't matter what header requirements your, uh, you know, instance metadata, uh, requires You'd still be able to gain access to the instance metadata. Why is this interesting? The instance metadata is because This provides information about the instance that is running and the node being an instance, virtual instance inside Azure Has some amount of metadata information that's attached to it including, uh, where it is running what kind of, uh, You know, if there's any SSH public keys attached, right? With AWS, if you have IAM role attached to the instance, you can actually extract the credentials from The instance metadata itself, which you can then reuse into an AWS command, right? The secret key and access and token that is going to be available through the instance metadata Another common, uh, vector that it's actually the most common attack vector if you have uh, pod access inside an AWS environment, right? Uh, as I said, this technique is described too late in the threat matrix because most of the attackers will obviously perform additional discovery in the earlier stages A lot of these techniques are already covered, uh, you know In the beginning of the tactic, the threat matrix itself either via, uh, Cube config or service tokens access to be a K to say pay service obtained, right? If an attacker has reached this far already, uh, the privileged access level is assumed at this point in time Uh, we also saw the tcp ports to 10 to 15 10 to 55 That give the attacker read write privileges to the qbp server Depending on what total is accessible and what kind of authorization is applicable to them Right attackers also tend to perform additional network mapping using tools like nmap, uh, again Some attackers tend to do this. Some attackers simply would read the service and end point information using using the api itself Right, uh, that is pretty approximate of what's that's actually exactly what is running within the network So you don't need to do an nmap scanning the on the power of the service network from your access that you've obtained Right for managed operators again access to the underlying cloud platform is possible through, uh, you know, uh service Account that's mounted right or secret strategy through the incident data API endpoint From moving from pod to the node or from pod to the cloud directly Right, so that's that's what you'd end up doing in your discovery tactic. As I said a lot of those repeats The same applies to lateral movement as well if you notice the, uh, techniques that I mentioned in lateral movement Are bunch of things that we've already covered in the past Right because as an attacker when you move from initial access all the way to impact you tend to end you tend to do A bunch of lateral movement things parallel while you're moving through the attack matrix Right, uh, this is an example of a dashboard access that I have had Once I have gained access to a cube config I can simply, uh, you know, proxies the proxy access to the dashboard and access it from my local machine And, uh, the dashboard itself allows you to gain access to A pod you can gain a shell and then you have access to any configuration information from there Using this shell you can gain access to the cloud provider if, uh, this is supposed to run a managed cluster So we have we have a question come in About, you know, how do the attackers find the API endpoints and then how do they authenticate to it? I think this is A couple of slides back And see Yeah, okay, so We're talking about right at the beginning of initial access, right as I mentioned, um To gain initial access the attacker requires some, uh, sense of Where the cluster is located I could run an internet wide sweep to figure out, uh, common You know for standard, uh The API is hosted on six or four three exposed to the internet or four four three. It's an htps api server It's like any other API server on the internet, right? But identifying that it's a kubernetes dashboard is a tricky one because there's for the default Uh kubernetes api server, there's no teletail sign in the headers that tells you it is it is an You know a kubernetes endpoint, right? But I showed you an example at the beginning or where if you have priority and fairness for example features enabled A specific header comes up In the in the responses that tells you that this is a kubernetes dashboard No, sorry, and this is a kubernetes api server, right? And in most cases you do require authorization But there are a bunch of examples on the internet live right now that don't require authorization to access the api server And this could simply be honeypot fitting out there for people Like you and me trying to figure out what to explore it, right? But the fact that Uh, not all of them may be honeypots is a definite, uh, you know possibility Right and again obtaining access through a kubernetes config file because you've Compromise a developer laptop you've compromised Um, you know an application that's hosted on kubernetes from there The example that I gave was the sssrf bug in etsy from there accessing Uh, you know the environment variables that contains the certificates Like you could construct your own kubernetes config and then access the api server so Finding the endpoint is a port scan away or using an open source of information like shodown And authorization is that piece requires some work where you have to either find an xbox dashboard or A compromised image in a registry That's those are the techniques that's listed there and it might be ever being the application vulnerability You know using an sssrf or something similar that allows you to gain access Share access possibly to upon running inside the kubernetes cluster It sounds like the authentication getting past that is the harder part, right? You know finding finding them easier Yes, yes because let's see if you're talking about an application vulnerability an application that's running on top of kubernetes The authentication layer is never reached Right because you're using the application which is running on kubernetes as your window Right you you compromise the app. You're already inside The cluster you you gain shell access to the pod right and you're already inside the cluster And once you're inside the cluster you can access a bunch of things Based on where you are you could use the service token you could gain privileges go to the node Use docker start a bunch of other things move from there to the cloud Right again as I said unstructured, but you eventually go through each of these Tactics at some point in time All right, great. Yeah, we've got about 10 minutes left for you. I'll let you get back to your slide But thanks for that I'll just wrap this up the mitre attack framework reaching the impact tactic right the Essential idea is for every attacker as I said could be a symbolic trophy in their Bucket of clusters that have compromised which is the rare old version But there is obviously some motive that attackers go with right and we're looking at Either access to data. They're looking at resource hijacking in this particular case. This is tesla's dashboard Kubernetes dashboard Which the folks at red lock when they gained access to the kubernetes dashboard and they realized that there was already a crypto mining deployment that was running on inside the cluster right and The it took time for the folks at tesla also to discover this they actually figured it out when red lock folks told them essentially because The deployment that was running Did not start any large instances or was not using a lot of resources So they wanted the the attackers wanted the crypto mining to happen But they kept the footprint really minimal so that it didn't flag any you know monitoring that they had in place Right, so the kind of levels tell the So resource hijacking definitely something that and even I spoke about right So this this is a definite outcome of what attackers would look for Right the potential end games Some of them documented data destruction by rolling back deployments removing volumes claims if you're cluster admin You could possibly print on the entire cluster hijacking resources to perform monitoring tasks Ordering any state to create an analysis right any of these could be the impact that the attacker would be looking for Okay, that's This is our attack tactic that we've taken the route that we've taken to reach to the impact Right, and we are a quick poll and we want to get Let's see what everyone says about all right great. Yeah, this is our last poll and You know, so, you know, I guess that the question here is you know How do you ensure security of your cloud infrastructure in cluster today? You know, what are you already doing? So do you rely on the cloud platform providers to Protect it and trust the defaults. Are you doing any kind of you know pen testing and and internal reviews? Have you done any perimeter hardening hardening? implementing Least privileges, you know following cis best practices. There's a kubernetes cis benchmark out there that For example aqua's cube bench will help you with and then You're doing both numbers two and three with log monitoring analysis and alerting. You're all over this. So let's go ahead and give folks a 10 or 20 seconds here to answer the poll and then we'll share the results Sure Looks like we still have some numbers coming in Yeah, but I think this has been a great talk reyes. I know we'll we'll wrap up here in a minute But I think I've been a lot of questions along the way. We'll get to q&a before we before we go So let's go ahead and share those results You know people are pretty confident that they're they're there you know more than half saying they They've got this down and then a bunch more I would say relying on You know some core best practices in terms of hardening and privileges Yeah, this is what do you think about these answers? They consistent with what you see in the real world Yes, so Nobody trusts that cloud platform itself to Do the job completely right? So that was just a filler option that I was just what I see What's the distribution around two three and four? Four being I think in my opinion least that people Do in personally people that I've spoken to right? I can't see the head also What's the option with the fourth one? What was the spread on that one? What was the statistic? Oh, so yeah, so almost 60 percent said they were they were doing both, you know pen testing and the hardening and least privileges. Yeah Oh I think I'm talking to the wrong side of people Yeah, yeah, you're looking at your slides and your video. It's always tricky. Anyway, all right great Let's let's get on to recommendations. What do you how do you suggest people to go with this? Right, so I have a bunch of different tricks that You know over a period I've collected Especially when I was trying to figure out the whole Kubernetes technology that I'm coming from in offensive security world What would Kubernetes administrators do that would you know that prevent me from Gaining access of figuring out what what is the state of the cluster? Right keeping Kubernetes updated is is a no brainer. This is something that your cloud providers Even if you're trying it as a managed cluster, this is something that they would take care of as as long as You only upgrade bit, right? I think this version one got 20.4 Which is currently what Kubernetes is right and they have different patch Metrics right so some of them they support patches for a year. I think version 1.18 and below is for nine months, right? this just be Conscious of whatever version of Kubernetes is and try and keep it updated A restricting access definitely To the nodes to only trusted IP addresses is something that you should definitely practice Using network policies to prevent attackers from jumping between main spaces, right? If even if the attacker is compromised one part of the cluster, they should not be able to see something else Or even in the sea, they should not be able to gain access to it Losing, you know that Network policies comes to you right there Definitely audit your roles, binding service accounts, IM access, look for shadow admins, use your R back The cluster, the Kubernetes cluster provide judiciously, right? Audit all your cluster roles and any role binding for them in specific service accounts Either attackers might have created Scan images in your pipeline using tools 3D as I mentioned landing a wonderful tool To identify vulnerabilities that your images may have that may become potential security hazards Remove deployments that are no longer required and Definitely run pods with a security context where you have a user defined, right? Runs zero to one, say definitely say no to it Right, where today if you go back and look at your Cluster, then if you do a get pods See how many of them describe your pods and see how many of them are running as well That itself could be a problem Use namespaces to create logical segregation, do not store secrets In environment variables, especially pod description If you have them, ensure minimum visibility from the outside It's like a defensive depth approach that you take Definitely take application security consideration You could have the most hardened cluster, but if an application is vulnerable The attacker has a direct door inside your cluster Right, be wary of suspicious looking normal traffic Use hardening guidelines, CIS, benchmarks, highly recommended for Kubernetes And tools like Qbench that we mentioned earlier to identify potential misconfigurations Monitoring and creating an actionable roadmap Right into what a cluster is doing Who has what kind of access, what privileges, different components have And actioning any monitoring alerts that you've got Is a definite way to ensure your cluster remains 100 minutes At the end of the day I plan on writing a blog post with expanding on all the tips that I've mentioned here With examples on how you could do this And creating an actionable blog post For updates, I have followers with a Twitter handle That's our company's Twitter handle Closing the talk, our favorite Dilbert This is like a perfectly obligatory image that I have on most Kubernetes talks I've ever done Right You know the technology has become mainstream when Dilbert starts talking about it I'm sure this went over the head Of many Dilbert readers But for everybody on this call it's like oh man I'm right in the middle of it Yeah, but we love it I don't believe that Fine, so let's go ahead There are a couple more questions that we can look at here We have just a few more minutes But again thank you for sharing this We did have a number of people say they thought this was great And can they get a copy of it We'll be sending that out After So people will be able to see that And just to add The general files that I described in the presentation I'll be tweeting Like a pick up report of an ad Where all the emails are hosted So you can try them out in your test cluster at home And our Twitter is right here on the screen That's my Twitter handle And you can definitely reach me on my email address Take a look at our blog as well We have a lot of interesting blog posts that we've written And a full rundown of the MITRE thread framework as well And that's what everyone's going to get So there was a question Thanks for that I think people have your email and you can reach out A couple of questions here One, in a managed Kubernetes environment So something like AKS or EKS Isn't the node managed by the cloud provider And in that case, would the instance metadata end point End up giving up cloud managed node details That's a good one Even though the cluster is managed by AKS and AWS The master node is what you don't have access to But the worker node is definitely you can get a shell And that's the screenshot I had earlier Let me try and see what it is Yeah, this is a shell that I've obtained on the worker node And this is running inside a managed cluster on EKS So the instance information is aligned Is off the worker node of the managed cluster And so the information definitely is part of the managed Kubernetes environment But there is definitely going to be some information That is going to become useful to the attacker All right, we have one more question We're running out of time But what's the best Someone said they were looking for your recommendations on the best option For secrets management on AKS Any suggestions? I don't have any personal favorites I'm coming from the offensive background I've heard word is a good use case But I'm not sure I think a lot of people are using Vaulted In that environment as well Let's go ahead and wrap up Great talk It's so great to see from the perspective of the attacker And what it looks like And the really technical presentation In terms of how things really look and work So thanks again And for those of you You'll see a link in the GoToWebinar meeting Next week's session And the topic Which will be taking the work out of network policy And this will be our last Our 10th of this series And you'll have that And you'll have the link to register as well So you can stop by the AquaSec.com website And just sign up If that's easier But thanks again Riaz And I appreciate you sharing your knowledge And everybody have a great day Stay safe Thank you so much Take care Bye folks