 אני שעול בן חי, ועד היום אנחנו נסתכל על קוברנטיס בלין-ספוט והעדות של סיסטם-סיסטם-ספוט. אז, קצת על אני, אני סיכריטי-רסורציה, בפריזמה-קלואו, בפלו-אלטו-נטוורק, ואני עושה סיכריטי-רסורציה ותתתפעת בקלואו-אקוסיסטם, ואני גם עושה על אופן-סורסי-רסורציה ובולנרביליטי-מנגמנט. אז,our lineup topic for today will start to talk about container escape and Kubernetes second stage attack. We will explore this concept and understand why they are so critical for cloud security, then we will try to understand better system pods about their function and privileges. Following that we will talk about the system pods paradox, פריבליג'ה against risk ושיחות של מיוחדת האחרית עוד נכון לתאירון המלכון של המלכון אנחנו תתאר על תאירון המלכון בין שני מלכון המלכון בו ולמה אנחנו נעבור את הכי תלמידי ובספרות היו. אז, כשתמיד כשחררון היו נעבור את השכה, אנחנו נראה יותר וכי חררון והשאלה שאנחנו צריכים להכתיב על ourselves הוא, כדי שהכתיבה נכון נכון There's no doubt that containers are great for packaging and deploying software This is the reason why we all use them But they are weak security boundary Mostly because of the shared kernel And containerscape will probably continue to happen And it's here to stay Even because vulnerability Or even because misconfiguration The most known one is Village container with access to the host So We must understand what is the impact What is the impact of container escape And the obvious impact In Kubernetes at least Is a compromise node An attacker escaped from one container Or one pod And now to control over the entire node But let's imagine that our an attacker Is an ambitious one And he might be not satisfied with only single compromise node He might want to take over the entire cluster Maybe for more business logic Or more compute resources for crypto mining for example So one of the questions That we will try to answer today Is if a scenario of a single container escape Could be escalated to full cluster admin And what is the part of system pods In this kind of scenario So before we will dive in We need to understand a little bit better What is exactly second stage attack in Kubernetes So of course Attack against some initial access To our environment And now we want to escalate these privileges Or move literally So our starting point is that container escape Already happened So how exactly we make sure That our environment is protected and strong enough That even if container escape happened An attacker could not be doing Any significant damage So let's try to think as an attacker What we will focus and what we will look for In this second stage attack So as an attacker probably I won't focus on demon set pods Because Sorry, deployment pods As an attacker I probably not focus on deployment pods Because deployment pods And the application instance And compromise them Probably not allow me to access some infrastructure In our cluster And I really don't know where I land In the cluster So there is no guarantee That I will met with deployment pods Same for user configuration And user management pod The chance to find some misconfiguration Or vulnerability and well configure And well management pod Are significantly reducing And of course low privileges pod Our goal is to escalate our privileges And Give ourselves a higher level of control In our cluster So this is a list That what an attacker will not focus on And anybody want to guess Which kind of pod attacker will focus In second stage attack Demon set Demon set guarantee us That no matter what No matter where we land our cluster We will meet demon set pod in every each node In our cluster Same for non-user configuration And user management The chance to find any kind of vulnerability Or misconfiguration In non-user or non-management pod Are significantly increased And of course high privileges Because compromise high privileges pod Provise extensive control Over the cluster So which kind of pod fit to those requirements Exactly system pods In most cases system pods Deploy as a demon set There was already there when we launch our cluster We didn't configure them We didn't set their permission We just inherited them by creating a cluster And by their nature And in order to operate They need high privileges So we understood that system pods Are very special pods So let's try to understand and get to know them better So as a definition we can say that They are responsible for executing key tasks And maintaining the cluster overall functionality And play pivotal role in managing critical component And ensure smooth operation of the cluster And basically they are the architecture behind the scenes And with this kind of operation They need to evaluate privilege to perform this critical task So we all know that great power comes With great responsibility As this privileges is much better in nature We must have in order to operate But we must use it wisely To ensure the security and the integrity of our cluster So in the following slide We will explore a couple of different groups Of system pods in Kubernetes This is not all the groups But probably the most common ones So the first group is storage pods And do their function They need access to sensitive paths Which significantly increase the attack surface Of our cluster And makes them very attractive for attackers Because if an attacker can exploit vulnerability Or misconfiguration in storage pods He could gain unauthorized access to sensitive data That's stored on the host The next group is monitoring and logging and aggregation pods Maintaining healthy and secure cluster Require monitoring and logging, right? So those pods play critical role by collecting And aggregating data from various pods within the cluster But they also increase our attack surface And part of the collection process of those pods Require level of interaction with pods, nodes And the Kubernetes API itself So this group also expands the attack surface of our cluster Another group Is secret management pods Which play critical role in managing life cycle of secret Used within the Kubernetes environment And they have very high permission because they need access They have access to API token, password, certificate That's stored within the cluster So this group also increases our attack surface And the final group is network pods That's responsible for mapping network routing And enhancing within the Kubernetes cluster And attacker may target these pods to manipulate network traffic Intercept communication between pods or services Or even launch a denial of service attack So this table summarizes everything we said About the different group of system pods Which high privilege each group has How each group increased the attack surface of our cluster And why the attractive target for an attacker It was very easy to know that system pods are very special They are essential for core functionality But also introduce a security risk This is exactly what creates the system pods paradox The heart of the paradox lays on the fundamental tension The need to privileges to ensure core functionality Versus inherit security risk associated to those privileges Without system pods The cluster will not be able to function properly So they require avalanche privileges Included resource manipulation Like the ability to create, modify or delete resources Network setting like configure the pod communication within the cluster And the price of this power Is that any kind of misconfiguration in those pods Will be a critical misconfiguration That probably allow some unauthorized access to our cluster Which make those pods a very high value target As an attacker it's worth to spend some time To find any kind of security or vulnerability in those kind of pods Because probably they will allow us to move literally And in the best case or the worst case is depend Allow us complete compromise of the cluster And some optimistic notes This paradox is manageable With proactive approach that mitigate the inherit security risk We can grant them the only minimum level of access they need to operate And limit the network access of system pods whatever is possible And finally, and of course the proactive monitoring and logging system pods activity By treating every operation as a potential suspicious activity So it was a lot of information in very short time So let's do some short recap We talk about container escape that unfortunately will continue to happen And we emphasize the importance to shift the focus to the second stage attack What happened after container escape will happen Then we will dive into the critical role of system pods in Kubernetes And finally we talk about the security patterns That with great power comes great responsibility So now that we are all aligned We can move on to the artistic part of the talk The dual privilege escalation chain This chain combining two misconfiguration, two default misconfiguration in GKE And each misconfiguration Not allow an attacker to do any significant damage But combine them together Allow us to escalate our privileges and operate as a cluster admin So let's get to know the component of the chain The first component is Fluent Bit Fluent Bit become last year the official logging agent of every GKE cluster It's deployed as a demon set And just we describe at the table About logging and monitoring by their nature They need access to some privileges to every pod in the cluster Which increase our attack surface for privileges escalation This is the reason why I look for some misconfiguration in Fluent Bit The second component is Anthos Service Smash Which is the Google Implementation for Istio Project It's an add-on in GKE which need to be enabled As we saw in the previous table Network pods needs avalanche privileges in order to manage network routing And load balancing and pod communication So we have two prerequisite in our attack First is container escape Because we are talking about second stage attack And the second prerequisite is Anthos Service Smash will be enabled So two days ago here in KubeCon Google have a great talk about Anthos Service Smash And according to them the number of users enabling Anthos is simply huge And they are implementing it in more and more feature So it's not quite rare that Anthos Service Smash is enabling in our cluster The first misconfiguration is that the Fluent Bit Mounted the varlib kubelet pods volume And it's not necessary in order to operate And under this volume We can find the service account token of each pod in our node So it's nice We can map the entire cluster But our goal is cluster admin So what next? So with the first misconfiguration of Fluent Bit I can control the Anthos Service Smash container network interface token And the misconfiguration in Anthos That it keeps its high privileges after the first installation In the first installation the installation of CNI Need high privileges to create pod Configure the network and so on But after the first installation completed It doesn't need it anymore So this was the second misconfiguration That the CNI demonset retain its high privileges And after the first installation So I escaped from container And I met two demonset And with the help of the misconfiguration in Fluent Bit I can reach the service account token of the Anthos Service Smash Which allow me to impersonate And take advantage of the fact that it retains its privileges And create a pod Cool, very nice But not enough Because I want to be a cluster admin So Luckily for me In GKE The cube system namespace Offer a number of preinstalled Extremely powerful service account And the cluster role aggregation controller service account Which I call Crack Is the leading candidate As it can add arbitrary permission To existing cluster role So I can update the cluster role bound to Crack To possess all privileges So let's connect it all together I will grant the Crack service account To the pod I can create with the Anthos Service Smash permission And save the token In one of the volume folder Which I have access with the first misconfiguration of Fluent Bit And once I have control on the Crack service account token I could escalate my own privileges And operate as a cluster admin So enough talking Let's see it in action So, just a sec Okay I'm using here My own script that imitates Container escape And also installing Cube Cattle and static bash Another thing that I need to launch the attack And now I'm taking control On the Fluent Bit pod All the attack is operated From the Fluent Bit pod only Here I verify that I'm on the Fluent Bit pod And now I'm taking the advantage that Now I can see that I have only permission to To get a pod This is the only permission that I have in Fluent Bit pod And now I'm taking advantage The fact that Fluent Bit mounts The varly Cubelet Pods Volume So now I can see all the folders All the volumes of all the pods In my nodes So I'll go straight to the Istio CNI installer And look under this folder Where is token located So I find this token Now I will check which kind of permission I have when I'm controlling The CNI installer token So As you can see now I Have more permission and I Can create a pod So I prepare to advance some Pod YAML To create a simple pod And grant it The cluster wall aggregation controller service account token And immediately Escalate Its own privileges To be able to Do any kind of operation In any kind of name space In the cluster And save the token In some TXT file Under some shared volume That I have access to With the Fluent Bit misconfiguration So I will create the pod I verify that the pod is running And back to the Varly Cubelet Pods And search for this TXT file That will be under one of the volumes So I found it And now I can use it I'm going to the pod where this file is located And I will see which kind of operation Which kind of permission I have with this token And I see that my RBAC is full of stars Which means that I can do any kind of operation within the cluster And operate as a cluster admin Thank you So About the disclosure and fixes So at the end of 2023 GKE fixed both issue By reducing and hardening the Fluent Bit permission In no longer mount the Varly Pod Cubelet Pods volume And do some architecture change In the enter service mesh And now we cannot create pod Anymore after the first installation completed You can see the fixed inversion And for some key recommendation So just like in Marathon Securing Kubernetes is not a one-time effort It's an ongoing process that requires continuous Sustain effort and commitment System pods deserve special security intention Because they are the gatekeeper of our cluster And in order to manage the paradox We need special security intention And special treatment for those system pods Every interaction, every permission Every configuration Related to system pods Required justification And as it comes to system pods We need extra caution When configuring them permission So we must ensure that each action An access level is justified And aligned with the security best practices So this is a QR code To the dual privilege escalation blog That I wrote in last December It's a very detailed and technical blog That elaborates more about the attack And I would like to take any question If you have and feel free also To come offline and ask some questions