 Hey, everyone. Wow, I'm loud. OK, welcome back. Hopefully you got some coffee and water and whatever else there may have been out there. So we're starting up again. So next, we have Moshe Zioni. And if you hadn't heard about the Argo CD Zero Day Vulnerability, you are going to now. Thank you. So hello, everyone. Thank you for the introduction. I'm honored and pretty excited to be here. But to be honest, this is the first time since COVID that I'm actually speaking to a crowd. So that's amazing by itself. And thank you for being here. We're going to dissect the discovery of the Argo CD supply chain attack. First of all, my name is Moshe Zioni. I said VP of Security Research at APIRO. Doing app security for the past 20-plus years on multiple fields. If it's with the app stack, in-search response, DFIR, malware analysis, and others. And if you'd like, you can contact me on email or Twitter. I'm active on both. What are we going to go over today? Is, first of all, to lay the ground for the upcoming items of what is Argo CD. We're not going to go full of a full course of what is Argo CD, but just the basics that will have a common ground to start from to understand what is the attack originated from. Why does the vulnerability that we are going to discuss here matters? Maybe more than other vulnerabilities, maybe not. And then going through the discovery itself. What are the discovery steps, some basics about how our vulnerabilities are being discovered, but also how this specific vulnerability, what led to the discovery of it? Eventually, of course, to tie the knot over everything, we're going to discuss the attack scenario and some extended scenarios that, to be honest, we didn't discuss up until now. And then go through remediation steps and suggestions that we can go with long-term remediation as well for the better of the Argo CD project and maybe other projects. We're going to sum up everything. And of course, I hope that we'll have some time for Q&A. So let's start with the basics. What is Argo CD? As said, Argo CD is a CNCF project. It's covering GitOps automations. And by that, continuous delivery specifically in open source, of course, which is pretty unique in this arena. Most of the CD systems are not open source. And Argo CD is maybe the most prominent example of that as an open source project. Then it's also true cloud native, meaning that it's built for the cloud. It's built to be on Kubernetes itself. And by that, the platform is very native to many cloud applications that would like to have this kind of packaging on common languages, manifests, and tools. The project itself is using multiple, very common open source projects by itself, like JSONET, KSNET, Helm charts, especially. And by that, this makes the curve of loading and implementing Argo CD a very, very efficient one. Lastly, it also allows multi-tenant, multi-users. And by that, we can conclude on what is exactly Argo CD is providing for organizations more than just a CD pre-system, but also covering a lot of security aspects of the whole ordeal. By that, when you are discussing security with Argo CD, we are discussing what kind of storing we'll have for our report stories. If we have a role-based authentication or we have something else, how do we deploy those kind of instances, especially when we are discussing different environments? That's where GitOps comes into place, because we need to tweak those kind of variables and by that needs to be automated, or at least as full as we can to be automated. And by that, Argo is providing this kind of health monitoring as well to those kind of instances. Secret management is part of the Argo CD, I would say, umbrella term, not specifically for the Argo CD project. There are several plugins and stuff that are connected to the Argo CD ecosystem that will provide you with secret management within it, but if not, you will also have the native way of managing the secret and we're going to get deeper into that because that's the culprit of the vulnerability. Lastly, it is widely connected to different CI and other SaaS systems, and that's why Argo CD is so good for many purposes. Why does the CVE that we are discussing right now, 2022, 24, 348, is important on matters for us, especially for me and as we are discussing with many of our contacts about supply chain attacks and of course this has been trending on the news, but more than that, there is a specific item of why it's being trending. The why is that we have much more detections and we are much more aware. And for me, this is an awareness act, like being into the, discussing the supply chain attacks is very, very important. That's not to say that this vulnerability is the mega vulnerability of everything. It's not. That said, it is a high impact vulnerability and should be discussed in a very serious manner, but not to exaggerate or over hyping it. It is not something that you need to call home right now and to understand if you have something, take the time to understand if you have the limitations and we'll discuss remediation at the end. A pattern that it targets the critical aspects of a supply chain, not just a supply chain attack, but it also covers the CD system. By that, it comes with a very native way of attacking that. So that means that you don't need a special port to be open, you don't need something that is out of the default, but you need some instances of algo CD that will be discussed further and by that kind of infrastructure that makes this very, very much interesting and compliant to the trend of not just attacking the supply chain attack, but also by the supply chain infrastructure. And lastly on this kind of list of why it matters, the utilization of the payload itself is also interesting because we are utilizing Helm charts specifically, but also we'll get to the extended diversion of what kind of other attacks we have and you'll see that everything that we are covering here is, as said, is using the most proper way of utilizing the CD systems, namely again, the Helm charts, and by that, you'll see the extension as well. Lastly on this slide, the availability encompasses a lot of what we discussed so far about the CD systems and its culprits and why it's very important for us to follow up on those kind of attacks. It was fixed on January this year and since then we have seen much more interest about supply chain, not because of that, but we see more and more instances of those kind of attacks happening because it's a very lucrative attack path for attackers in terms of merely ROI. Lastly, as we'll see, this specific vulnerability was actually predicted by the developers, so the developers were ready for this kind of attack, at least in the metagame, so they will be able to comply with that. That said, of course, nothing is perfect and by that we found a way of circumventing this kind of defences and throwing it all together. So let's discuss the discovery itself. How the research strategy goes, nothing more different than anything else. That said, there is maybe some basics that researchers can agree on. What is the basic research routine? I split it into three just for convenience, but this is more of a personal thing, but again, we are going over that. But in general, we go through a phase called exploration. We just want to scheme through and to understand the ecosystem, to understand the code base. Remember, I'm not a developer of August CD, meaning that I'm not familiar with the code as much as the developers are and the community itself, which is a huge community. So we'll have to go through this kind of exploration step to explore, to understand the ecosystem, to talk the talk at least, to understand what is going on. Manda, that we are going through documentations and we are going through other items that will provide us more insights about how development occurs, because we are into the thing that is called, we can't go through the old code base. You can do that, but first of all, it's very boring process and it's more of an automating process, but more than that, it's not efficient. For you to discuss actually what needs to be secure, to have security reviews, you will have to consider what is deserving security review. That means that you have some kind of a risk level for each commit maybe, to each feature that you have and by that to enforce the security review on those specifically to some degree that you would like and by that you'll be much more efficient and that's exactly what we did right now, right here. We understood what is going on, what kind of development we're on, the August city platform until that point in time and understood what is wrong or what is the sniff test is called, what is smelly a bit, so by that we'll follow up. After the exploration step that we just covered, we actually go through the tutorials themselves and build the model, meaning that we actually have instances of August city at our lab and by that trying to of course play with that, a very basic routine. But more than that to understand what is extended to this kind of ecosystem, what kind of plugins there are and what is the default systems, what kind of tweaks you can go through, what is the basic things that many people are going through once using August city. This is an important part because first of all, once you have a vulnerability, you would like to ask yourselves how common this vulnerability is, how scary it is. And second, if not common, what kind of scenarios will need to be converged into the issue that is handled right here and by that the vulnerability will take effect or maybe a greater effect than before. So this is the second step, to actually realize the use cases, to draw them out, to have this kind of infestation of flow charts. And it's also important because as we discussed before, we need to be very focused on what we would like to research because we would like to be efficient. And by that once we have the use cases placed out, we can actually say, okay, this is the use case or the logic we would like to attack. And remember, once we, someone discussing the vulnerability, many turn to talking about the actual vector of attack like a remote or code execution or the health charts. But that's merely important if you don't have the realization of what is the business logic that you are going to attack. So the use cases are based on business logic, mostly most of the time and by that you are following up with the dive into the mechanics of those kind of business logics. Some very basics, maybe terminologies here, sources and six is a very important part of it. To understand once you have this kind of use cases, you would like to see where, what kind of impact it can have at a maximum, like what are the sync of it, that's called sync because the function itself syncs into something like file system maybe, maybe some kind of a network traffic, et cetera, but the source itself is the input that you are providing. So from now on I'm going to call it input parsing just for the sake of it, but remember this is a very thorough method that everyone has. And lastly, development history is very important because you have much more context, you are covering that as part of your exploration, but once you have this kind of homing down into specific use cases and by that diving into those kind of mechanics, you will also want to have a very good understanding of what followed until now to understand what kind of development history were under those kind of features. And lastly I said, what is passing the sniff test? This is more of an experienced thing, but it's necessary to understand what kind of attacks I'm presuming I'm going to look for in those kind of logics. And maybe something else triggered the sniff test as we see right now, something else in our case triggered the specific sniff test to be positive. Okay, so first of all, we are going to go through the parsing itself. So we hung down into specific use case, but maybe the most common use case of August CD is getting a manifest in and by this kind of manifest it's being parsed by the HelmChart itself is being parsed into those fields and the fields themselves are acting upon that to deliver the application after that, of course, to maintain it on the cloud. Now once you have that, we are going through the actual HelmCharts and understand what kind of quirks we are going to find in this kind of HelmCharts. Now remember that those kind of charts are controlled by the user. So the user can provide August CD with any kind of chart that she wants and by that to follow up on what kind of build that she wants to build by that. And we found by that the installation itself consists of a cluster. If you have the common input of the HelmChart is providing you with all the manifest that you need. And then we found out that the, an actual discussion that went on the threads themselves about something called the value files on HelmChart. This is a very specific part of the standard that the value fields is a list of values that will give you either excuse me, value files, give you a list of file paths. It can also be an HTTP path. So we are going to go back to that in a second. And those kind of paths are useful because you don't want to have all the secrets on all the values into one place because if you have different environments and if someone else, like a different team, like DevOps may be controlling this kind of value file and you don't want to mix the two. And it's a very good practice in overall to have those kind of things split it. So if you have the value files list, the actual thing that will happen by the algo CD is that they will take those kind of strings and going to search for those kind of files or websites and read those values from there. The values can be by the way either JSON or any kind of YAML. And by that you have some variability about what can be controlled by those kind of value files. Now why it's interesting and why it's called our eyes because it's important to understand that the actual repositories that are holding on the algo CD system, the algo CD is built into pods. One of the pods is called algo CD repo server. And this repo server is monolithic, meaning it has some kind of a file structure very deterministic one. So you can predict those kind of file structures if you know the application name, that's all you need. And by that, if you have some kind of a path traversal to another file, you can read other files on the systems. Now fortunately enough on the common system of algo CD there is no YAML or JSON file that you can just read from the system and gain some kind of a God mode control. That said, if the system is multi-tenant and multi-user as many of the common infrastructures are you can potentially maybe read other files on the system. Now that's why it's interesting. That said, it wasn't also interesting just for us. It was interesting for the developers in 2019. We found a thread that was discussing the exact case of this kind of exploitation. For one, the developer wrote, what is concerning me here is this type of behavior can be used as a directory traversal attack and he is completely right. Potentially this can be used as a traversal attack. He shows here a dot dot slash which means it will go through some kind of a parent directory and so forth. And they started to discuss how can we manifest that into code and how can we protect on a sanity check on those kind of values. And they commit those values and put us to the team there. It's a very great starter to begin with. So you have this kind of notion of security within the developer team. And they implement this kind of sanity checks on the files themselves. Let's map out what kind of sanity checks are going on the system and by that also how can we follow that. So first of all, there is one file named path traversal which you can, of course, understand what it means to do which goes through several security checks and the security checks themselves are waterfall based. So if one, it doesn't have a tool, it becomes falls in the actual test. It can be broken, it can be maybe something else but they try to do it, it's a fail open meaning that once you have something you will kill it as much as you can in terms of cleanup and that way it will be safe to use most of the time. So the first phase and the most important phase is what is called file path clean. File path is a Golang library and the clean is very common way of cleaning up those kind of strings. Once you have a string, you would like to have it very common way by the standard and this is what clean does. So this is the first and maybe the most important phase of path traversal cleanup. The second one is the parse path that's that way you are going to split the path itself on the file component. And step three is that there is requested under current route which means that it's going to compare the path that you just broken up for the actual value and going to compare it with the actual PWD, the directory that the system is residing or reading from right now. And by that if they have a match it's okay. It is a subdirectory under this directory and if not it's going to fail. Okay so those are the three steps and by that I omitted the last part it's just going to fix the trailing. If it was just a directory it will add a slash at the end. Now so far so good. We've actually implemented just for research purposes. We've implemented this code, a very light one on a different script and we actually ran through those kind of steps many times on a high level debug system. This is a very common practice as well so we have a very rapid way of fuzzing it maybe manually but at least you have this kind of, you are cleaning up the table from any kind of noise from other system components and by that you can differentiate what is actually happening. So we haven't found something very, very crucial on this kind of path. So the path traversal on the face of it works. That said there is another part of this kind of cleanup which is very, very important is the repository actual reading of the repository and by that the helm chart itself. So where it starts on another way to look at it what calls the path traversal.go the actual function that goes there. So the first thing that happens once the value files field is shown is going to iterate through each of those kind of list items and each list item is going to go through parse request URI which is maybe the most important part of this presentation, parse request URI is the thing that's going to split through what is going to go through path traversal and what is not. Other than that it's going to check if the path is absolute and not relative and lastly if the enforced to current route is happening or not. If it's not a no current route it's going to change it to current route and then load the file and what is happening again is going to check for the path is absolute is going to go through the security checks and by that going to return to the original code and then load the value file if all went well. Now why I mentioned the parse request URI is very, very important because as mentioned before those kind of values can have HTTPS values as well and for a good reason maybe you have those kind of values somewhere on GitHub and you want to pull from there as said maybe it's a different team maybe it's a different package you would like to load and by that you will have this kind of control this is a very basic GitOps and very common GitOps request and why it's important because here is the step that you would need to be very, very careful and really understand what is going on. So reading the actual documentation from Golang parse request URI is as I said, this is a Golang a library nothing related to AlboCD internally that the URI parser is parsing a raw URL into a URI structure it assumes that URI has received an HTTP request so the URL is interpreted only as an absolute URI or an absolute path. If you noticed you can somehow mix the two and think that the default of parsing the URI is going to be the absolute path but that's what will happen if I'm going to present an absolute path to the parse request URI simply put the parse request URI will think of it as a URI and by that we'll ignore it as an absolute path altogether so I'll be able to input absolute path and it will never go through the path reversal go functions and by that that's pretty easy from now once you understand those kind of application repositories are deterministic that you can foresee them in terms of if you have the application name you can construct the whole path to it and by that read those kind of secrets and also not mentioning here that the actual error returned if you have an error it's very verbose so you can understand from it a lot about the system structure a lot about the files that are residing or not residing and by that you can enumerate those to have a better homing down on to the target that you have in mind and as said all of that together comes into a circumvention of the actual platform and by that you can read secrets of the other applications and as said we're going to go through some extended if you have the time some extended attacks besides that but this is the most important one and the common one in our discussion to close up the attack flow so an attacker which is a user or an authorized user on the system and a different authorized user on the system logs in into the system and provide it with a malicious Helmchart YAML and by that the Helmchart will point into different files it can be like thousands of files and read a different files on the system and once a file is processed as if the system can process a file it will load those kind of values and will gain insight in those kind of secrets and again it's going to because of the absolute pattern that we are going to provide it with it's going to traverse the path traversal traversal and path traversal algorithm and by that is going to actually get to the files then the attacker will read it off screen or under the API because it will be present on their new application that they try to build with the Helmchart and the whole circle comes into close the extension of those of this attack is maybe two extensions one is that would not mention here but there is another variation to the attack that the actual August CD people found by themselves that there isn't a possibility of the Helmchart to be presented as a compressed Helmchart and a compressed Helmchart can also be using a potentially a SIM links within the files that are within the compressed table and those kind of SIM links can also be interpreted to different paths and you can point to different paths on the file system but again this is a much more extended I would say a bit more complicated but a very valid attack path the second one is that not only repositories are residing on the repo server and one very crucial example is that plugins are also can be presented or at least plugin files can be presented on the repo server itself I won't go into details of why but this is again a very common scenario once you are installing some plugins on the system and if those plugins have values you would like to keep a secret those can be read if they are under JSON or YAML that can be read by the same attack path remediation so first of all as I said remediation is the first of all you have to have a multi-user on the system you need to see that you don't have any other users that you are not familiar with or not necessary users that will minimize the risk and of course if planned for an upgrade for the August CD system starting from 2.2.24 and above you'll be the CVE is not relevant to those in general the path reversal a traversal ingestion system was completely replaced as an overhaul by the August CD folks and by that the attack is not relevant at least for the time being and this kind of attack is not relevant to the old path reversal issue and it is much more robust right now and a bit more generic in the way it was reading long-term remediation is more for the August CD and the community but of course the monolithic and flat in terms of permissions file system for the repo server it's a very dangerous culprit for future attacks as well so I would like to see some kind of architectural change and we're actually discussing some of those on kind of several threads and lastly implement the file level permission model for the especially if you have an ARBRAC if you have something that you can control or maybe from user perspective so you'll be able to split those into different levels and for organizations audit your files, audit your users so you'll be able to catch those kind of attempts once something is happening and does not supposed to be on the system. As a summary and conclusion for that Helm chart YAML itself manifests can contain malicious fields those malicious fields can lead to secrets revealing on systems that are using multi-user approaches and since sensitive data can be exposed by that the path traversal was suspected and also of course the big takeaway here is of course you need those kind of tests by the community and if I can contribute something to the community this is my contribution and a special thanks for the August CD folks that were more than amazing I've been through maybe 10s of those kind of CVE disclosures this was the fastest most professional engagement I had in years so thank you for that for the seriousness you take and professionalism you take the security of your users and ecosystem so thank you for that and thank you for the organizers to have me on stage thank you for listening.