 Thanks everyone for joining us today. Welcome to the CNCF live webinar hacking cloud native applications through open source I'm Libby Schultz and I'll be moderating today's webinar I'm going to read our code of conduct and then I'll hand over to Arthur Oliarch security researcher with Palo Alto Networks a Few housekeeping items before we get started during the webinar. You're not able to speak as an attendee, but there's a chat box on the side Feel free to add and drop your questions there throughout the webinar and we'll get to as many as we can at the end This is an official webinar of the CNCF and as such as code to the CNCF code of conduct Please do not add anything to the chat or questions that would be in violation of that code of conduct And please be respectful of all of your fellow participants and presenters Please also note that the recording and slides will be posted later today to the CNCF online programs page at community.cncf.io under online programs They're also available via your registration link you use today and also on our online programs YouTube playlist post event With that I will hand it over to Arthur to kick off today's presentation Thank you Libby for the introduction so a good day everyone Thank you for joining this webinar My name is Arthur Liarsh, and I'm a security researcher for Palo Alto Networks One of the things that I enjoy I do the most is I like to look at different kind of projects which mostly open-source project and I like to peek into the code and Look into what the developers might do wrong and how could I exploit it and target application? Which are using those code in those packages or modules? So and today I'm going to talk about a bit about the security in cloud native open-source projects If you will have any questions by the end of this presentation Please drop them in the text boxes below and by the end of the presentation, I will try to address as many questions as I can So with that being said, let's get started Our agenda for today would be firstly we're going to go on a very high level on Why organizations shifting their applications to the cloud and how they shift in the services to the cloud? What is the open-source software role which helps them to achieve this goal? Also, we will take a look at some of the challenges and consortium organization might have when shifting the application to the cloud I will show you some vulnerabilities observed in open-source cloud native world and Show you even some examples on how I can use some of the vulnerabilities to attack an application which is running within the cloud We're going to talk about how we can reduce the risk and mitigate against those attacks even early in the development life cycle So let's get started Why organization choosing to move their application or services to the cloud? So there is a lot of reasons to do so and I decided to Briefly go over some of the reasons So it depends on how big your organization is it may reduce some cost when you're running your Service or application into the in the cloud. You don't need to maintain a server form You don't need to also maintain an IT staff, which is taking care of all the service, right? Which is quite trivial Also, you don't need to rent a physical space where you will place all the service and IT staff and will save some money On energy which always does consuming in terms of scalability you want that your IT infrastructure will adopt your business needs for example if you're planning to double the amount of users you have so if your service is Based on the cloud When your database need to grow you just space you go for more storage and you get it immediately and you don't need to build Before ahead of the time a new data storage or data or servers And you don't know if you're ever going to use it fault tolerance, which means If one of the components fails You will have immediately a backup available within seconds if you're running your service on the cloud And you don't need to maintain a physical copy Which is another data center somewhere of your service which cost you additional amount of money You just want to use your backup and when you need it and pay for it in terms of performance and regulations you want to be as close as possible to your audience and clients and Cloud service providers they have a lot of servers across the globe and this is how The performance can be improved because all the service are located in in countries where your clients are So like how Organization are achieving those goals how they start planning and shifting to the cloud So according to Gardner research that was made in 2010, which is a long time ago, right? But still somehow relevant to these days They came up with what's so-called five hours, which is a five approaches organizations are taking When they want to shift to the cloud So one of the approaches is called to rehost or also known as shift and lift Which basically means that you're taking your application which is installed on bare metal and you are reinstalling it into the cloud maybe you want to Containerize it and run it as a VM within the cloud right you basically taking our application and reinstalling it within the cloud The second approach is revise which basically means that you maybe want to take a small portion of your application and Adjust it to benefit from the cloud the provided services that what they have to offer Maybe you want to move your storage to the cloud for example The third approach would be a refactor where you are refactoring an application through Run on the cloud providers infrastructure, so you could benefit for more services that they have to offer The force a method is in my opinion more of a cloud native approach Which means that you rebuild your application from scratch you are taking all of your code and you rewrite your code So like you Building a new architecture of how your application is will look on the clouds You're basically you are being building for the cloud in the cloud and this is the place where open source software Tools cloud native open source software tools really it can come in handy being called because like you are going to Look for new solutions and sometimes you don't have the time to build your own cloud native solution from scratch So you turn it to some open source software available out there The the fifth approach I didn't mention here and it calls replace or repurchase Which basically means that you take your application you put it aside and you? bind a new application Software as a service which is will run on the cloud and made for the cloud So let's take a look what our open source software role with this With all this immigration process So first of all using open source will reduce your costs because now your development teams will need to find new solutions to new problems and adopt your application to To be a cloud native so in terms of development They don't need to build a whole new solution from scratch They just find the right tool and they adjust it to your needs You also don't need to pay for licenses for open source software, right? So it reduces some of the costs and because like your teams don't need to build a new solution from scratch It saves some time as we know time is money today, right? It's platform neutral, which is very important in my opinion when you build your solution with open source software You can run it on any platform. So let's say you're running your service on Some provider X and you want to switch later to provide your wife on some reasons you don't need to Rebuild your application or readjust it because your application is ready to go on any platform This is a really cool feature to be a platform neutral innovative in my opinion people who are participating in building new solutions and sharing it with the community They're looking to solve creatively some existing problems and They do it sometimes in a really brilliant way and they share it with the community and all the software becomes available to everybody So you really benefit from cutting edge solutions free to use There are different solutions to the same problem So you have a different tools which which basically solve the same problem and you just need the one which will suit you so you just not Having a one solution for some problem, right? Also, it maintained by people from a cloud native community where cloud technologies is not something new to them And this is where I find it really in educational because like when your developers start to starting that learning curve developing their applications For to be cloud native for example, if you are using some module or a package that you are not sure How you should implement it? So we just go to the registry which is open source For example github you open an issue and you ask a question and sometimes and some people from the community or even the maintainers We'll start to try to answer it or or people who already saw Or had the same issue will give you an answer that will can help you which and all of that together Makes the application migration to the cloud easier and faster So I saw some really cool research that was done by a cloud native computing foundation where they asked Some people from a lot of organization if they're adopting a cloud native open source projects within On production or they thinking to using it maybe they have some something or staging they're testing and The night numbers are really really high as you can see a Kubernetes is adopted by 96% of the people who answer the survey Which is in my opinion crazy. Also, I think that the numbers here are going to Be higher with the following years because more and more organization will shift part of their services All of their services within to the cloud So are there any challenges and concerns? That organization having when they're trying to shift their applications to the cloud so this kind of a trivial question because there are a lot of challenges and concerns and Let's say it just complex because now Your development team need to Rebuild your application from scratch and they are trying to ask a lot of questions and start like a learning curve They're raising a lot of questions like, okay What is a Kubernetes cluster? What's it's an atomic? How should I containerize my application? How microservices need to communicate between each other? What are role-based access controls? what the service may should we apply what orchestration tool we need to choose and All of these Questions have answers out there. So online. There is a tons of resources today where you can read and start building something but to take all of these pieces that available out there and Bring it together to make a working solution will take a lot of time which can slow down development process so lack of proper training were really slow down a development process and this is where like sometimes you start to Think or or maybe question yourself. Maybe I need to hire a new staff, which is more cloud Native oriented and stuff like that. So also, it's hard to choose the right tools because there are a lot of open source tools Available out there and you don't know which one best will suit your needs, which is also takes a lot of research All of this raises also a lot of security concern concerns And actually there is a cool survey again done by a cloud Native Computing Foundation where they ask organization about cloud Native security concern they have and one I want to mention here is a vulnerability management one Basically, let's say you built an application already you maybe you have it on staging or even a production and Let's say you applied some security out there to your application or service and you scan the images you scan your dependencies And now you have a bunch of vulnerabilities going on may have compliance issues and Vulnerabilities and packages you're using and you have like a long long list of vulnerabilities And you just need to solve them one by one So you kind of need to prioritize all those vulnerabilities and address those so this is not a trivial question Which vulnerability should I fix first? Because fixing vulnerabilities, it's very time-consuming and doing in the right way. It's very hard So sometimes just like updating the package version module version. It's not enough because it's can break things in your Development and it's not a great solution. Maybe there is a workaround available so all these things are really really time-consuming and vulnerability management is a very very hard task to do and this is I'm not surprised why it's like on top of the list also another Thing which quote my eye is that our organization been asked about security incidents, which is cloud native related. So Almost 50% Answered that they prefer not to disclose. So what will learn from it will basically learn from it that there was That pretty much high percent that that there was a security incident But they don't want to talk about it and the second one is the vulnerability exploited Which we will talk about today And I see this very important because when you're applying new technologies and new tools You are using open source packages and modules. You may be Using a not updated version. Maybe you are using a default configuration You can misuse many things and many things can go wrong and expose your a bigger Application and services to Security breaches and attacks So let's Take a look at vulnerabilities Which are observed in open source and what they are and what types of vulnerability is there so What I did I took a list of very known projects by clown native computing foundation which consists of more than 100 projects and I search for nvd database and For a github repository github advisories to see how many vulnerabilities can I find? So I found out that between 2021 to 2022 There were 164 vulnerabilities registered so and I try to To put them into a groups by type and we can see that there is three groups Which are really catching our eye one of one of them is like the Nile of service vulnerabilities Second one is code execution and direct retroversal Where where we're looking at direct retroversal and code execution vulnerabilities are almost always high to critical severity and you never want that your Application will be exposed to one of those vulnerabilities, right? So I Also think that if we will expand this research to all of the cloud native open source software the number will be very high and it's not like means that I Think like for example, I see I saw some Rising numbers between 2021 to 2022 and I think the numbers are getting higher as a security awareness Getting more attention in the cloud native landscape So I was thinking how hard is to pick some vulnerabilities from the list that I found to see and how Some application could be attacked. So I pick up some vulnerabilities from the list that I want to share with you and show how Some application running within the cloud can be attacked with them. So let's see some examples So let's say we have Microservices application running with a Kubernetes cluster and we have two services One is a public service and one is a secret service We have an envoy proxy which is handling our interest traffic and navigating whether to the public service or the secret service With some access policy applied. Also, we have a continuous delivery solution, which is a Argo CD Which is a continuous delivery cloud native solution And let's say that Envoy proxy is not up-to-date and it's containing non-vulnerabilities Also, let's say that Argo CD is not up-to-date and Argo CD server is publicly exposed with anonymous access enabled Just don't worry about the anonymous access and what it is. We will elaborate on it further and Just to say that anonymous access is not enabled by default, but we'll explain it later So in that in mind what can go wrong? So let me introduce you to a vulnerability that was found for envoy proxy So this vulnerability, it's took advantage of a URL path normalization. So the way URL path is treated is very important because once they are not normalized and treated well it can expose you for a different variety of security vulnerability one of which is the past reversal and By that I mean Where is no where is there is no a past normalization in place? For example a relative path like route public and a segment with double dot and admin can trick your access access policy In some systems a double dot segment means that you want to go to the parent directory Which in this example means that you will simply go not to public admin But double dot will go up to it to a parent directory Which is route and then you will end up with route admin if there is no past normalization in place If you were applying a past normalization It was ending up with route public admin without the double dot segment because it was sanitized and then you were not granting access to the resource So with that in mind and the attack scenario would be an attacker would grab a request with a crafted URL containing Relative Relative path containing double dot segment is the proxy will receive this URL and due to lack of path normalization The access policy will be bypassed and we will get to the research we should not get to So an attacker will see in an HTTP request containing crafted URL let's say your shop com public service double dot secret service and when it will be parched by a proxy It will say like okay, you want to go to the public service, but where you want to reach within the public service Let's say access access policy is allowing us to go whenever we want within the public service So it says okay. We're in the public service you want to go So it sees the double dot and secret service and since there's not normalizing the path It goes up to the root directory and navigate to the secret service and that's how we bypassed the access policy So if it will go into straight to the secret service by your shop com secret service We was denied the access by the access policy But as simple as that using double dot segment, we were able to bypass the policy and get to the secret service So this is a classic past rehearsal attack another vulnerability I want to share with you it was discovered for Argo CD project and more specifically for Argo CD server So this vulnerability could elevate privileges taken advantage of Jason web token authentication and basically Argo CD Trusted invalid claims within the Jason web token Basically Jason web token is a token which made up of three parts. It's coming in encoded part within the authorization header and Once decoded it gets the following Structure, it's consist of a header payload and signature Rather when the payload contain the claims claims are just parameters describing things about a user In most cases, it's a user. So for example, it can state that the user is not mean for how long this token is Valid and etc. So once Argo CD has anonymous users enabled Which can be enabled via the Argo CD config map yaml file the Anonymous users will get the default role permission specified in Argo CD Rubbock config map and Rubbock is just a feature which is restricts data to Argo CD instances and Default role basically means that you will have a read-only access to all of the Argo CD resources So in that in mind if we continue reading the description of this vulnerability We will see that the attacker window need to have an account on Argo CD instance to explore this vulnerability Also, he can impersonate the if he can impersonate a user a built-in user Which is admin account on the Argo CD and when he's escalate the privileges He will be able to get the privileges the same privileges on the cluster as the Argo CD instance Which is by default the cluster admin. So basically with forging a GWT with some Forging GWT claims, which is just with talking claims, which I talked before Which I talked about before you can become a cluster admin which basically allows you to delete and add Or or manipulate in some other way as resources on the cluster, which is pretty crazy So the attacker will send a request with forged WT containing invalid claims and When Argo CD authentication will trust the forged GWT token It will allow us to become a cluster admin now. This is really crazy once This can if this can happen This is a really cool vulnerability also because Argo CD sometimes receiving and full information from some GitHub repository, right and this is where another attack vector That can be shown to the to the attacker now now he maybe can later exploit all of the CI CD pipeline So another thing is that some vulnerabilities can hide And if you remember the first vulnerability that introduced you is was that passed through our solver and ability within and with proxy So what's so big deal? So this vulnerability was first reported to Istio in February 18th by security researcher, so Istio Investigated it farther and they found out that the issue is within the envoy proxy Istio If someone don't know it's like a service mesh where it's injects and void proxy as a sidecar to each Pod within the cluster. So you see how it's complex It gets one project which is using an envoy proxy becomes vulnerable Because the other project have been normalization issue so Istio reported this issue to an envoy proxy team and in what proxy and in white opened publicly available issue on the GitHub repository and If we will look closely at the issue that was open So we a technical person can read through this and say hey I can I can pretty much understand what's going on And I maybe can try out and write some and pass through our cell exploit for and see if it's working And if I'm an attacker and I know that My target is running Cloud native proxy even if I don't know if that in web proxy or other proxy Out there. I maybe want to target all the publicly available cloud native Proxy repositories right and see for a scan for issues and pull requests that are available publicly To see if there is any hints that can point out to some security witness, which I might exploit So after the issue was realized and the severity was realized by the more team They moved to a private fix process which is was already after 20 days after the issue was pushed So there was a 20 days gap. So someone could pick up an exploit And if you do even if you think in that attackers is this not enough days for an attackers to do something So there was already Not related to avoid but there was some Stories out there that some some some things escalated less than two days So Hidden vulnerabilities as whole often subject for example the vulnerability. I just talked about Pastro Velser and what proxy this issue on GitHub was available before even the CV was assigned to the vulnerability so this is a really big subject and Two buddies and colleagues of mine have a great talk for Linux foundation Which is called hidden vulnerabilities in open source, which you can watch by typing this hidden vulnerability and open source on YouTube or just check for the link below Just another small thing to add Not always there are to do this hidden vulnerability subject Sometimes there are vulnerability But there is no CV and the reason is that sometimes developing teams are not aware enough that What they fixed was a security issue that can be exploited Unless someone points it out to them so there are a Lot more reasons to why this happens and There it is like explained in the hidden vulnerabilities in open source Talk so you can pick and see more information So how should we reduce the risk? How should we mitigate as much as we can against those attacks? So first of all, it is always great to have a map of your application and its building blocks You want to see which types of packages modules? or tools you're using within your solution and then you want to check for updates and release notes for those entities and see if there is some release notes related to security that you might address maybe there are Some work around and no fix yet. If there are fix maybe you want to patch and update the version also strongly advice through to go to Security advisory of your of the projects you are using most of the cloud native open source project are Repositories are on GitHub and GitHub is maintaining a security advisory you and this is a publicly available to have also API and You can search for a project. You're using a module or package and and see for non vulnerabilities also, you can go to NVD and look for non vulnerabilities and CVs within that database Also, there are a lot of scanners in open source available. So you can scan for images or dependencies so you can see if your Direct or transitive dependencies have some non vulnerabilities, right? So as you can see For example, we talked about Istio and the vulnerability for envoy proxy was reported from Istio team because Istio team is making use of envoy. So Scanning for that is really really important another great tool is SCA tools software composition analysis tools are There it's like a Methodologies that show allows you to track all of your open source modules packages and tools that you're using within your project And also those tools sometimes really Developer friendly so you can think of how you can integrate them into your CI CD pipeline So you can regularly scan for those dependencies and packages and modules and packages and discover the vulnerabilities early in the CI CD process and software development lifecycle Also, those tools will help you to prioritize risk. We should talk before right? So one of the issues and great issues for security teams or engineering teams is that after scanning for the Vulnerabilities, they have a long list of vulnerabilities. So they need to know what to prioritize first And and what to fix first sometimes it's not about the critical vulnerability But it for the vulnerability that this can be mostly exploited on your container For example, if your container is exposed to the network and running those high privileges even medium severity vulnerability can be a really danger Also all this information which is gathered And Can be shown through a CA tools are maintained by security researchers behind those tools, right? So if you remember for example, there's hidden vulnerabilities topic that they talked previously There are security research team which dedicate the time to find those vulnerability and then they integrate this in the CA tools and once you're using them you'll get trashed for those vulnerabilities even before they get assigned CVs and this is how you can really really mitigate and Reduce the risk from your application being exploited or exposed to cyber attacks so This all I have to say for the moment Thank you for listening and If you have any questions, I would like to answer them Thank you Libby Okay Anyone have any questions they want to pop into the chat for Arthur Give them just a minute or so see if anybody comes up with something. I See some problem in the chat. So I don't see the chest I'm watching it and I don't see any questions give everybody one more minute. Is there anything else you want to add any? Ways to contact you or places they can opt in to find out more information Well, if no one has any more questions, thanks everyone for joining us. Thank you Arthur for the presentation and These will be online later today The slides and the presentation will be on the website So thank you everyone for joining us. Thank you Arthur and we'll see y'all again next time. Yep Thank you for the opportunity. Thank you everyone. Have a good evening. Good day