 Thank you. Thank you so much. Thank you for the introduction. I really wish I could have been there in person, but unfortunately, travel is still a little elusive, at least to Canada at this point in time. Nevertheless, I'm really happy to be here. This is my first ever talk at NotSec and I'm really looking forward to it. I'm looking forward to hearing, getting some questions and comments on the talk that I have as well. So I'm going to quickly get to it because I have 30 minutes and I don't have too much time and my talk is loaded with a lot of content. So I'm going to quickly start sharing my screen. I'm guessing everyone is able to see my screen. Yeah. Okay. Sorry about that. I think you're, yeah, I'm presenting from one screen. So I'm, this is unlike probably most speakers who are going to be doing it remotely. I'm just going to start my slide deck and I'm going to get started. All right. So this talk actually is a culmination of a bunch of research efforts from 2021 that is now being showcased all over. So this is called Hookline and Sinker Pillaging API Webhooks. Now, webhooks are something that a lot of us use and I'm sure members of this audience as well are quite familiar with what webhooks are. We're going to quickly get into what it is and how it works and all of that stuff and then I'm going to be looking at attacking it and some case studies around how we actually went about this particular piece of research. So just a quick introduction. My name is Abhay. I work for a company called V45 and I'm also, I'm also the chief research officer of AppSec engineer. We run a training platform on AppSec, cloud security, Kubernetes and so on. I do a lot of training. I do a lot of talking. I do a lot of research on mostly defensive stuff. So this talk actually is a departure from my typical style of talks, which is way more defensive than offensive. So most of my research is defensive, but in this case, the talk I'm going to be focusing on is explicitly offensive. So I'm going to get started with it. If you're interested, we have a very interesting YouTube channel where you can check out a lot of interesting stuff on AppSec and so on. And if you want to check out my blog, that's there as well. I highly encourage you ask your questions on Twitter because I am currently suffering from a back injury and I'm not going to be there for a Q&A session on the NotSec channel. So if you want to ask questions, please reach out to me on Twitter. I'm going to respond. I'm barely holding it together, sitting and talking right now. I'm just on a lot of pain. So that's, I just wanted to let you know that if you want Q&A, please reach out to me on Twitter. That's the best way that I can respond to you. All right. So my talk today is going to be full of memes. I'm sure, I'm not sure how many presenters use memes in NotSec, but my talk is going to be nauseatingly full of memes. And I have one live demo as well. So I'm going to have to pray to the demo god. I'm going to have to ask all of you to pray to the demo gods on my behalf so that everything works because live demos as they go, a lot of things can go really, really wrong. So I'm hoping that everything goes pretty well. I did test it before I just started, but you know, things can go wrong every single time. So you never, never really know. Anyway, today's agenda is going to be pretty quickly. What are webhooks? How do they work? We're going to get into this pretty quickly. I have 30 minutes. So I'm not going to take too much time in the explanatory segments. We're going to be looking at some common webhook attack patterns. And this attack is not in the common webhook attack pattern. So the attack that I'm going to be referring to in this specific presentation is not a very commonly seen attack, but I think it has the potential to really explore on the scene. We've seen it explored on the scene, at least with the kind of bug bounty targets that we've been working with. It's really performed really well against a lot of these targets. We're going to do a quick introduction to SSRF because that's the underpinning of this. And SSRF, I'm sure a lot of you have heard of SSRF, probably do a lot of SSRF as part of your offensive work or even do defend against SSRF in which case I pity you. But nevertheless, SSRF is something that we're going to be focusing on. This talk is a lot about SSRF. We're going to look at SSRF. We're going to look at a new class of flaws that we dub webhook boomerang flaws. Now these flaws have been there. They've been in and around, but they've not been as well exploited, I think, as they could have been. So this is where we're going to be talking about. This is the meat of the presentation. And then I'll talk about some sub variants and so on. So basically, when you are building an API, the first thought you have is interaction. You want to respond to events. And one of the big things when you're building an API is to respond to events. So when a new user is created or when there's a new customer that signs up or a new payment that comes in or you know, blah, blah, blah, whatever it is, you are typically dealing with events, right? APIs, especially with APIs, you have a lot of events that you want to deal with. And a lot of times, when you are dealing with events, you need webhooks because you need to post that event somewhere. So let us say I'm a new e-commerce merchant and I have set up a new site on whatever and I'm using Stripe or Shopify or what have you. Now, in that case, the first thing that I'd like to know is, hey, when am I getting paid? Have I gotten this customer? When have I gotten this customer? So webhooks makes it really, really powerful way for your API to start reacting to these events. So when a new customer is created, you can send a webhook to Slack and say that, hey, a new customer signed up or you receive payment of this. Webhooks make all of that possible, right? So I'm sure a lot of you have heard of webhooks. Most of you are probably dealing with webhooks on a day-to-day basis. Most of you are probably writing a lot of webhooks as well. Now, so webhooks basically are user-generated callbacks. So I set up a webhook, so I'm a user and I set up something called a consumer. So I say that, you know what, whenever a new user is created on my e-commerce store, I want you to make a webhook request to my consumer app. So I've written this app somewhere here and I want you to make a request to that with a particular JSON that says that this user with this email, with this whatever other information is signed up. So in a webhook transaction, there are typically two entities that are involved. One is the provider, which would be your Shopify or Stripe or what have you. Any of these providers that are reacting to the events and then there would be a consumer that would actually be receiving that event and saying that and they're using it for processing whatever they need to process, right? So this is a very common setup that you have. So these are essentially user-generated callbacks, right? So that's how webhooks work. So in this case, we have a webhook setup and whenever a new user is created, there is this event that is triggered that hits the consumer. The consumer app essentially uses that to maybe store it or send them an email or process it forward or whatever they need to do, right? So these are basically how webhooks work. Now, webhooks are literally everywhere, right? Now you'll see webhooks in not only apps like Stripe and Zapier and Shopify and all of that jazz, you see this all over the place, you see this with Kubernetes, you see that with CICD systems, you see it with CICD systems, I would say use them extensively, right? Any kind of build systems or CICD tool use webhooks all over the place. Even basic, even apps that are related to marketing automation and so on, use webhooks extensively. Anything that needs to integrate with a whole bunch of other applications, especially through event-driven workflows, webhooks are the number one way to do that. And it's very popular, very commonly seen pattern that you do. In fact, companies like Zapier and Shari have realized also they run their entire business on top of running webhooks or triggering webhooks as consumers and providers depending on who they are in the piece of the transaction. Now common webhooks are trades, at least they're event-driven. They're typically post requests, which is basically post JSON. Of course, you still have the odd XML one, but most of them today are post JSON, which means they post JSON to the consumer. They also are sometimes protected by HMAC or API keys with an HTTP header. So just to ensure that the webhook consumer is somebody that is actually legitimate, you also need to authenticate the transaction through an HMAC or an API key depending on the kind of webhook. Sometimes they're there and sometimes they're not. It really depends on the provider and the kind of configuration the provider is using. And of course, in some cases, remember, especially in the case of build tools or CICD tools, you will see that producer systems allow you to add your own custom header. So let's say you need to add an API key for your system or your application, which is your consumer application, you can set up a webhook with additional headers as well. It's not only that you just give it a URL, you might also be able to add your own custom headers so that you can actually get your event and that would be identified with this HTTP header with a bunch of custom headers. So that's basically what the common webhook trades are. Now, if you come to think of it, the natural type of assumptions or attack focus that one would have. So let's say you're a pen tester or somebody doing offensive security on a webhook, the natural assumptions would be that let's try and compromise the consumer. Most of the attacks are trying to compromise the consumer and to prevent other kind of replays and things like that. Again, most of it is trying to compromise the consumer. So can I compromise the consumer with the deserialization payload, which means that let's say I put in some kind of malicious JSON or YAML or whatever it is, can I run my code on the consumer? Can I compromise a whole bunch of consumers with an ecosystem style attack? Can I tamper with the payload? Can the consumer detect tampered payload? Can I replay the attacks? Can I attack from unknown sources? So it depends on whether the provider can make random requests to another consumer. So a lot of these are typical attack scenarios that you come up with. So these are typical attack scenarios that you see and a lot of them and a lot of you I'm sure do this as well. So when you're trying to, can you do a replay? Can you run the same transaction a whole bunch of times? Is there a way for the consumer to detect that it's a replay happening? So all of these things are natural attacks. But in our case, what we're going to do is we're kind of flipping the script in this case. What we're trying to do is can I compromise the provider? Our focus today and our focus in this entire class of attacks is can I compromise that provider? This provider that is sending me the event payload, can I compromise that particular provider through some type of attack? So basically what I'm trying to do in this case is that I have a provider which is my, Stripe, Shopify, what have you. I mean, I'm just using them as examples. I'm not saying that they're vulnerable. Please keep that in mind. So whatever, I have this provider. Now this provider makes request, obviously sends this HTTP request to the consumer. Now the consumer in turn, instead of processing this legitimately like a normal or good consumer would, the consumer makes a request or somehow finds a way of attacking and compromising the provider and the internal applications of that provider. It could be a database. It could be an internal application. It could be metadata. It could be whatever. Now, as soon as you think of a scenario like this, I'm sure the first thing that comes to mind is naturally SSRF. Now SSRF is one of those attacks that orient itself towards doing something like this. Because SSRF allows you to redirect or facilitates a redirect based on a user controlled URL. So in this case, it has all the natural makings of a classic SSRF attack. The idea here is that when I, the user can use a particular URL of my choosing. So it could be a URL that I can select or I can enter or I can use as part of the attributes. And I can get this application, this target or victim application to make an internal request to one of their internal URLs or to a metadata URL. So let's say I want to compromise AWS credentials. I make a call to a metadata URL and that metadata URL gets, I get access to the credentials. And from there, I get those credentials and then start exfiltrating information from AWS. I can escalate privileges on top of your AWS account. So this is SSRF. This is classically SSRF, right? Most of us are aware of this. Most of us have seen this. SSRF is super common. You see this in nearly every third bug bounty report at least is, I mean, obviously don't have the exact stats, but SSRF is super popular. It's very, very commonly seen. Now you see, of course, SSRF, you have a lot of attacks, the capital one breach where the attacker was able to find SSRF on the mod security, mod security deployed by capital one, they were able to use that, get the make a call to the internal 169.254.169.254 AWS metadata, they were able to get the credentials. Those credentials had a large scope access to the Amazon S3 infrastructure that was run by capital one, and they were able to gain access to all those S3 buckets and exfiltrate a whole bunch of user information from there. Then of course, you've constantly seen this SSRF is super common. You've seen this as part of the proxy logon exploit chain. It is one of those extremely common and extremely popular attacks that is also hard to prevent by the way. It's not very easy to prevent SSRF. There are a lot of different nuances to prevent SSRF. Anyway, so the idea here is that with SSRF, you can do all of this stuff. One of the reasons why SSRF has become so popular is because SSRF is used extensively to compromise cloud environment. So if you're running Kubernetes or AWS or Azure or GCP, the idea here is that you have some metadata URL or metadata file path, and you can access that particular file path, gain access to sensitive credentials that are mounted on that particular file path or URL, and then use that to escalate privileges into the account. SSRF can also be, you're reading a remote file and executing that through a remote port execution. It can be denial of service. It can be information disclosure. So through SSRF, I'm able to access internal hosts, and those internal hosts have sensitive information that could be information disclosure. So you have a wide array of possibilities with what you can do with SSRF. So I'm sure a lot of you are aware of this. Now here, what we want to do is leverage SSRF against our provider. So we want to use this attack essentially as a boomerang. So what we want to do is to say that, look, hey, the provider is going to make a request to me, the consumer. I am an evil consumer, and I get a request through the provider. Now, I want to be able to use that same request chain against the provider. So I want to redirect a request back to the provider where I do something, where I'm able to compromise internal assets. So the idea here is that my provider makes a webhook request, whereas my consumer redirects that particular request back to an internal URL or metadata or what have you, and I'm able to compromise that as part of this. Now remember, a lot of times the reason that this also works is because a lot of these providers store previous results of webhook execution. So for instance, let's say I'm Stripe and every single time I receive a payment, I trigger a webhook or I've, the user has configured a webhook where I can trigger a webhook on this particular event. Now I keep a log of those requests and responses. So Stripe maintains a log of that. And a lot of times you as the user are able to see that log. So the idea here is that the consumer, who's also a legitimate user of the application, wants to leverage this attack and then use the webhook that they have set up to compromise that provider application or that victim application. That's basically what they want to do. But there is also a problem. Now, if you are familiar with SSRF, you'll know that SSRF works extensively with get requests. SSRF works when there is a get request involved. So when you take a get request and you are able to use another get request, you can do SSRF. You can manage SSRF. However, most of the webhooks, I'm not saying all, but probably close to all, most webhooks make post request or put request because they are sending data. They're not only making a get request. There's no point of a get request. Most of the time they are making post request because they're posting JSON to that particular consumer. So the idea here is that you are getting a post request, but how do you use that to make a, to redirect that into a get request? So it's hard to weaponize especially with pure SSRF. So this is where a very interesting header comes in. Now, there's a header which is a very popular, I wouldn't say very popular, but at least a well-known redirect header which is called the HTTP 303 redirect. Now, the way this works is that if you send it a post or a put, this redirects as a get. So you can set up a redirect on your consumer application that says, hey, whenever you get a post request to this particular path, let's say whatever, slash webhook, whatever, you need to 303 redirect. Now, a 303 redirect automatically converts that post request to a get request on the redirect and it requests a particular new location. And that is where this really works. This attack really works because 303 converts that redirect into a get request and sends it back to wherever you've redirected it. Now, obviously people creating the standard didn't imagine that this would be the case or this would be how it would use, but 303 allows that to happen unlike most other redirects that you have. So the idea here is that in a response, whenever you have HTTP 303, which is C other, it essentially says, hey, this way please and makes it a get request. So basically redirects it as a get request and your clients essentially just follow along. They just said, oh, okay, I need to redirect to this location. So I'm going to make a get request this particular location and that request turned now into a get request now, which means that you can leverage that SSRF in full scope. So this is something that 303 allows you to do. So what we want is essentially we want to provide our application, we want this webhook request to come in. And once this webhook post request hits my consumer, the consumer, remember, the consumer is another web application that's just listening for these requests. So as soon as this post request comes in, this sends back an HTTP 303 that redirects to an internal metadata service like 169.254 or some local host service or some internal IP address or whatever it is that it's able to access that URL and get whatever information, whatever response that comes from that particular URL get request is stored as part of the webhook logs. So that's the idea. So the attacker wants to be able to do this because the attacker wants to be able to store redirect, boomerang this attack back to the provider, get that provider to talk to an internal metadata service, steal credentials and then use that in the webhook logs, use those stolen credentials and escalate privileges into their AWS account or use that to access an internal resource or what have you, right? There are a whole bunch of things that you can potentially do with this. Now, this is something that we have extensively used against a whole bunch of public bug bounty targets wherever we've seen bug bounty system, wherever we've had bounty programs, we have tried this out and it has had amazing results. The kind of results that we've seen with the webhook system that they had have been quite amazing. And we've used this on Docker and we were able to use this successfully on Docker, right? Now, Docker obviously as all of you, I'm sure a lot of you use Docker every day, Docker is a massive service that allows you to host your container images in their container registry. Docker was the original registry. Of course, now you have a whole bunch of other registries, but we were able to do this on Docker. We were able to compromise and gain complete access to their whatever their AWS credential provided, which was quite highly scoped at this particular point in time. So I'm going to explain how we did this. Now, in Docker Hub, there is a functionality where you can create your own webhook, right? So we created a webhook in this case and we set up the webhook to a Ngrok URL, to a URL that we controlled, of course. And behind the scenes, we essentially wrote a small piece of code with that said, you know what? Whenever you get a request, a post request here, what we want you to do is run a 303 redirect back to, you know, 169.254.169.254 slash latest slash metadata, which is a very common path for the metadata in an AWS EC2 instance, not only EC2, but any many other compute instances of AWS. We're talking about Elastic Beanstalk, we're talking about Fargate. I think Fargate has a slightly different IP address, but it's pretty much the same idea, regardless, right? So, but it is essentially gets you access to the instance metadata on that particular node. And through that, we were able to identify this. So once we did this, we pushed. So this webhook would be triggered every time we pushed a new container image to this particular, to this particular Docker registry, Docker repository, right? So we pushed this container image, which was generally a random container image. And once we did that, we saw that the Docker Hub, the webhook trigger, the Docker Hub essentially was triggering the webhook, of course, but we were not able to get the webhook logs. We were not able to get the log of the webhook. And when we actually saw the request and responses, we found the webhook log in the request. It was not being shown up on screen, but we found the webhook log. And in the webhook log, we found that they use the metadata credential called the AWS roll name called this, right? So prod, whatever, whatever, something they use some AWS roll name here. And once we were, once we use that particular URL in a subsequent request, we were able to dump the AWS credentials. And as it so happens, those AWS credentials were related to their EKS infrastructure that they were used, they used to run Docker Hub with. So the EKS infrastructure and probably a lot more, we didn't go beyond this because obviously we wanted to report it as soon as possible in the interest of vulnerability disclosure. So as soon as we found this, and we, and of course they mentioned to us that this was coped to the EKS cluster on Amazon. And obviously this means that this could have been a much larger attack because the Docker Hub, which is, which contains a lot of the base images and the trusted images potentially, I don't know, maybe they could have been compromised as well. So the idea here was those AWS credentials, we were able to identify and using those AWS credentials, we were able to access their AWS account. We were able to potentially compromise their AWS account and gain access to whatever scope that particular roll and that particular privileges granted to that roll could provide us. So this was a huge attack simply because Docker is, as you all know, a extremely public service that all of us use. And this could have had a pretty huge impact. So we immediately reported it to them. In fact, we just did a get color identity and it was giving us a legitimate roll on their AWS account. So we were immediately reported it to them. And they, of course, they fixed it in a matter of about two hours. As soon as we reported it a couple of hours later, they said that we fixed it, we re ran the checks and we found that, that actually indeed fixed it. So that is one of the things we found and thankfully it was fixed quite a while back. So it's quite, this is not something that we need to worry about anymore. So they were able to address it almost immediately. So we have, I have a quick demo that I have set up here as well. I'm going to quickly run this. Now, if you see here, we have our application. So I'm just going to quickly walk you through how the demo works or the demo app. So I have a provider application, as you can see here. And I have my evil webhook consumer that I've set up. Now, every time I make a webhook post request payload, when I trigger a post request payload here, my evil consumer with HTTP 303 with a redirect to a CouchDB database. So CouchDB is a very popular no SQL database that you can directly access through HTTP like a REST API. So the idea here is that I'm going to use this to access my data internally on the CouchDB. So CouchDB should not be exposed, but because I'm redirecting it and because this application has access to CouchDB, the idea here is to get this application to connect to CouchDB and store those results as part of the webhook request and response framework. So now let's quickly go into this. I've already set up the demo. I am running my application stack here. So this I'm running on our training platform on AppSecEngineer. In fact, so now let's, I think I've already set up everything I need to. I'm just going to log in and explain how the attack. So I have, I'm going to log in as the user. So this is me logging in as the user and I get a JWT or an authentication token once I've logged in as this particular user. So if I do an echo dollar token, I get this. So if I just paste it here, it's a genuine jot that I can use on my application. So in this application, so you'll see that it's a valid JWT, blah, blah, blah, it's logged in. Now I am going to list the user's data. So I'm going to list the user's information as part of my application. So the user in this case has set up a webhook and this webhook makes a post request whenever a new build happens. So imagine that this is a CI CD system and a new build happens. This triggers a request to my, to this particular URL, right? So that's basically what happens. Now I'm going to quickly simulate this. I'm going to simulate a request. In this case, what I'm going to do is here I have set up my webhook. Now in my webhook, what I have done is I have set up my redirect URA to the CouchDB service. So I've said that whenever you're trying to do this, redirect it internally to a CouchDB instance. So make a 303 request back to the CouchDB instance, right? I've just set it up as a redirect. So if you see my code, you'll see what I'm talking about. You'll see that every single time a request comes in, make a 303 redirect to this URL, this hook URL that I've set up, right? So that's basically what's happened. So I'm going to run my hook. This is me running my webhook, my malicious webhook. And once I make a request, which is basically like, I'm going to trigger an event. You'll see once I trigger an event, this comes back with the CouchDB all databases information. So if I see what is actually returned by this, you'll see that the CouchDB database has a database called users. Now if I change this request, your hook URI, and instead of all databases, I make this users. Now this is me escalating privileges, 984 slash user slash all docs. Sorry about that. So every time my hook is reconfigured to redirect to another location. So in this case, now if I trigger another webhook event, you will see that it fetches. I'm not sure what happened. I think there's something wrong with the URL that I was trying to access. Oh, that was the issue. Sorry. Looks like I've added some unnecessary characters here. So let me just rerun the webhook simulation. So you'll see that it returned with a whole bunch of characters. So if I do echo, like we should have just done a jq here. So you'll see that it returned all these user values, right? So these are just user IDs, but of course I can fetch pretty much anything at this particular point in time. Now, so this was the demo of how this would work, right? Now, of course, one of the common things is, okay, this is a get request. What's so great? Yeah, in many cases, if you have metadata, a lot of times those metadata to access cloud metadata, if it's not AWS, in many cases, you have to add additional headers. But a lot of times you will see that a lot of services allow you to add custom headers, right, to the webhook. So in that case, one of the things you can do is add custom headers. So in this case, we were able to compromise a CI system where they were using GCP. And because we were able to add custom headers, we were able to simulate a system where we were able to add the metadata flavor header, which means that this would make a get request to the GCP metadata with that mandatory HTTP header, which is metadata flavor Google. And through that, we were able to still perform this attack. So if you have a webhook system that allows for custom headers, you have that possibility as well. I don't think I think I've run out of time. Now, I'm just going to quickly run through the defense. The first thing you have to do is one of the simplest things that you should do for this is to not follow redirect. So when you have a webhook setup, when you're using APIs and you're using webhook functionality, the first thing you have to do is turn off follow redirect on our HTTP client. That would essentially bring down the possibility of this attack by quite a bit. You have to figure out what type of HTTP client you're using and not follow redirect in that particular HTTP client. That's something that you have to do. Of course, you can add additional security elements like network security policy. You can add DNS so that it does not resolve to a local internal IP address. You can add a validating validation for the webhook URL, although that's a little difficult to do if you have a publicly public web users are providing their own URL. So that may be difficult to do. Or you might have an IP denialist to say that anything that tries to talk to 169.254 or localhost or 10 series or whatever it is, you do not allow that in your webhook URL. So these are some of the areas of defense that you can use. That's really it. Conclusions again, webhooks are a very powerful way to integrate. But again, with these kind of boomerang attacks, you can have a very serious set of issues with webhooks. So definitely look at those possibilities and consider this as part of your threat model. I think the first thing you should do is consider this as part of your threat model that really makes it a lot more powerful for you to think about these type of attacks and how you can potentially get hosed by these kind of attacks. So that's something you'll see SSRF, of course, is the attackers very good friend in this particular case. With that, I come to the end of my talk. I realized that it was probably a little rushed, but I hope I was able to convey everything I wanted to convey. And I hope it was useful. Thank you so much for having me. And that's really it.