 Okay, great. Hola, everybody. Thank you for sticking around. I know it's been a long two days. We're at the end, and I really appreciate you guys sticking with me right to the very end. Yeah, thank you for having me. Thank you for CodeGram for putting all this together. It's as well, it's awesome. My name is Ritz Jones. I'm the founder of Gundot.io, as our fantastic MC already mentioned. Awesome freelance gigs for free and open source hackers. I'm also the author of Zappa, the best damn serverless framework in the world. It allows you to run any Python web application on AWS Lambda. You can build event-driven applications. You can scale to 500,000 connections per second globally with no ops right out of the box. It's used by all sorts of big famous tech companies. It's awesome. You should try it out. Welcome to my talk today. It's called Gone In 60 Milliseconds, aka Having a Sexy Title Gets Your Talk, Accepted at Conferences, aka Intrusion and Exfiltration in Serverless Architectures. Ooh. What does that actually mean? That's a lot of buzzwords in there. So, quick poll, who here is familiar with AWS Lambda? And it should be everybody now because of the awesome talks that we've seen over the past few days. If you're not here... Oh, so yeah, we're mostly going to talk about Lambda in this one, just as the example. But if you work with other stacks, if you work with Azure or even Kubernetes and stuff like that, you should be able to apply some of the same principles in this even though the specific examples we give won't match one to one. For the unfamiliar, unfamiliar, quick Lambda primer, in the good old days of monolithic web servers, the LAMP stack, you pretty much just had... You had the web server, you had the database, and you had your CDN serving the static files, and it was all pretty simple. One server, lots of services, which was great if you were trying to compromise a server because as soon as you found one SQL injection, you could use your favorite SQL injection tool like SQLMap and totally take over the whole system. Hooray. In the serverless world, the architecture might look a bit more like this. So we have an ephemeral API gateway, and we have this elastic network of isolated Lambda functions which only exist for a short period of time, and the database is protected... protected by this thing called the virtual private cloud. There's no permanent infrastructure besides the DB in the whole system. Every single function, every single page load, every single thing that happens is cached by Amazon. It's spawned and destroyed on a per-request basis, so for every single page load, there's an entire lifecycle for the underlying web server. The code execution can also be triggered by cloud event sources, so when somebody uploads data or if something in your big data processing pipeline is done, that will then cause the execution of the next function in the sequence, stuff like that. Every request is in an isolated container as well, but as we'll see in a bit, isolated is maybe a marketing claim that doesn't hold too much water. The flip side is that it's super duper scalable, so because one request corresponds to one server, 10 requests correspond to 10 servers, and you can just keep adding zeros, and it just works, which is sweet. You can literally scale this to trillions of events a year with no ops, no infrastructure. It just works. It's pretty cool. It's also, if you do it right, orders of magnitude less expensive because you pay by the millisecond, so it's a lot of zeros, $2 or euros as well, because the exchange rate right now, milliseconds. Security patches happen automatically. You just don't have to worry about that in the same way that you used to because Amazon takes care of it. It saves you a lot of time. It saves you a lot of money. It lets you fire all your ops team. Sorry. Some common patterns you might see. We'll blast through this. Yeah, the web server, so this is pretty much what a lot of standard Zappa deployments look like if they're running Flask or Django on Lambda. It's the one we saw earlier. You might also see some for asynchronous data processing, which have, you know, so somebody uploads something to S3, triggers the Lambda. That stores a result in a database and spits out something to, you know, an output bucket on S3 as well, so you can see there's sort of like a circuit that happens with little Lambda functions along the way, making everything keep spinning. Chatbots are another common one, so if you're talking to what seems like a robot over email or Instant Messenger or through one of those little JavaScript widgets, it's quite possible that you're talking to a Lambda on the back end. And if you work in FinTech or, you know, medical technology or scientific technology, it's quite possible to use Lambda as part of your big data processing pipeline. So you can use Lambda to build and manage queues, which are then picked up by your GPU clusters that are EC2 instances that would be elastic as well. So this is another pattern that is fairly common that uses Lambda. There's loads more. They're super flexible, so it's really easy to fit them to whatever specific architecture you're trying to build at your company or for a hackerspace product. But what happens if we find a SQL injection in one of these Lambda-based APIs? So we'll call our favorite SQL injection tool, we'll call SQL Map, and then we'll wait and it won't work, which is bummer. So because the container dies after every function, even if we get code execution on the Lambda, it doesn't matter because the Lambda only has a couple of milliseconds lifecycle. So, you know, we have to update the way that we do our offensive security or fun hacking on these, you know, little Lambda widgets. Yeah, what does that mean for us as the red team, as intruders into these systems? Yeah, it's harder to infiltrate. There's less common code. You can't use, you know, battle-tested exploits and frameworks as much. The services are isolated. The functions are isolated. There's no user system that we're used to on a normal big old Unix box. There's no sys admins to trick into running our fun little executables because you already fired them all as part of your DevOps switch. It's also a lot harder to persist, like how can we implant malware into a system that only lives for 40 milliseconds? It's also a lockdown read-only file system, so we can't use external tools in our new environment, and it has a sub-second lifecycle. There's no initialization system to infect. Amazon takes care of all of that. It's tough out there. It's looking pretty bleak. Yeah, it's harder to get our data back out because the Lambda probably doesn't have direct internet network connections, right? And there's also this virtual private cloud thing that sounds like a big headache. Each function will also have specifically defined roles about what that function is allowed to access. Super strict. We can't get a reverse shell because we're not allowed to contact remote servers. We're not allowed to persist anything. So it sucks. It sucks. We're totally boned, right? Oh, hell, no, we're not boned because when God closes the door, when Jeff Bezos closes the door, he opens a window for us. Jeff Bezos. Photoshop skills. So today, we are going to be learning recon. We're going to be learning infiltration. We're going to be learning exploitation. We're going to be learning exfiltration. It's my favorite one. And cleanup, which is my least favorite one. Part zero, recon, aka, how the hell do we know what we're attacking? There's two attack surfaces that we have to worry about. There's the outer attack surface, which is kind of like the public-facing one. And there's the inner attack surface, which is stuff that happens inside of their AWS architecture that can affect the Lambda function. So the outer attack surface, typically this will mean the web requests that are coming in through the API gateway. If you look in the headers and see the dynamic content is being served through CloudFront, that could be an indication that it's an API gateway server, even better. Most people don't know this. If you run API gateway, these two endpoints get hijacked to return just like a healthy function. So if you see that, that probably means it's running a Lambda. Or even better, if Amazon announces who all their Lambda users are during the keynote. So if anybody wants to try any of this stuff, I didn't tell you, but maybe those would be good targets to try out. You can also use file uploads, of course. Those are public-facing emails. If they're using SES, you can check that in the email headers. The inner attack surface is pretty much every other AWS service that they might use as part of their architecture. So queues, they might be using if they're doing some long-running data stuff, database events, streams of information if they're a big data company, their user system, all this stuff. So now we've mapped, we have a good understanding of what our target is. They use some of these AWS services. We want to infiltrate. How are we going to weaponize all that? So a lot of these new microservice-based systems are kind of like this. You know, it's a big mechanical machine with lots of moving parts and stuff being passed around. So we're going to borrow a term from the auto industry and we're going to use something called destructive mechanics, which is basically the process of dropping a bolt into the engine and seeing what happens when it breaks. You know, listening to the sound of the break and then trying to infer the parts that went wrong. The TLDR of this whole process that we're going to be demonstrating here is we attack the points of exchange in between the services. We fire off every cloud and system call that we can and we just see what happens and then build an attack based on the services that are available in that architecture. So here the usual suspects still apply, unsanitized user input, bugs in your deserializer, server-side script injection, malicious binary files, and all your favorite web app exploit patterns still apply. It's just the next step that we're going to change. So a quick example. Let's see. So here's some vulnerable code. You know, this is pretty typical code. Some of you may see, like, where the problem is in this. Because the problem is that Amazon allows semicolon curl-exploit-server-pipe-bash as a valid filename for anything that is uploaded to S3. So if we can craft something, craft a filename that then gets expanded into something that's called as a system call, you know, it'll call the magic program, kill that, and then call our infection handler and load all the tools that we want. So yeah, so this works, and then we get a call back to our infection server just by creating an S3 object in the bucket that has a call to our exploit server. Hooray, we did it. Part two, exploitation. AKA, how can we escalate our infection? AKA, what the hell is a lambda anyway? AKA, what is worth stealing from a lambda? So we can't run Find, unfortunately. You have to, you know, do this process through a lambda, but I've already mapped it out for you. You can look at the whole file system of a lambda. The whole TLDR of that is it's pretty much just Red Hat 6 running off of an AMI. It has a whole bunch of goodies that we might want to use already installed, which is super nice. Yeah, it's Amazon Linux AMI, so it looks like it's an EC2 server. Some of you who are familiar with EC2 may already know about the MetaInfo server. Some of you who use EC2 are probably not familiar with the MetaInfo server because it's pretty deeply buried inside of the AWS documentation. So let's check that out. Instance metadata, data about your instance that you can use to configure or manage a running instance. Anyone who can access the instance can view its metadata. Therefore, you should take suitable precaution to protect sensitive data. This is buried in, like, page 1,000 of the AWS docs. Nobody does this. It's great. So we access the metadata server, and we get, you know, all of the keys, all of the users of the system, a map of the network, a map of all the policies that are available there, all of the block storage, a bunch of fun stuff. Nice idea. This doesn't work from lambda. EC2, so it's a good trick to have in your toolbox, but it's not going to work for our purposes right now. Let's check the environment variables. You know, print out the environments. Bingo. So you'll see that there's these things called security tokens that look interesting. What are those? Yeah, what's a security token in this context? So now we have to learn about this thing called AWS identity and access management. Uh-oh. Per resource authentication and authorization definition, one task, one lambda function, corresponds to a single authorization. It sounds bad. It is. The good news is that it's super easy to fuck up. So almost everything that we're going to show from here depends on slightly wonky IAM. That's okay, because that's pretty common. In fact, every single serverless framework out there will set you up with insecure defaults, including my own. I swear to God, I'm going to fix that at some point. And most people who are running these things probably don't check that. So there's something called the Lambda Execution Policy, which is the specific IAM role and policy for any given Lambda function. It uses this thing called Pass Roll to take a predefined policy, create a temporary user with those permissions. That's the user that runs inside of the Lambda every time, gives those credentials to the Lambda user space so it's able to make the API calls to the other AWS network services that are available there. It'll generally look something like this, JSON document with a bunch of stuff it can do in it. This one, we're seeing that we're able to create log groups and log streams, we can create a network interface so we can connect to a network and we can describe a network interface. And that'll come in handy for us. So we can describe the network it looks like. What about infecting the application source code? Where does the code live? If you check the environmental variables again, you'll see there's this thing called VAR Task. If we try to put our backdoor into the VAR Task application, it won't work because it's a read-only file system. And even if you could write to it, it would only live for 60 milliseconds. But where can I put my beloved tools? So here's what Amazon calls ephemeral disk capacity, which is a fancy way of saying temp. So we can just use temp. It turns out temp works. It's actually not a read-only file system. There's also a temp directory. And what's interesting here is the lambda executions aren't completely isolated from any other invocation. Because they're cached in memory across executions, that's how they're so fast. Because temp is part of this RAM disk and RAM gets cached, temp gets cached across multiple executions as well. So we can load up all of our favorite tools into the temp directory and they'll persist across other invocations, which is pretty cool. As long as that function is kept in warm in the memory, which is about four minutes and 30 seconds, it turns out. Somebody violated their NDA with Amazon to tell you that. And that also applies because processes are in memory as well. If you're running an N-map of the network that's available to that lambda, it'll actually persist across other executions as well because memory is cached. So, yeah, we can load up all of the favorite, like, hacking tools, N-map, SQL map, Metasploit, we can run that all. So now we have the keys for execution and we have all of our favorite hacking tools. What now? First, see what we're allowed to do. Use AWS provides fancy tools for us to be able to list everything that we can do. If you're lucky and the DevOps team just wanted it to work and didn't really think about security very much, you're going to see a lot of these stars. So, star, star, star, star, star, star. That means jackpot. So, and then once you have those, you're also able to create a new admin user that that company is now out of business. You have completely dominated everything that they could ever possibly do. Yeah, right. So, we're going to find, you know, something a bit more locked down than that. That's a brief interlude. If you do hit the jackpot and get, you know, eBay's star, star, star, star, star, star, star, star, don't sell the user info to spammers. Don't mine Bitcoin on all their GPU instances. I know that's really popular these days. Don't claim a bug bounty. Bug bounties are boring. Use your skills for something awesome. Maybe put up your badass hacking crew name, you know? Like that one. That looks pretty cool. Put spooky skull gifts on their website. Give a shout out to your IRC homies. Shout out to all my IRC homies. Basically, I want the 90s to come back and hacking to be fun again. I'm tired of all this corporate bullshit. So, far more likely than hitting a star.star, you're going to get some semi-strict permissioning system like this one, you know, for some Lambda that's interacting with a bunch of S3 buckets, or this one that's interacting with a database, or this one that's doing all sorts of crazy stuff for some combination of all that stuff, depending on what, you know, the function that you're attacking does. How you doing? So, how can we abuse that information for nefarious purposes? Part three, exfiltration, a.k.a. the fun part. This is what, you know, this is the good bit. So, TLDR, when you don't have a direct network connection to the outside, so we can't get a reverse shell, we can't call our exploit server, we can use tags, meta information, and the cloud services themselves to shuttle information out of the cloud. You know, the cloud becomes the vector from which we extract data from the cloud. Here's a real-world example. I hope I don't get in trouble for this one. So, in America, we have an organization called the Federal Communications Commission. They're in charge of running the Internet, censoring the Internet. They recently got a new chairman, a guy named Ajit Pai. He's not a popular guy. He's Trump's pick. He's super pro-business. He's not a super popular guy in the Internet freedom or hacking community. So, somebody figured out that if you make a certain call to the FCC.gov's API, that will then generate a PDF that will be uploaded into the FCC's, you know, government S3 bucket. And a hacker figured out that he can get control of this black box and was able to control the output of the FCC's official document generator. So, this document started... Just so the transcriber who's been doing a really good job has to write this out. It says, Dear American citizenry, we're sorry that Ajit Pai is such a filthy, spineless cock sincerely, the FCC. So, this immediately went viral. It's like red alarm at the FCC. They had to issue a statement. It was totally covered by the tech media. So, it shows that this kind of attack is very real like today. If he wanted to escalate that, he probably could have done much worse damage than just creating a little protest PDF. But, you know, this stuff is actually real. But back to our proof of concept. So, the easiest thing, if we see that we have permission to do SES, then we'll send out an email. So, we'll execute our code and then have the results sent back to our email address. Or, if it's, you know, a chatbot, we can have it sent to our phone slightly harder. If it's just access to the S3 buckets, we could gzip and encrypt all of the application source code and then put that onto S3 with default expiry and then get that tar ball of their source code from their own S3 bucket, which will then self-remove. The fun mode is VPC exfiltration. So, VPC, as we saw in this slide before, oh, yeah, what is a VPC? Really good question. It's software-defined networking, basically. It's an advanced security feature that enables access control lists to enable inbound and outbound filtering at the instance level and subnet level. Sounds bad. It is. The good news, it's really easy to fuck up, especially if you follow Amazon's, the documentation on Amazon's forum, that'll lead you straight to hell, which is great, so keep reading that. Because Lambda has access to VPC resources, it essentially serves as a hole puncher for us to be able to access all of these internal VPC resources once we have control of a Lambda function. In addition, you can connect securely to your corporate data center. It uses the industry standard encrypted IPsec, IPsec hardware VPN connection, which is really nice of them to give us this encrypted channel directly to your internal corporate network, so thank you for that. But we don't even have to actually use that network. We can use the metaproperties of the VPC as a way of extracting information from within the VPC. So, step zero, upload your malicious file to get code execution in Lambda, like we showed before. So it goes from S3 to the Lambda. I'm using cool spooky skull gifts, you'll notice. Then we fire off all of our services to see which Amazon services we can use to get information back out of the cloud. We'll see that there's, you know, in this case, one of our S3 buckets is writable, so we can write all of our information to a particular S3 bucket. In the policy that we get back in that bucket, we'll see that it has this pattern, which allows us to access VPC resources. You know, this might be a data elastic compute cluster with some cool custom code that we want to steal or probably a database in this situation with sexy financial data that we can use for nefarious purposes. So we'll then, we see that, so you can use any kind of exploit, you know, that you like in this case. I like malicious pickling through salary. You can poison their package server. You can try to hijack the DNS traffic. You can poison the AMI image or use a stolen S8 key pair, or basically whatever trick that you prefer when you're doing this right team stuff, we'll probably work at that point. Once you have code execute, so now we have code execution on the remote cluster inside of the VPC, but we can't get the information back out because of the VPC, so this is the tricky part. What do we do now? So what we can do is tag the VPC itself. So because Lambda's, it's part of the default Lambda execution role, have the ability to describe network interfaces, that means that we can list all of the tags that are available for the VPC that we have access to, like this. So in the tag set, we'll see that our canary fired by our exploratory malware has been fired, so we're able to carry information out of the VPC network, and then we can copy it back out to the outside world the same way we're using our S3 bucket. Boom, so we have total Pone. This works for application binaries and binary data as well. You can just base 64 any binary data that you want to get out of the secure network and just use a delimited tag system. It's great. Similarly, you don't have to use tags on the VPC. If you have an elastic cluster, there's probably some networking there, but it's probably not a good idea. So if you have access to these things, probably has access to Route 53 to do the DNS traffic, create a new DNS entry with the information that you want to steal. Create, if there's centralized logging and you're able to read writes or even read what log groups are available, you can try that as well. Be creative. Every cloud service is a threat to all of the other cloud services now because they have these metadata properties every day that you can try. Cool, but what if they fix the bug? Part four, persistence, and we're going to speed it up now. AKA, how can we permanently infect a system with no permanent infrastructure? So another neat Lambda feature is it has this thing called aliases, which is useful. It stores all the old functions by default. So every time you update, it keeps the old one around, and there's an emergency situation and you need to roll back. I'm just going to shamelessly plug some of the awesome features that Zappa has as well. You can just zap or roll back. And it's useful for separating your staging and your dev environments or whatever, and for auditing, checking your logs and stuff, but it's also useful for persisting malware. So we can get the current function, get the code, install our backdoor into the current function code, and then create an alias to an old one that nobody's really ever going to check the very first function of the code to see if that matches the previous signature that it did ever. So then we can have our malware just kind of live in there for whenever we need it. Another route that you can take is if they're using cloud formation templates to update their whole Lambda stack, that depends on the Lambda code being hosted on S3. So if we infect the S3 file where that stack is being launched from, then every time they update the production stack, it's going to call the infected package. We can even do with that technique, we can attack the other Lambda functions that they might be using, but we can go even further than that. We can build a serverless event-driven malware. So here's how to design a persistent event-driven serverless malware. Imagine that there's some Lambda function that is designed for checking the status of an SQS queue. Normally this is always going to be alias to the latest. Imagine we're able to get code execution on this Lambda. Then we create a backdoor version of that code, make that the latest Lambda, so every new user is going to be looking at the backdoor version. Then we create an infector in one of the aliases. So what happens and then we'll also set up an event trigger to call the infector every time there's an upload to the S3 bucket where new Lambda code goes that will then infect every single time that the code is updated. So the deployment system, the DevOps team they upload their new version. This fires off the function, the event handler in our infector. We get the latest source code, we reinfect it, and then we make that the head that everybody calls. So new code uploads are the thing which calls the trigger for reinfection of this malware. So part 5, cleaning up, we're going to blast through this because I'm running out of time. This is the boring part anyway. Full disclosure, I'm not very tidy. You should see my room. Be warned about this, like if you're doing some like super secret CIA stuff using this technique, which you probably aren't, I don't think anybody's ever going to use this for anything. Don't use this part, you should think this through because it's probably, you'll get found out. But we can attack it. It's a really cheap hack. But just say stop logging, see if that works. If we can maybe do better than that and try not to log anything to begin with. Because Lambda functions all have a pretty small size, or like 256 or 512 megabytes of memory. If we just consume all of that memory, then the logger stops working. So we just wrap everything that we do, if there's a fail in it, just allocate a shitload of memory, and then it doesn't cause an invocation error so the cloud watch alarm doesn't go off and the admin doesn't wake up. Great. One thing to watch out for is if they are just logging every event that comes in, that will probably reveal your attack. But that also means that they're logging everything that comes in, including all of their users names and passwords and stuff as well. So there's probably something that's worth stealing in there that you should take. It's great. So, you know, two sides of every coin. The final part, synthesis. Here's a present for you guys. I've packaged a lot of this stuff into an Adios Lambda Infection Toolkit that we can play around with called McKenzie. It has a lot of the features that I described in the demo. It has a lot of the features that you can test it on as well. It's on GitHub. I'll be releasing it in the course of the conference. In conclusion, as we move to a serverless distributed cloud hosted containerized event-driven world, the more malware will span platforms, services, and connections. Wow. So what will malware look like in the distant future? We're going to go back to our computerless malware. So, there's an incredible zine, underground zine, for people who write viruses called Valhalla. In issue number four, a guy named Second Part to Hell wrote a paper, brilliant paper, Infection of Biological DNA with Computer Code. It's a virus that makes the step from the digital to the biological world. It infects the machine that creates new life with the virus itself. When that virus is scanned, it exploits the XML reader in Craig Venter's DNA reader, which then infects that host system and every bioprinter that is connected to that network. It's a brilliant paper. It's genius, like straight out of Star Track. It's pretty much the distant future, except somebody's already done it. This happened last week. Here's the first synthesized DNA exploit, and here's a paper by the team who did it. They weren't even aware of the underground doing it first, but they did make the actual DNA that contains the exploit code. So, yeah, think about it. Do you need secure serverless apps? You should hire me. Come meet me afterwards. If you want to contribute to making open source serverless frameworks more secure or featureful or awesome, you should check out Zappa on GitHub. We also have a vibrant Slack channel with some of the smartest Amazon users and builders just anywhere that I've ever met. Great community. Come join us. Thank you.