 Let's get started. So today we're gonna be talking about an incident that happened to one of our customers about a year ago. And I thought it was important to share the details of the incident and hopefully share with the community and hopefully we can prevent those kind of incidents from occurring at our own companies. So just a quick introduction. Thank you for the intro, Lauren. But I currently work at the Cloud Health Technologies company that's focused on managing cost and governance across multiple cloud environments. And my main responsibility is mostly security and compliance now, although I did start it in technical operations, DevOps, et cetera. As Lauren mentioned, I also do a little bit of car racing on the side and I juggle and do some Accra stuff. It's part of our circus community here in Boston. So how many of you guys are familiar with cryptocurrency? Just if you don't mind raising your hands. Okay, all right, it's a pretty big crowd. And how many actually lost money recently on cryptocurrencies? All right, it's pretty embarrassing thing to admit, but thank you for sharing that. What I wanna point out is the Cryptojack in and out of itself is a way to leverage somebody else's compute resources to mine cryptocurrencies. And there's a variety of ways of doing it nowadays. You can do it through browsers and JavaScript. You can do it on your home routers or your personal laptops. But you can also leverage physical resources in data centers and also cloud environments. So it's pretty pervasive. Things like, for example, Pirate Bay recently had a JavaScript that was a bit mining monero. And it happens to real companies as well. Most recently in this year in particular, Tesla was in the news recently. They've had apparently an open Kubernetes cluster that somebody leveraged to deploy containers and run Bitcoin mining operations in their cloud environments. Same thing happened with insurance provided in UK, Aviva and Jamalto as well. And most of those are due to leaked AWS credentials. However, I wanna point out that many of those things go unreported, like it's a pretty embarrassing thing to admit that your account's been hijacked for crypto mining operations. And it's not like a data breach where you have to report it to authorities, et cetera. So a lot of those things, because they're embarrassing, they don't get reported out in public. The thing, for example, like about the AWS leaked credentials and account takeovers is a pretty serious thing where it can potentially destroy the company. In 2014, if any of you remember, there was a company called Codespaces where their root account, AWS account, got compromised and the person that took over that account demanded ransom that was, which was not paid. And essentially, because the company was completely based in the cloud, the hijacker destroyed the company, including the backups and everything else. So speaking of setting up property strategy, so the company had to close shop because they had no way to recover the data, their customer data, their own data. So it could be serious. So let's talk about the incident itself and the timeline. We, about a year ago, when this event happened, we got a support email to a support personnel, essentially stating that, hey, we discovered that we have about 200 instances running in our AWS account across multiple regions and they were generating about $12,000 a month in EC2 compute cost. The customer stated that they didn't have any cloud trail enabled and absolutely no audit logs of how it happened. But it did notice is that one of the accounts, I am a user that was typically given read only permissions, all of a sudden had admin rights. So they sent us this email hoping that we can help them investigate because we do monitor our customers' accounts and we do have some data on them. So as a part of the handling this incident, they obviously reached out to AWS support. AWS had them completely thoroughly clean out their AWS account that was affected by this. They just right away, a little bit too late now, enabled cloud trail in that account to capture the audit data. And then they worked with a customer on the refund. Again, the refund part is case dependent, kind of at the mercy of AWS on this. And what we did on our side, because our platform essentially captures the state of the accounts at like 15 minute intervals, we went through database backups at like certain point of time to reestablish the timeline of what exactly happened during the incident. And I'm gonna go into the details of that timeline a little bit on. But we also configured our cloud health platform to help customers to monitor for this kind of things, the security module on that platform and then some of the best practices in terms of cost reporting and cost spike reporting like this. So the timeline part, this is where it's interesting. So it happened around March and what we found out that there was a series of AMIs created throughout the world in multiple regions and they were publicly shared AMIs, windows based. And what we noticed that when we're doing forensics the first thing that happened during the incident is that one of the users in the environment had admin privileges, their access key, that Amazon has two slots right, AWS one key and two, the access key one got rotated. And right following that, our platform started detecting other changes like the user that was affected that was read only supposedly was giving admin rights and console access, et cetera. But then immediately following that, we started seeing multiple resources around the world in multiple regions, started getting created. And you could see that in terms of the span, all those activities happened in less than 10 minutes. Right, and then we started detecting that, hey, 200 or so windows, C4 actually, C4, 8x relage, windows machines, starting getting detected. We started getting collecting the performance metrics of those machines and we immediately started seeing 100% CPU utilization across the board and all of them. And then the day later, a customer kind of discovered the spike in cost in their AWS account. So a couple of things I want to point out on this timeline is that for us, the majority of the data about changes in IEM and permissions came from one particular report called Generated Credentials Report. In that report we were able to exactly establish the timeline of one of the user. Credential got changed, when the user permissions got changed, when things changed on that particular account. Another thing I want to point out is that, I don't know if you noticed that the first operations that we saw, right, was that access key one got rotated. And does anybody think like, why that's peculiar or a little bit strange? That why this is the first operations that an attacker would do? Yeah, exactly. Not just you, but if you imagine that, if you somehow share your credentials, right, those guys are not the only ones looking for them. So not only they want to lock you out, but they want to lock out anyone else that's scanning at the same intervals. So it's like the first one in, wants to lock everybody else out. So another thing I want to point out was highly animated, templated, right, they had the pre-baked AMIs already ready to go, all they need to do is to spin them up. All the configurations for AWS accounts, VPC, security groups, knackles, everything, right, they all got preset and spun up. It was limited to 200 instances, and this particular reason is because that's the default AWS account limit. So they probably could have spun up a lot more if they could. So that's kind of like the summary of the timeline. I kind of want to go into discussion about how we can actually prevent this kind of incident, right, and those instances are fairly easy to prevent. Couple of points I want to point out, like the root account is the most important account you have in AWS. You want to make sure that it's locked away in a safe, and the best way to do this is get a physical MFA setup for that root account, disable all in any API access to that account, and just really don't use it at all. That's just the most basic thing you can do. It's a God mode like account, so treat it with respect. The second thing, there's various types of users that you generally would use in AWS, and for typical users and operators that people that would log into your AWS accounts, there's a couple of things you can do. A, you can enforce the MFA on all those users and operators, and Amazon provides you this amazing IAM policy that you can overlay on any and all your existing IAM policies assigned to a user, and it will just enable MFA and will allow the user to manage their own self credentials without disrupting anything else you may have set up in your own policies. So it's a very clever policy that they come up with. By all means, do use it. Well, obviously the MFA works really well for AWS console access. It gives you a little feel to put in a one-time token. Sometimes it's a challenge to use it on the CLI. There's a tool that you can use for CLI AWS vault, quite a handy tool. Other solutions that you may be already using, leverage Staml or SSO, like Google, Octa, Ping Identity One login, all good providers for SSO login to your AWS account. So there's no need for actual user in AWS. And some of those do provide CLI-based tools, leveraging the MFA as well. So one last thing there, do not give out direct permissions to users. IAM permissions ask users to assume role that they actually need to use in their accounts. So on the application and application services accounts, those are very important. Obviously a lot of us run services and kind of need to generate AWS credentials for the services to run and operate in our cloud environments. But Amazon provides you really good solutions to bypass that, right? Come up with instance profiles that you can sign permissions to instances where you're running your services. And you can leverage IAM roles for those services. Another addition that works, I've seen it work very well in the very contained environments where you exactly know where the services are running and which IP addresses they're gonna be using. One of the suggestions I've heard is adding conditional statements to your policies for the service accounts to restrict it to specific IP addresses that you're using for your infrastructure. Couple of other quick pointers, right? Do not ever please follow me as give out the start out star to your service accounts. It's a bad idea. Couple of ways to prevent from, if you do have to use AWS key and secret key, have prevented from getting into your source code, especially to your public repos. There are tools out there that can allow you to scan code before it gets committed to your repos. And in some cases, like Key Nuker could potentially go and invalidate those keys before they end up on your public GitHub repos. So, other suggestions on a general side, on the protection, just keep the default limits. I know a person that by default, whenever they would spin up a new brand new account with requests from AWS, there's exorbitant limits to like 10,000 instances, 20,000 instances, so you doesn't have to deal with it later, but in a situation where I had before that could have been devastating to the company. There are a couple of benchmarks you can apply to AWS accounts and a couple of companies here, like Threadstack, for example, that are a sponsor of this event. They can help you configure your AWS account to the best practices. And by all means, please enable CloudTrail in your configuration. That's the only way you can audit to see exactly what's happening in your accounts. So pretty important. Couple of other quick notes. You can set up billing alerts directly from your AWS accounts and trigger an email warnings whenever you detect sudden costs increase or spike. Or you can use tools like CloudHealth to configure similar alerts in your environments. Some other points about the CloudTrail monitoring. One of the best practices and part of the well-architected review from AWS, they do recommend and mandate now that all your CloudTrail accounts, all your CloudTrail data from multiple accounts should be all dropped into a centralized, kind of secure AWS account. So it can be tampered with or accessed by somebody else. And then on the real-time monitoring and alerting piece, couple of tools like Threadstack, SumoLogic, Splunk, all this alerting and monitoring services, you can feed CloudTrail data into them and then you can look for suspicious activity. Like for example, the first one I pointed out, the way the access key got rotated, you can flag those activities and report on them through those alerting and monitoring tools. So that's it. One last item there on this particular topic. I did notice, right? Because those AMIs were publicly shared from an attacker's AWS accounts. And you typically can determine the provenance of those AMIs that you're using your infrastructure. They either be coming from your OS provider like you, Ubuntu or Red Hat, et cetera. So there's a kind of static account IDs or your own accounts where you're baking your own instances, for example. You know those account IDs, so you can potentially track to see if you have some foreign AMIs being spun up in your accounts as well. So I think that's it. One of the things I wanna just make sure that when you leave this conference and you start looking to your account configurations that you focus on three things, right? Key takeaways that I wanna make sure that everybody takes in considerations when they go back to work. Enforcing the MFA on the root account and users, essentially, very essential and it will prevent majority of the attacks. The attack that we saw on our own infrastructure when one of our developers committed by mistake the content of their home directory to a public repo including that dot files. A minute later, I kid you not, a minute he committed this data to a public group, a minute later after we were back to look at the cloud trail data, a minute later we saw denied access requests to change the AWS key. It takes a minute from you posting that data to a public repo and somebody tracking it and using it. So MFA prevented this particular attack, right? And the user would have pretty significant privileges, right? So essential to enforce MFA. Get rid of AWS key and secret key. Amazon provides you tools and services to free not to use them at all. It takes a little bit of work, but please do that. IAM role's a more secure way of doling out privileges in AWS. And then for the love of God, please, enable cloud trail, I can't stress that enough. It was painful to reconstruct this whole incident by looking at database snapshots like every 30 minutes. Cloud trail would have made it so much easier. All the data is in there. So that's it. Thank you guys. So with 200 instances for about, what, six, seven days, do you have any estimates for what the profit of that attack was? A year, well, it was March last year. So I assume that at the time it was quite significant. I mean, it's actually free money, right? The fact that we still don't know exactly which type of currency was used. It could have been Monero, Bitcoin, it could be Ethereum. There's no indication of which one it was, but I know for sure that it definitely was not like protein folding that they were computing on this. Thanks. Thanks for that. That was great. What, do you get a sense of how automated these attacks are, like how much of this human seeing vulnerability and pressing a button, or if it's just like troves of bots roving the internet automatically setting up machines? I believe it's highly automated. And there's, if you look at the timeline there, there was a, obviously the attackers are prepped for it. So they already have pre-bundled AMIs baked all over the regions. And they, as soon as they discover the credentials that they can use, within 10 minutes you can expect that you're gonna have infrastructure running in your environment, mining Bitcoin or cryptocurrency or other. Highly automated. I don't know exactly what was used in automation. If we had cloud trail logs, it would have been probably easier, but I suspect something like Terraform CloudFormation templates or just straight API calls. Hey, good presentation. What about MFA on mobile devices like Google Authenticator, is that fine? Yeah, absolutely. You could use it for your root account as well. Makes it a little bit less secure. But for the users, mobile MFA is perfectly fine. Thanks. What's your advice about robot accounts? Robot accounts, you mean service? And MFA. Service accounts? No, like AWS accounts that are not human people automation accounts. I think IP whitelisting restrictions are given out very specific privileges for the robot accounts to be able to do its work. Typically works. You know, key, secret key. And again, I made a recommendation to not use them at all. Amazon allows you to set up trust between AWS accounts using IAM roles. By all means use that. This is the safest, most secure way of giving some other account in AWS, giving access to your policies or resources. I'm kinda curious, cause you talked a lot about the financial cost of when it happened. They had to use this as a spun up. But from the time they contacted support and got everything wiped and no one was really accessing the account, what was the logistical cost to them of how long until they were able to actually legitimately use their account again, get in there and start pushing out builds and making progress on what they were expecting to be doing? Yeah, in this particular case, the customer in question had about thousands of AWS accounts. And it was only one of the accounts that got compromised. It was a development account with very little things in it. It literally had like five or so IAM users and no resources at all. So for them, the cost of cleanup was or time to get it back was negligible. But I can suspect that for other customers it could be a major headache, right? Because you need to audit and check absolutely everything to make sure that nothing else got changed or hijacked. Somebody didn't create another AWS account with kind of backdoor privileges in it. So it could be costly and time consuming. What was the fallout like for your team when your developer committed his home directory to a public repo? Yeah. Yeah. Just a side story, we use Slack quite heavily. And the developer had, along with his own data and his home directory, had a Slack token that was used for some of the integration work. And Slack token, if you've seen it, it has a very specific set of characters at the beginning. So it's very detectable. And about 10 minutes after you're committed to the GitHub repo, we got an email from Slack Zendesk, notifying that they've detected and invalidated their own token. And they provided a link where they discovered it. And upon looking at it, I'm like, oh, uh-oh. I see the guy's name, GitHub, public repo, and I'm seeing a whole bunch of other files that should not be there. And it was a panic mode, trying to call the person up. Obviously it's his own private repo. I can't do anything about it. But I finally got hold of him when we started doing the cleanup effort. And then during the investigation, when we were looking at our own CloudTrail data, we quickly determined that whatever credentials he had for AWS in his own repository got rejected, essentially, because of the two-factor authentication. So by all means, set up two-factor. And the policy that AWS, the one I provided, makes it so much easier. It's an overlay that snaps right in, over-existing policies and roles, that makes it much more secure. Just going back to MFA real quick. So you know how some password managers, like one password, let you save the MFA token into the vault. And I know that sometimes people have talked to insecurity, say that they're not a fan of that kind of feature, because then when they try to take down someone's access, they might still have a copy of that URL. So I guess what you're paying on people who try to get around the multi of the MFA, and use one device. It's hard to get around the multi-factor authentication because the user identity itself is stored and managed in AWS. So if you disable that user in there, they don't have access anymore. But in terms of using the on the endpoint kind of MFA generated tools, essentially downgrading itself to a single factor, right? You're typing in your password on the same machine where you have that token being displayed. If somebody has access to your machine, key logger, screenshots, they essentially can get access right from your machine, just being on your machine. Having it on a separate device, mobile device, is essential, just keep it separated. Can you please give more detail or examples on how the keys and secret keys can be abused or misused? Well, the worst one I pointed out was when the company got completely obliterated and destroyed, that's the worst case scenario. I wanna make sure that everybody realizes that storing your data in a single cloud or a single AWS account is not safe. You wanna make sure that you have access strategy backups going somewhere else in a separate S3 bucket in a completely standalone separate AWS account. You can set up the S3 bucket to S3 bucket replication to avoid that. There's multiple ways of, other ways of abusing those key and secret keys, obviously data breaches, right? Tesla, in this particular case, and one of the examples I gave had an open Kubernetes cluster running in their kind of development account, and the Kubernetes cluster had a couple of workloads running on them that exposed certain secrets in the environment variables, including key and secret key that then was used by the attackers to go and forge their S3 buckets that belong to that account. So we're able to access data that was related to car metrics data used for development purposes. So it ranges from obviously something like crypto mining, to hijacking data, or using it to stage another attack on somebody else, or completely destroying a company and holding a company for ransom. How do you recommend handling service account access from containers if you're not using environment variables or a secret key access key? It depends where you're running the containers. A lot of the, for example, Kubernetes allows you to, if you're running on top of EC2, for example, right? It allows you to leverage and dole out specific permissions from the instance profile to the pods of services. They're running this Qt2 IAM or something like that. There's a plugin that allows you to leverage IAM roles that are doled out to the instances. So you don't have to store that key and secret key. Cool, and that's all the time we have. Thanks again, Anton, awesome. Give him a round of applause. Thank you. Thank you.