 Hi. Thanks for joining us today. My name is Ben Feinstein. I'm the Director of Operations and Analysis for the Dell SecureWorks Counterthreat Unit. To my left is my colleague Jeff Jarmak. If we could get his mic hot as well. Can we get that mic on? So I'm Jeff Jarmak. I'm a security researcher with the Dell SecureWorks Counterthreat Unit. So today we'll be giving a talk we call Get Off My Cloud. Just to start with a little thought or a pictograph, if you will. This is one conception of multi-tenancy in the cloud. Sort of a crazy bazaar of different customers. All sitting next to each other on the same shared infrastructure. This is one way you can conceive of that. So what are we going to be talking about today? Well, when we set out to start this work some months ago, we wanted to first understand more about Amazon's cloud platform. We didn't set out to do this to pick on Amazon by any chance. But really, they're the big gorilla in the room when it comes to public clouds and infrastructure as a service cloud. So if you're going to be doing research as it relates to public cloud and infrastructure cloud, really Amazon is the biggest player right now. So we wanted to understand all the different types of credentials that you use when you're using Amazon's web services. We really wanted to understand sort of the order of precedence of all those different credentials. So if you were to have one type of credential, what could you do with it and what other types of credentials could you control or manipulate? Also understanding common mistakes and pitfalls of people or organizations that are using Amazon's cloud services. So part of this is looking at best practices and the guidance that's published, but also understanding what are some really easy ways to make some bad mistakes out there. And then with that in mind, we set off to develop a set of tools to detect instances of these problems. Basically cases where credentials could be exposed within Amazon's virtual machine images or rather virtual machine images that are published within Amazon's cloud by third parties. And also these tools would detect malicious images or backdoor images that could be out there in the public set of images. Also we performed an experiment to better quantify the scale of potential victims. If you were to release a malicious machine image and we'll define some of these terms, so I apologize if this is a terminology up front, but if you were to release a malicious virtual machine image and publish it in Amazon's cloud, we did an experiment to get a sense of how many victims you could count on receiving and how many of those victims you might actually be able to take control of their virtual machine. And also throughout this work, we maintained, our work was consistent with our reading of Amazon Web Services customer agreement and their terms of service that are published on their website. So organizations are moving all their infrastructure into the cloud at a rapid pace. There's a number of reasons why they're doing that, but really you shouldn't fear the cloud or you shouldn't blindly embrace the cloud. It's really just a tool. Like any other tool, there's good uses, there's bad uses and there's really suicidal uses. So you could think of it sort of like a knife. You know, there's obviously some good uses for knives here. In this case, Crocodile Dundee, he has a knife. There's lower costs in the cloud. There's decreased time to market, which is very attractive for organizations today. You can rapidly scale out your infrastructure without having to purchase data centers and hardware and servers and storage. And also, you could inexpensively get geographically diverse infrastructure by using cloud services such as Amazon's. So there's obviously some bad uses for the cloud, just like bad uses for a knife. There's relative anonymity to be found. Basically you can spin up instances in different regions of the world. It's very difficult for a third party to determine who's the actual actor that's behind that virtual machine image or behind that IP address that's sitting in Amazon's cloud. It's inexpensive or if you perhaps have stolen credit cards or you've stolen credentials to other cloud users, it's free. It's very hard for defenders to blacklist cloud infrastructure. The IP addresses are ephemeral. They change rapidly. You're going to do a tremendous impact to your business if you just blacklist large swaths of Amazon's cloud because there's a lot of legitimate uses and legitimate services that are running there. It makes geo IP address blocking or geographical blocking much more difficult because you can just spin up a virtual instance in any number of places around the globe to kind of hide where the actual location of the attacker is. And again, it's highly scalable. That plays for both good and bad. If a malicious party wishes to have a highly scalable infrastructure to support cyber crime or some other aspect, well, the cloud will offer that as well. You can find any number of press reports about spammers using cloud services. Even malware is now starting to attack cloud services. There was a case of a spy itrogen that had been modified to access Amazon S3 and compromise some buckets in that. And what we're really talking about also is the suicidal uses of the cloud here. And inadvertently, we found that many publishers of third-party images but also many users of the cloud that are using third-party images are putting themselves at great risk. So don't be this guy. You need to look at the cloud but do it with open eyes and do it by considering the risks and looking at the best practices and making sure that you're really adhering to the published guidance out there so that you can use these cloud services but using it in a safe way and take assessment of the risk. So some terminology. I apologize for those of you that are already, this is already an old hat to you, but when we gave some dry runs of this, many folks weren't really intimately familiar with Amazon's web services yet and there's a whole lot of acronyms out there that we'll be using. So AWS, that's Amazon web services. That's sort of the overarching suite of different services that they offer for their cloud service delivery. EC2, you'll hear us use that term. That's their elastic compute cloud. It's essentially an infrastructure as a service. You can get shared compute, storage and network through the EC2. An AMI or AMI, I've heard it called as well, is the Amazon machine image. It's essentially, what it is, it's a virtual appliance container format. So you would pick an Amazon machine image and you could launch any number of instances of that image and those actually become virtual machines that would be running in the EC2 cloud. S3 or simple storage service. That's object storage that Amazon offers its customers. You basically use what's known as buckets and you read and write data into those buckets as objects. And then a more recent addition, though it's probably been out for a few years now, is the elastic block store. And that is a virtualized block device. Just like on a Unix or Linux system, you can mount a block device on that system and read and write to it just like a file system. EBS is a cloud file system, a cloud block device that you can mount on your images there. So when we first set out to do this project, we wanted to understand all the different kinds of credentials out there and what the different uses of those credentials are and then which ones control which other credentials. So there's really three broad categories of credentials for Amazon Web Services. There's access credentials, there's sign-in credentials and then there's a set of different account identifiers that you have to use. In terms of access credentials, you've got access keys. An access key is merely a long, unique string of digits. There's a public and a secret part of that. It's a little bit analogous to certificates, but there are certificates in play as well. We'll talk about those. But your access key ID also has a secret access key and what you use that for are authenticated Web Services APIs like SOAP APIs to use simple storage service or Amazon's Mechanical Turk, which is essentially a service where you can get real human beings to do your bidding for pennies per request. There's a screenshot on the next slide where basically you manage all these different credentials within Amazon's Web Console. And these keys are also, the access keys are also used for CloudFront, which is Amazon's own content distribution network that they offer as part of their Web Services suite. Amazon's recommendation here, rotate those keys at least every 90 days. Here's a quick screenshot of the Nifty Web Interface where you can manage these access keys, both the secret key and the public key part of it. Another important credential is X509 Certificates. So many parts of Amazon Web Services use a certificate and private key to access or to sign your images, to bundle images. These are again, they're managed through Amazon's Web Services Console, but you also can do some of this with APIs. You can generate your own certificate or private key or you can provide your own certificate. This is sort of a trade-off of convenience for security. Either you let Amazon generate the secret key for you or you do that on a system. It's obviously recommended that you probably want to generate your secret key yourself and then provide up to Amazon. Use it to bundle your AMIs at cryptographically signs and encrypts AMIs that are private or just cryptographically signs the machine images that you're going to make public. Again, Amazon's recommendation, rotate these things every 90 days. So go issue new certificate and private key at least every 90 days for your infrastructure. Another screenshot here. EC2 key pairs. This is one of the biggest findings that we'll get into later of what we actually found when we scanned the US East Cloud with our tools that we're releasing today. They're SSH key pairs for all intensive purposes. When you spin up, when you launch an instance of a machine image, you specify an SSH key pair for that image to load. And on boot up, it essentially loads the private, excuse me, loads the public key into the SSH authorized key file and then you use your private key to access it to SSH and get a console on that virtual machine. So it's a very convenient way to get secure access to a virtual machine. You don't need to bake an authorized key file into the image itself. You specify it at runtime when you spin up a virtual machine. There's no explicit security recommendations I've found from Amazon around these key pairs or rotating them or such. And there's also interestingly, there's Windows virtual machines that you can run in the EC2. And these key pairs play a role there where literally the administrative RDP password is encrypted with the private key and you would decrypt it and then access the RDP port on the Windows image. So it plays a role in accessing Windows images as well. And you can log into the web console and play with these things and allocate your keys and such. Cloud front key pairs, yet another set of credentials. What we found is there's so many different access keys, private keys, cloud front key pairs that it's quite confusing. And really the first part of our research was just figuring out and identifying and enumerating all the different credentials that you have to use when you use Amazon's web services. This is the key you use when you're using their CDN network to generate signed URLs. It's essentially a way of offering private content within Amazon's content distribution network. So pretty much the one key to rule them all is the sign in credential. So you'll log in that you log into their administrative web interface and control all aspects of your Amazon web services account. So if you're going to defend any of these and secure them strongly, this is probably the most important set of credentials there are. What it is, if you ever bought a book from Amazon or a CD, it's your username and password. All you do is take a normal Amazon account, you may have purchased the book and you activate web services on that account. Amazon's web services now the credential website is now the credential to your web services account. They've just also offered multi-factor authentication. It's not RSA Security. So you can protect this with multi-factor auth. You do a purchase of $12 or $13 token online. You activate your account with this token and then now you've got multi-factor authentication to add to just the username and password to protect this. This is really important for the web services. So if you're going to be using cloud services so that you're not, your whole kingdom isn't just relying on one username and password here. They haven't rolled out the kit and retinal scan just yet. Account identifiers, there's two account identifiers. There's a canonical one which you use for certain APIs and there's just another ID that you use for other APIs and it's not really a secret but it's a long story. So as we're going through this, we found some prior research that we thought was interesting and we wanted to call out. So a few years ago, back at Black Hat 09, DEF CON 17, a group of researchers including Alex Stamos delivered a talk called Cloud Computing Models and Vulnerabilities Raining on the Trendy New Parade. And they showed two, among other things, they showed two interesting techniques. One was a way to essentially game the system and then the list of public machine images of their own machine image. So literally there's hundreds of different, say, Fedora images that are available in the public cloud. And the key is if you're not really on that first page of search results, people probably aren't going to pick your machine image. So they were doing this so that they could get a set of victim users that would run, I wouldn't say a back door machine image but a machine that had a phone home mechanism in it. They got prime placement for their virtual machine image and then they had a phone home script that was running every time you spun it up and they could measure how many people did this. There's some interesting precedent that was published earlier this year. So in a tipping points blog published a copy of this letter that Amazon sent to their customers back in April. And essentially someone had found that an image had been shared with and had authorized SSH keys baked into it. So the publisher of this image could potentially have full access as root to anybody that chose this machine image and spun up instances of it. So this also reinforced to us that this is something we wanted to look at more closely and this is why we built these tools to go ahead and evaluate large numbers of machine images to look for things such as SSH authorized keys that shouldn't be there. So there was after we were accepted for this talk there was a group in Germany, some postdoc researchers at a university there that released some researchers that's very similar to ours. They also released a tool that scans AMIs in a very similar fashion to what we do. Their tool is a python script that essentially you have to run on the instances itself. Because our tool interacts with the Amazon APIs to spin up instances, SSH to them, scan them remotely and turn them down. So they're similar, but the approaches are slightly different and we've scanned more images and have a little bit larger data set from our tool. But I just wanted to highlight that there's other people working on this and it's possible that there's others that aren't publishing it. So this is probably something that's being done in the wild. So when creating a public AMI or an AMI that you share with other users there's a number of ways that you can accidentally leak your own credential material. On the file system itself you may have your Amazon web service certificate and private key. You might have SSH key pairs either an authorized key that Ben mentioned or an actual identity key. You might have SSL certificates and private keys that are on the image. You need to be conscious as a publisher of an image what you're putting out there because once it's public it is truly public and anybody can access these images and search for these sorts of credential materials and that's essentially what we've done. There's also other ways to leak information. Bash History Files is a big one. If you run the image as an instance yourself and done any sort of work on it and then made a public image out of it that Bash History might expose that and it might also expose some of these credential materials that you've used. Files like Bash RC, Bash Profile often contain environment variables that are set to these values for scripts to access and things like that. So we looked for these as well. And then also Vim Info Files occasionally there's a chance someone might be trying to clean up and go in and edit a file to delete it but they search for that and their Vim Info stores that search for that string so it could be there as well. And then we tried to do a little bit of looking for signs of a malicious AMI. This is primarily the SSH authorized keys. There's possibilities for root kits, trojan binaries reverse shells connect backs those sorts of things with the PV grub kernels that they support now you can run your own kernel on EC2 so you could back to where the kernel directly if you wanted to. There was a talk today in SkyTalks Anche with 303 demonstrated an AMI image that essentially when it was launched it would phone back, download a piece of code execute it and he'd get a interpreter shell from it anytime somebody spun that instance up so that's also related work to this. So there's been other work on malicious AMIs. Ben mentioned some of the work from 2009 from DEF CON 17. This is something anybody who's looked at Amazon Web Services thinks of this that these images are public and there's not much verification of who the publishers are and what their intentions are and in a lot of cases it's really difficult to detect malicious intent and malicious behavior. So we undertook a project to try and gather some data surrounding this. How easy is it to find victims what kind of instances are they running it on where regions are most popular and kind of most importantly with their security groups which is similar to a firewall policy for an EC2 instance would those be configured sanely don't allow the world to connect to your SSH ports, for example which would be effective at being somewhat of a hedge against an AMI that has atrogened SSH to AMN or pre-existing authorized key. And then we also released our tool well developed our tool that we're releasing to perform large scale scans of images within the cloud. So the name of the tool is AMI Exposed. So it's kind of a pun on AMI and asking a question and the tool seeks to find the answer. So it's a framework essentially for scanning AMI's for these credential leakage patterns. I tried to follow a sort of similar model to what Metasploit does although my code is nowhere near as elegant and nowhere near as capable as the stuff they've got. But in a similar vein there's test modules that you can plug in to add new functionality with very little code and leverage the existing classes the existing object model to add new tests. So at its heart it uses Amazon's API to automate the retrieval of a list of images defining your scope of your tests. I'd imagine common use cases would be testing all images that I own maybe before I make them public if I'm planning to do that. As a penetration tester you might be interested in testing all images that are owned by a specific client to see if you can find any other credentials being stored publicly. So once the scope's generated we basically iterate through all the images for instance run tests over SSH and record findings to a database. The tools released on our website secureworks.com slash research slash tools there's currently a little bit of a problem with the file that we're going to get corrected probably Monday morning. So if you have any trouble downloading it in the meantime I apologize for that. As I said there's several test modules included at release. There tend to be precise. They check for the things that we've been discussing and I'm not going to keep repeating all these different vectors. It also tests within system files so it'll search for a number of files and then search within them for specific strings. These strings are all definable by a configuration file. You can just pop a regex in there and it'll search for that regex and you don't necessarily even need to write a new module to add a new string that you're interested in. So here's kind of the flow of how it executes up at the top there we've got our scope definition phase. You can either do this manually just by going into the AWS console and putting tags on images that you're interested in or there's a tool that's included called tagimages.rb that allows you to define larger scopes more easily. You can't go and click through thousands of images. Again query for the tagged images, iterate over them and obviously we want it to thread it. Oh, I should mention this is all in Ruby. I didn't even say that but down the right here we have some of the gems that are used. So we're using the threaded collections gem for managing threads and then we're using the, well, the official Amazon Web Service SDK for Ruby which was released about three weeks ago. So I actually kind of wrote two versions of this tool. I spent a lot of time writing one that interacted directly with the SOAP APIs over parsing the XML to my own object model. I was just about done with that and about to do our large scale scan when Amazon released their own SDK. It was much more elegant, much cleaner, much more efficient. So I ported my code over to adopt that rather than duplicating that functionality. They've also been improving it at a pretty rapid pace. So there's been a couple releases that have made minor adjustments and bug fixes just in the past few weeks. So once we launched the instance, we have to figure out what account we're going to use. So that's down there on determining SSH username. So we try and log in as root. If that fails, we try and parse any response that comes back. A lot of times you log in as root and you get a response that says please use ABC instead of root. So we try and parse that on the banner. And then if that fails we just iterate through a list of usernames that we've discovered are commonly used out there. So generally we're pretty capable of being able to determine the username. And then we run our tests. So we go through each of the test modules, load the module, run it and record all the findings in the database. So we're using net SSH for the SSH components and active record for the database component. So I'm going to demo it. It's going to take a little while to run so we'll talk a little bit more about the internals while it's running. Give me a moment here. Apologize for the delay. Okay, you see my screen? Yeah. Okay, so out here this is the AWS console. I'm showing three images. These are all images that I've created. I'm not going to show anybody else's image, but my own. So two of these, this first one is a public image that I created that's an AMI that allows you to run backtrack 5 within Amazon's cloud. And we'll talk about that a little bit more later. The second one is from when I was developing that image. This is a private instance or a private image. But you'll notice I scanned it previously and it's marked as failed. You know, as I was iterating through the process I broke the SSH statement on this one. It starts, but I can't SSH to it. So I just wanted to show that it flags it as failed. And then the other one down there is one that I made just for the demo here that demonstrates some of these key material leakages. So I've got things there that look like real keys. They are real keys, but I'm not using them anywhere and it's not live data. It's just for demonstration purposes. But you'll notice there's a tag on these and two of them are in pending state. And that's how we define our scope. And that is those two images. So let me just do this full screen. Oops. Okay, is that still there? All right. So this is actually I'm SSH'd into a running instance on Amazon's cloud and that's where I did all my tests from. So it's kind of interesting that I was using the cloud to scan the cloud for security problems. So it was all taking place in Amazon's infrastructure. So this is a directory where I've got the tool here. You'll see this tag images file that we talked about that allows defining the scope. And then scan.rb is really the main the main heart of the tool. So I'm going to go ahead and launch scan.rb. It looks a little bit weird here. My ASCII art in scale to this font size. So you see there it says it's storing logs for each AMI under a logs AMI category. Starting in tests of two AMIs. I've got to configure for 13 threads but it's not going to run 13 because it's only got two images. And then just timestamps saying it's scanning. So this probably takes about 10 or 15 minutes. So we will go back and come check on this. I keep doing that. Come check on this in a little while. Okay, we got through this. Sorry, lost my spot. Okay. So while that's running we talk about some of the things I did to expand the Amazon Web Service SDK. I basically just opened their classes and defined some additional methods to give me the functionality I needed to do this that they didn't provide through the SDK. So we modified the instance, added some methods to interact with instances, run a command over SSH, either via SSH XAC or PDY. And then it's got a run test method that basically loads a test module, runs it against that instance. I'm going to go through these kind of fast. The slides are a little bit wordy, but that's mostly meant as a reference. I modified the image class to add another method there. Added some places to store parameters. We discovered that SSH user and I need to store it somewhere. So I just stored as a tag. Also that test status updates that tag that defined our scope and defines the results of the test. Created a finding class. It's a really simple class. It's just backed by ActiveRecord. So when you instantiate a finding object, it just logs it in the database. And then there's testRB is our base test class. So this is the parent class of all the test modules. It has some common functionality that's used across test modules, so you don't have to rewrite it on each one. And then it implements the file helper module that allows us to test within files. We're not redefining that everywhere. Any test that needs to look within a file for contents can just call this method and it's already taken care of. And then the individual test themselves derive from the base test. They have an init method that just basically defines some metadata of the name of the test. The severity of various findings, etc. And then a run method that runs the test. So we're trying to keep things simple by having a sound object model and keeping all of well, not all, but keeping a lot of the heavy lifting out of the tests. So here's an example of a test file at the end of the day. This is a pretty simple one. Just checks for a bash history file if it exists. It looks within that bash history file for things that we've defined as potential key materials or credentials. This one's similar here, although I got the title of the slide wrong. This one's SSH authorized keys. So this is where we're looking for any authorized key file. Looking within that file to see if there's any keys present other than the one that is provided through the Amazon interface. So this will identify unknown keys. This one's a little bit more complicated. It was an attempt to try and catch any sort of persistent connections. So we're looking at a net stat and seeing if there's active connections from the host, excluding a few IPs, not going to look at loopback connections, not going to look at connections that are coming from my test box, et cetera. This isn't terribly effective because it has to be a really long-lived connection in order to be caught. If you're doing something like hitting SSH and downloading a script, or I'm sorry, hitting an HTTP server and downloading the script, that's going to be a really quick connection, and the odds that we would catch that are pretty slim. If somebody had an SSH session persistent for a long period of time and we happen to test at the right time, there's a chance. So let's talk about the scope of our testing. We started with the idea of looking at all US East images. We removed windows images from the scope. They're generally paid images because of licensing constraints and they're more difficult to interact with remotely because you have to do it all over RDP. It's a little harder to programmatically access. So that left us with 5,515 non-windows images. This was the list of images that we attempted to process with the tool. There were 771 images that were paid, so you need to have an additional license, pay an additional fee to use those, and we just kind of skipped those because we were trying to keep costs under control. There was 2,767 that failed to test, and this is really kind of interesting. It seems like almost all of these were due to some sort of error with the image itself. So you saw the one that I had where my SSHD was broken. That was a private image. It's broken. Why would I make it public? But it seems like a lot of people make these public anyway. So I couldn't interact with them over SSH because they won't start. You can't find the root volume on boot. SSHD doesn't work, etc. So we ended up with 19, well 1,977 images that were scanned by our tool. We had 580 unique images that had one finding or another. So that's 29.3% of the images that completed all the tests had at least one security problem. We shared these findings with the Amazon security team. We didn't verify any of the credentials. We didn't attempt to use any of this stuff. So it's possible some of these have been revoked or something like that. I have no way of knowing. And Amazon has been responsive about trying to contact the image owners and the users if it's applicable and make everyone aware and remove images if necessary, etc. So I really want to applaud Amazon's security team. They take the threat seriously and they've been tremendously cooperative with us. So the largest finding was the problem of SSH authorized keys. I suspect that many of these are people that just didn't realize they were there. They're probably not it's hard to say how many of them are malicious. But the fact is when you download an image that has an authorized key file, whoever has that private key, if they can reach the SSH port, can authenticate. So there's no real way to separate what's accidental and what's malicious. It's a pretty severe finding. So 19.5 2% of the AMIs we tested nearly 20% had an unknown author as key file. So by that statistic if you randomly launched 5 public AMIs there's a good chance that somebody else has access to one of them. This represented 44.96 of our findings. So it's far and away the most prevalent problem that we've encountered. This chart here breaks down all the different findings that our various test modules encountered. The number of occurrences of each, the number of unique images that was affected by that finding. The percentage of our total number of findings that were represented by that class. And the percentage of AMIs that had that problem. Or I'm sorry, the percentage of the percentage of AMIs. So that's the percentage of all AMIs with findings. So if we take the number of images that had on our SSH keys as the numerator and the number of images that had some problem that we discovered as the denominator that's what that percentage represents. And then we have the percentage of our test case. So we see there again on authorized SSH keys existed in nearly 20%. I believe the German researchers, their number was 30%. But I'm not quite sure what the scope, they tested 1,000 images. We tested nearly 2,000. So that's going to vary a little bit. And I'm not quite sure what region they tested. I sort of suspect it might be EU just because they're from Germany. But I'm really not sure. So I don't know how directly comparable our findings are with each other, but it seems we're both in the same ballpark. Our second highest finding was just the existence of a history file. So that in and of itself may or may not be a significant finding. If that history file contains sensitive environment variables, key materials, et cetera, we would log another finding. But in some cases the history file might have some sensitive info that we weren't scanning for. You know, the other tops there environment variables that are commonly used to point to Amazon Web Services key materials. The access keys themselves that's a significant one. It's not a huge number of images. But with those keys essentially if we'd wanted to if we were so inclined we could have scanned until we found one of these and then continued the project by using that credential and reduced our cost quite a bit, but we're not trying to be criminals here, we're just trying to gather the scope of the problem. Other issues SSH identity keys, so that's the actual private key, which is kind of unusual. In at least one case I was able to find the identity key file and then go back to the history file and find a remote host name that they'd logged into with that identity key. So chances are good if I'd used that same key against that same host, I probably would have gotten that host, which isn't on Amazon's cloud but may have had its security compromised as a result of this lax practice of keeping the key there. And there's various other findings much lower proportion, so I'm not going to discuss them all. The SSH paths were authentication enabled. We actually added that test as we were like most of the way done with our scan. So those numbers are really incomplete. But that's a module that's looking for an SSHD config that has password enabled. Amazon Web Service EC2 is really built strongly around key authentication. So if you have a password authentication, now the strength of passwords and accounts that are configured on the image comes into play. So we didn't find a whole lot but as I said we can test all of these with that test. So let's talk a little bit about the costs. Excuse me. My Amazon Web Service bill last month was $1,333. I completed all these tests within the month and the vast majority of that is due to this project. It's a little too bad. $3.84 more than I would have been at $1,337. So that would have been kind of cool. I wish I would have figured that out. I wonder if they'll let me overpay the bill. Just here's $3.84 more. Anyhow that breaks down to $0.67 per image scan. $0.98 per finding. $15.68 per Amazon Web Service credential. So that seems like a bargain. With a little bit more manual review, these things are fine. That are for example a batch history file, if you were to look at it you might be able to find other things and have more findings. Kind of conversely you might find that some are more false positives. So our numbers are obviously rough and the costs per finding are a little bit rough there as well. We could have leveraged spot instances to lower costs. So spot instances use off peak pricing. If you don't care when you run a task you can run it whenever Amazon has more capacity available at a lower cost. It would have made the script a little bit more complex to keep state of what machines are running and not. And there's also the thought of analyzing the EBS boot images themselves without launching the instance. This might reduce costs. It might also increase the amount of images we're able to test. Some of those public AMIs that wouldn't boot properly if they have a valid EBS volume, that volume might still have data on it. The fact that the machine doesn't boot doesn't mean there's no data there. And I sort of suspect that if someone's going to go through trouble of making a broken image public they're probably not going to be following some of the best practices in securing their credentials as well. So that might be interesting. And again, as I said before if we were malicious about this we could have reduced our costs pretty greatly. Let me go back to our test there. That should be just about complete now. Ten minutes? Oh wow. Okay. I'm going to have to pick up the pace here. Okay. So keep changes. Okay. So this is our test here and you can see those two threads completed. Those tests completed. By going to the log AMIs directory this might be a little hard to read at this size. But let me look at the log file for the first one here. O4BE. So, yeah, it doesn't wrap around too well here with this size. Is that, maybe I can shrink it as well. Anyway, you can see here it starts up at the beginning, launches an instance. It's got the instance ID number there. Temps test this H2, SSH hadn't started yet so it sleeps for a little while. Discovers the SSH username and then starts running tests. This particular image didn't have any issues. So it runs a test, completes a test, runs a test, completes a test, etc. Not a whole lot interesting there. It eventually terminates and shuts it down. If we look at the other the other test this one's a little more interesting. So same sort of things. Start an instance, discover SSH. Now there you can see we found an unauthorized SSH key. We called that high severity and there's the key. You know, we checked for selects 509 certificates. Found a certificate file. There's the file name and path. It just kind of continues on like that with the various findings. These are also all stored in a database if I go back to using SQLite here but it's active record so you can use whatever adapter you want. Select start from findings and you'll see we've got our database there you know that logs all our findings where things were found and the details that go along with them. So that was a demo showing two machines being scanned in real time. At our peak we were scanning about 110 images an hour and when I say scanning that includes identifying images as paid images which obviously doesn't take very long but that seemed like a pretty decent rate. It took about two days it was two and a half, three days to scan the entire scope of our tests. But the question then kind of becomes the SSH keys was our biggest finding. Don't people use security groups to protect their instances? If I have an SSH key it shouldn't matter if I can't reach the SSH port. So here's just a slide of explaining EC2 security groups. It's just like a firewall policy there's screen shot there. The interesting thing is they're inbound only unless you're in virtual private cloud. So outbound traffic from a potentially malicious AMI there's really not a strong way to filter that. You can use IP tables or something on the instance but you know when it's someone else's provided AMI at least the first time you run it you're running their configuration. So we did some testing of security group practices. I released these AMIs across all regions to allow running Backtrack 5 in EC2 on June 23rd. Just kind of announced it publicly in a couple of different places. You know I thought this was an interesting sample because it's something that you know people that are more security minded might be interested in. Has anybody here actually used these at all? I'm just curious. No. Okay. Well it was also useful to me for gathering data on statistics. There was a phone home script in it. Now this wasn't a true backdoor it wasn't anything that I could log into Icon and Execute Code in the boxes but it did report some data back to me when it was launched. Here's the script. Oh it's on the next page. So there was a script that was part of you know part of the start up collected some metadata from the EC2 metadata API and sent it back to me. Here's the script itself and you'll notice there there's a comments kind of explaining like you know what we're trying to do and trying to calm people down that it's not real malicious. Notice the curl directs the entire output to DevNol so I can't save it as a script and run it. And you know it's got a website there if you have questions visit this. The website explained it and kind of solicited feedback to say how did you find this you know I'd like to kind of gather processes that have been helpful in discovering these sorts of things and I was planning on crediting people who had found it but as you can see nobody contacted me nobody hit that webpage so it doesn't seem like too many people found it. So we received 95 phone homes as of July 31st this was 69 unique instances and when we received the phone home we connect back on 22 and pull an SSH banner we were successful in 71.5% of cases 72% if you counted it by unique instances rather than all instances to get rid of the effects of people rebooting an existing instance so yeah that's kind of concerning that tells me that 70% of people who are presumably somewhat security minded because they're downloading backtrack after all you know the running backtrack in the cloud didn't bother to properly firewall off their SSH port if instead of putting this phone home you know I'd put the phone home in addition to the SSH key which you'll remember was our most common finding it's a pretty safe bet that I would have had root access to most if not all of these boxes so our lessons here we kind of summarize them already one interesting thing I think is that there's too much of a stretch to say that the average user is probably worse off than our sample again you know more sophisticated backdoors would be harder to detect than this and nobody detected this so that doesn't bode well for using malicious AMIs and actually finding them when you use a public AMI you're putting a lot of trust in the publisher so at the very least consider the source of the image who that person is super safe build your own images but at the very least consider that images from Amazon or another trustworthy provider official open source maintainers maybe more safe than trust in me so as we're in the midst of this project Amazon published a number of new documents given some security guidance and best practices we wanted to offer some links in our slide deck to these so if you're using our web services we encourage you to go check out all the guidance that Amazon published out there about sharing AMIs how to do it safely how to use public AMIs in a safe fashion and there's a whole bunch of articles out there on this elastic.com site from Eric Hammond so we would encourage you to check those out there's a lot of good material out there and Jeff mentioned this earlier so where would you obtain trustworthy third party AMIs besides from scratch yourself well Amazon themselves offer supported and maintained images you can follow the link up there or find it yourself and they've got a number of images out there they have their own yum repository security updates product life cycle if you want to pay for their premium support service these are the images they're going to support so this makes a good option to enterprises or other organizations that are using their services there there's a number of third party vendors that also provide their own AMI images right scale, cloud era, a number of other vendors and one of the best practices we've talked to some large organizations that heavily leverage web services from Amazon is they many of them take Amazon's own images as a base and then build a customized AMI for their own application suite on top of it and then they can leverage Amazon's life cycle and support and package updates to keep their base up to date and then just all they have to worry about is updating whatever the application code or other suite they added on top of that was if you find issues in the cloud we had a really good experience working with the Amazon security team they're very responsive they're easy to get a hold of they investigate every report so you can find them online they have a form PGP keys, email addresses it's not hard to get a hold of them I want to give thanks to Beatle from the Amazon web services security team and the rest of his team for working with us as we did this project we were creating work for them as they were doing customer notifications as we fed these findings to them and also thank our bosses at Dell SecureWorks for authorizing us to do this project and giving us a little bit of budget to make it happen we'll be moving over to the Q&A room here shortly across the hall whether using cloud services have questions or have some information to share with us so I hope some of y'all will join us in the Q&A room across the hall, thank you appreciate your time