 Okay. We'll get started again. So those of you who stuck around for both presentations, thank you very much. I'm honored and welcome to anyone new. This presentation is about entropy, something that I'm even more passionate about than encryption itself because entropy is essential to have quality, secure encryption actually happen. Some of this is, if you're familiar with the problem of entropy in virtual machines, I think you'll understand where this is coming from, maybe some of the suggestions toward the end can generate some conversation and some feedback, and maybe we can talk about how we can do a bit better in an open stack itself. So the background, if you're not aware, computers are predictable. Inherently, they're deterministic. That's usually what you want. You want to plug some inputs into a computer and you want it to come back with the same result every time. Fundamentally, that's the way we design processors. We want processors to do lots and lots of math for us, and we want it to come up with the right answer every time. The last time Intel created a processor that didn't come up with the right answer, I think we got really mad back in, so about 1996, 97 when the Pentium flaw. That would have been a great machine for generating randomness, but instead we scrapped about a billion processors. So typically, we actually do rely on hardware, real hardware to give us valid entropy. In terms of our laptops, our desktops, our personal computers, we actually have real hardware that's associated with it. We've got a physical keyboard, we've got a mouse, we've got a microphone. Nowadays, we have a camera. All of these inputs can grab environmental information that provides entropic input into the system. But in cloud instances, in virtual machine guests, we don't have real hardware. We don't have a physical keyboard that we can take timestamps in between key presses. We don't have a mouse. We don't have hardware. Any hardware we do have, the IRQs or the hardware itself is simulated, and the IRQs really don't have reliable timestamps or reliable differences in the timestamps. The block devices themselves are virtual and heavily cached by the hypervisor. We really want really fast storage coming through our hypervisor to our instance, and what that means is that that caching reduces the actual randomness of those reads and writes so much that there's not enough entropy to gather from block reads and writes. And the real-time clock is shared across instances, which has some problems in some situations and in other situations, it's not a big deal. Perhaps the biggest problem and the thing that I think would be most essential to address inside of OpenStack is the fact that the initial seed for these instances are typically part of the cloud image when it's generated. When Ubuntu generates its AMIs, there's an initial seed that's part of that cloud build that's part of that instance, and every AMI that starts has that same initial seed. Thankfully, that's not the only thing that goes into DevU-Random for seeding it, but it's part of it and it's the same every time. Having something different happen when you launch your instance from when an attacker launches an instance that they're trying to mimic yours, having something different between those two are essential to seeding the initial pseudo-randomness and then feeding that in over time helps keep the pool filled over the course of the history of the instance. So looking at the history, six years ago, there was a white paper called the Analysis of the Linux Random Number Generator 2006. The conclusions from that paper, number one, there was very little prior documentation of Linux, the Random Number Generator in Linux at the time. The authors of the paper, they were students at a university in Europe. Basically, they said they reverse engineered the Random Number Generator because the documentation in the kernel code was just that bad. They actually had to read the code and see the outputs and instrument it to tell how it was gathering its entropy. They also showed that random bits are a limited resource. We've known this for a long time, but it's actually pretty trivial to launch a local denial of service attack against the system's entropy. Any user, DevRandom in Linux's permission 666, any user can drain the entropy pool and then block other users that are depending on that entropy pool. In fact, remote denial of service attacks are actually possible because that entropy pool at the time at least was, DevRandom was being used for TCP packets, the sequence number in TCP is random. It takes a couple of bits of randomness from the Linux Random Number Generator to create your TCP packet. So a remote service or a remote malicious user could just open up a bunch of sends to any service you're running on TCP and over time drain your entropy pool, which would cause other processes that depended on that entropy pool to block. One of the suggestions from that paper was that each user should actually have their own Random Number Generator. We've never really gotten around to that, but we have drastically improved DevViewRandom. That was sort of the alternative to that. So the non-blocking pseudo-random Number Generator. The other thing that this paper showed was that there was very little entropy in embedded environments. This is 2006, a little bit before most of us had caught the cloud wave. They weren't really looking at virtual machines, but the equivalent to that back in the day was embedded environments. And so their research showed that OpenWRT, which is an embedded router distribution, or a Linux distribution that's run on routers, embedded routers, if you wanna flash your hardware. They showed how vulnerable OpenWRT was to attacks on the entropy pool. The fact that those devices don't have any hardware inputs. They don't have a keyboard or a mouse or anything like that. That they used a static seed at every single boot, and they showed how easy it was to cause collisions on OpenWRT routers at the time. And in general, the paper showed that OS startup in general consists of a sequence of the same routine, predictable processes every time. We've stirred that up a little bit in modern boot sequences with Upstart and SystemD parallelizing some of that. But for the most part, it's still a pretty regular set of processes that happen at boot time. What that means is that there's very little entropy right after, shortly after boot. So three years ago at the Black Hat Conference, this was, these are two slides I took from a presentation, a really good presentation, where these guys showed a handful of, I'm always embarrassed when I do this, but are any of these gentlemen in the room right now? No? Okay. It happens sometimes. You're talking about someone and they're sitting right there on the front row. Excellent presentation. I highly recommend you check it out. But basically at this conference, they defined a rather simple bit of pseudocode, a little algorithm for what they thought it would take to actually reverse the private key to an Amazon instance. Now this seems a little bit far-fetched. They would take a lot of resources to do this, but this isn't a real complicated process. I'll read through this because they're maybe a little hard to see in the back, but fingerprint a remote EC2 victim to the Terminance AMI. The AMIs are published if you're running an EC2, you're probably running an Ubuntu or an Amazon Linux or one of the rel images. It's easy to guess those. If it's not the one you want, move on to the next until you find the victim you want. Grab that image, grab the random seed that's inside of the image, re-bundle that image with an instrument module, run it several hundred times to get the IRQ disk timings and then use a pseudo-random number simulator to estimate the initial real-time clock. When was this booted? You can guess that sometimes within a few days or weeks. Simulate the SSH keygen, test against the fingerprint and do that ad nauseam. Is that practical? Well, you know, maybe, maybe not, but can we do better to protect this? And I think yes is the answer and I don't know that we've come real far, very far since 2009. So I'm here suggesting some of the things I think we could do different. Earlier this year, a paper that got a lot of press was published on factorable.net. This was another research paper out of university. It's called Minding Your Peas and Cues. It's a great paper. I re-read it on the plane over here yesterday. Basically, they did the biggest, most comprehensive internet-wide scan of public SSH keys, host keys and TLS certificates, grabbed as many public keys as they possibly could and showed how many insecure, poorly seated random number generators were in widespread use. And I think this statistic right here is the most sobering. Whoops, this second bullet down here. The fact that they were able to remotely obtain the RSA private keys for half of a percent of TLS hosts and three hundredths of a percent of SSH hosts, which doesn't sound like a lot, but that's a few root keys that some college students were able to come up with. Because they're public keys, because the keys were generated with poor randomness. Was that a comment, excuse me? Let's scale that out over a botnet of a few hundred thousand nodes, and that actually starts to look, so the first suggestion, we need to improve the random seeds, the seeds that we start our instances up with. There's some of these are suggestions, things we can do, some of the things that require new code to be written. First of all, I think it's a safe assumption that the host, the physical host, actually has access to real hardware. They've got real disks and real USB devices and other inputs. They could actually have PCI cards or entropy keys. For 50 bucks, you can buy a USB hardware entropy key. Hosts, physical hardware, actually have access to real hardware and better entropy. I would suggest, this is a proposal that part of the OpenStack Compute Instance Launch Process is for the host to write some random data to a file. And pass that into the hypervisor as a secondary disk image. I use the floppy disk here, KVM and a floppy disk. So grab a temp file on the host, DD some, either you random or real random data if you've got entropy key, use real random data to a file. Pass that in as a block device to that instance. KVM-FDB says, present this as if it's a floppy disk. And then the guest would simply read that at first boot. So test if that exists inside of the initRD, inside of the initRAM disk, bring up of the instance. If a floppy disk exists in a cloud environment, sure, I'm gonna take that as my random data and write that as a seed. I can't imagine any other realistic situation where a floppy disk might be attached to a cloud instance. But I think this would be a great, simple, easy to implement way inside of OpenStack Compute on instance instantiation to pass a better seed into an instance at boot. Ha-ha! The next slide. So the other way to do it would be to do it through the metadata service. This is again another feature request, not yet implemented, but the OpenStack metadata service should offer a random seed to each instance. That's a private channel between host and guest. The only reason I would suggest we do this as a block device is so that we can do it at the earliest point in a initRAM FS. To do it in the metadata service, we're gonna need networking, at least the metadata service networking up and readable. For bonus points, let's bake this into the metadata service and actually implement the Heisenberg principle where every time you read that, it changes. So a single read gives you 4096 bytes of random seed. You read it again, it's another different 4096 bytes. That way, you can't ever see what came before or what's coming next. You would. So this is a collection of suggestions. Hopefully the concatenation of which makes it spreads that risk around. The comments you'd really have to trust your cloud provider because your cloud provider is giving you the random seed. Both of these do require your cloud provider's trust here. I did this earlier this week with some help from Scott Moser here who's the author of Cloudinit, actually. Scott, help me show how to pass in the seed yourself as part of your metadata or your user data through Cloudinit. So in Cloudinit, you've got these stanzas. One of the stanzas supported is something called write files where you can tell it. I'm just gonna put some data in this in my user data but I wanna have it encoded this way. I want it installed as a file at this place with this user and permissions. So in this case, I generate a seed by grabbing some data from uRandom, I base64 encode it and wordwrap it so that it's all just one line. And write a file, I say that the encoding is base64. The content is that seed which is the big random blob. The owner's route, the path is gonna write it to dev uRandom or random. Doesn't really matter here. They're muxed together in such a way that that doesn't make a difference. And the permissions on that file traditionally and at least in Ubuntu are 666. So in this case, if you didn't trust your Cloud provider to give you good entropy or if they gave you some entropy but you wanna give yourself even more, which the concatenation of those are gonna be hashed and whitened together using AES and SHA, the output of that is gonna be even more difficult to predict if you provide a piece of it as well. So I would suggest using this in conjunction. If you're launching instances, basically, I'll just say this. If you're launching instances using Cloud Init and user data, I would do this everywhere all the time. It's so easy to do. If you're generating your user data, put this one line above your generation of that file and then in your file do something like this and pass in that randomness right there. I don't think Cloud Init can do this alone. So the question is would this, would it be worth just getting Cloud Init to do this? I don't think Cloud Init can do this all by itself. It can only, because you have to generate that randomness and put it into a template of your user data. So my statement is if you're using the user data to launch your instances, just add this to your template, right? If you're using Chef or Puppet or Juju or something else to launch instances and you're using user data to launch that, this is super simple. It's not gonna hurt you. It will definitely make your instance different than another instance launched at the exact same time, you know, clearly because this entropy was generated on your local desktop or laptop or something like that. So then another one, I've been working on a service and an actual an RFC to submit to the internet RFC community. But I've been working on a pretty simple process, a simple demon. I've got an implementation packaged in a PPA for Ubuntu. It'll be in 1304. Called a nerd for network asynchronous, network exchange of randomness demon. The idea here is for instances that are in a pool, collaborate with one another to exchange entropy over a UDP port, which conveniently drops packets and doesn't necessarily keep the packets in the same order. So that adds a bit of noise to what's going on here. Now obviously it's UDP, it's inside of a network, that's sniffable, that's viewable, but if you're outside of that, if you're the authors of the P's and Q's paper, this completely thwarts the P's and Q's approach. All they were doing was observing public keys from outside of a firewall. They had no idea what went on at the generation of that, but they made some guesses. They guessed the time that these machines were booted, the random seeds that these had. The idea here is that you've got a server and a client process. The server is a simple socket. It's written in a simple socket written in C. It listens on a UDP port. It takes whatever input it gets. It salts it with some timestamps of when it received those bytes. It writes those to its own entropy pool. You can't really dilute your entropy pool. It's the way it's mixed and mucked and hashed together. There's not a danger of bad entropy. All zeros would just get hashed in with the data that wasn't all zeros before. So there's not really a risk of diluting the entropy pool, but it would mix that into its own pool. Hopefully it's good entropy, if not no loss. And then it replies over UDP with the same number of bytes that it was sent. So if you send this server, there's 64 bytes on UDP 26373, which if you dial A and E-R-D on your phone, it's 26373, that's the code, the joke there. However many bytes you throw at it, it comes back with that same number of bytes. By the way, who knows what order they're gonna be in. It's UDP, right? On the client side, a client can initiate that using a real simple shell script. It uses SO-CAT or SO-CAT to send entropy to a pool of servers. It can actually use multicast, broadcast or multicast. It supports IPv4 and 6 already, which is kind of cool. Had a high school intern do that for me this summer. Who really likes IPv6, that's fun. Yeah, question? Yeah, so I was actually, I have a slide on the Intel digital random number generator that's coming out as well. So I'll speak to that directly, then we'll go back. This is a great way of the ongoing entropy. So this presentation I divided into two parts. One is the getting a better random seed to start with, and that helps your pseudo random number generator. But over time, in refreshing that entropy pool, the Intel DRNG, it's incredibly fast. Super, super fast. In fact, one of my former colleagues, Colin King, tested this on an Ubuntu system, and there's the specs that are put out with it, but he actually did this on a real system on his laptop. He was getting 40 to 45 nanoseconds per call to it, and each call gives back a 64-bit random number, which means that he could do 22 to 25 million 64-bit random numbers per second. That's incredible performance out of a physical random number generator. Yeah, and it's great, and I can't wait till I have one of those, but I bought a brand new laptop earlier this year, and I missed it. I got an i7 that doesn't have this. You have to wait for Ivy Bridge, and in the case of Cloud, I have to wait for my providers to buy that hardware and have it. Exactly. Yeah, very good. Yep, thank you. Yes, sir. Wow, you guys are way ahead of me. Look at this. Havage. So Havage is another great way of getting that ongoing entropy. In fact, I run Havage everywhere. 100%, it's in my default seeded. My cloud and NIT scripts add Havage all the time. I've been pushing for Havage to be in the default of going to install maybe next time. It's extremely fast. It's not three gigabits a second fast, but it's hundreds of kilobits a second, which is a lot of entropy. It keeps my entropy pool at 496 bits full just about all the time. The more loaded a system is, the more entropy it generates, which is kind of cool. The more CPU instructions that happen, the more entropy it's able to gather from that. It works actually really similarly to the DRNG from Intel. It's measuring the states of the virtual core, of the registers on the virtual core and deriving entropy from that. There's a paper about that one as well. Excellent paper if you're an entropy geek like me. It works really well in virtual environments and I highly recommend Havage just about anywhere. So that's another great suggestion. Yes, very random, even a virtual machine. I absolutely agree with you. So just to close the loop on a nerd, it does support IPv4, IPv6. You can talk to a pool of servers. If you have a favorite server, you can go out and talk to. I'm running a public one if you wanna test it out. I'm focusing on Ubuntu 1304. We should see it in 1304 pretty soon. The server is a separate package in the client. You can have clients that just do their initial seed, grab a seed at boot. Maybe they salt it or mix it or encrypt it with some local private information so that even the server knows what it gave you, but if you take it and then hash that with some bit of, you fuzz it some way, you can keep the server from knowing what seed you ended up with. Yeah, right. Yeah, I don't think this is ever used by itself or in isolation, but the way you salt it and hash it on your local side prevents that attack from being very practical because maybe you know what you're giving me. You're giving me just zeros. Let's just say you're just giving me all zeros. Here's your entropy. Take that, right? But you would have to do that for every one of the servers that I'm talking to if I've got a pool. And if any one of those is not malicious, yeah, exactly. The idea here is that you come up, you're a VM, you boot up. You're brand new. You're in this environment. You wake up for the first time. There's a bunch of other VMs around you and you're booting just like every other one started from Genesis. And you've taken all the other things I've told you to take in to compile into your random seed. But what you really would like to do, you're not very old. What you really want is to leverage some of the entropy from your peers. Perhaps they're collaborative. Maybe some are malicious, but some are collaborating with you. If you can inherit some of their randomness and exchange that with them, you do this UDP blast where just the ordering of those packets throws some entropy into the mix. That's a good idea, thanks. So I don't want to spend too much time on that. I do want to talk about improving the ongoing entropy. So the seed is important. I think we've got a couple of ways we can improve that. When it comes to ongoing entropy, the Intel DRNG, that's great. I can't wait to have three gigabits of randomness per second. As soon as those chips are available, we'll be very, very happy. I remember saying the same thing about KVM compatible CPUs years before those were ubiquitous too. And now we're finally, you can get KVM everywhere and that's awesome. So one day it'll come. But in the meantime, I had to run a lot of Zen. I had to run a lot of QMU and VMWare until I got to the point where I could run exclusively KVM. And that's kind of where we're at right now. When it comes to KVM, the VertIO RNG package or a module driver is also pretty awesome. This is a way for the host to provide guest access to a hardware device, random number device. So if your host has a hardware device, I've got these little USB entropy keys they're about 50 bucks a piece. If your host has that, that device can export its entropy using the forgot the protocol R and D. I've got them anyway, there's a randomness protocol. It's exported from the host to the guest through the VertIO RNG module. I think Rusty Russell among others worked on this driver over the last couple of years. It's finally in KVM. The next version, Chris, or did it release already? It's already there. And this is great. We should add RNGD, that's the daemon and there's a protocol that's implemented by it. As soon as OpenStack can export this driver to KVM guests, that'd be great. Now it does require hardware to exist on the host. So that part is, you've got to have a hardware random number generator. But once you've got that, this is a great way of doing it as well. Havage, as I mentioned, is absolutely the easiest way to do that. Pure software solution. All the tests that it's been put through, you can run it through the NIST test, you can run it through ENT. The quality of the random numbers looks outstanding and it's been more than adequate for our use cases. Has that been your experience? Yeah, so I've yet to see a compelling reason why other than its software, it clearly can't be good as hardware. Other than that, just kind of, truism that we always fall back on from a security perspective. I've yet to see any reason why Havage shouldn't be run pretty much across the board in virtual machines. My company has a key management service. As part of that key management service, there is a trusted channel and encrypted channel and we can pass entropy to and from guests. So to this lady's point, when you have a trusted relationship with another server and you have an encrypted channel and you don't mind that server knowing your entropy because you're gonna hash it, but it's still a friendly server, this is a good way of doing it over TCP, over SSL and not using the broadcast, the broadcast UDP service. So this is just, it's not anything we really sell, it just kind of comes with our key server. Right, this is 443, yeah, this is 443 traffic. Yeah, that's a very good point. So, in conclusion, there's two things that we really need to do and we can do around OpenStack. The first one is improve the seeding of the pseudo random number generators. There's a couple of ways we should or could do that. The host could contribute that through the block device or the metadata service or both. I think the block device would be interesting for being able to do it before the networking comes up. The metadata service should be easy. I don't see any reason why we wouldn't want to just do that. These are some of the things that I would, you know, when I got really interested in OpenStack, what I really wanted was not something to be just an equivalent to Amazon. I wanted to be part of an infrastructure that could go way beyond what EC2 can do and these are some of the things where I think, you know, a little focus on security here could really differentiate OpenStack in a super positive way. The user should contribute some random seed through the cloud and net user data. So in the user data, just like I showed, throw a random seed into the mix that comes from your system, from your laptop, that's contributed to the pool as well. Presumably that's not something an attacker is gonna have easy access to. The OS could contribute through this A-Nerd service. You combine all those together and you make it a lot more difficult for an attacker to guess the initial state of your pseudo random number generator. In terms of improving the collection over the life of the instance, as soon as we have this digital random number generator hardware, that would be my choice of hardware to run my cloud on. Would absolutely, I would absolutely use Ver.io RNG if I've got hardware random number generators on my hosts and I can pass that through to guests. I certainly would do that. I just run Havage always everywhere. I really would like to see that in the instances by default or consider a dedicated entropy service. A random.net has something similar. There's a handful of ways you can pull random numbers over the internet, so. That's that, I kind of introduced the Gazang. We're based in Austin, Texas. We do encryption key management and monitoring diagnostics and we've worked with a number of vendors helping our number of users of cloud and big data applications. We've helped them secure their data and manage their keys. That's it, thank you very much. I appreciate it. Yeah, absolutely. If you, if you, they're gonna go on my blog later today as soon as I'm done or Twitter. I'll give you a card if I may email them to you. But yeah, they'll be up later today. Yeah, you bet, thank you. Any other questions? Yes, sir? Yeah, the conference chair emailed me earlier or emailed us earlier and said, put your slides in this Dropbox location and I gotta go get a Dropbox account because I don't have one. But I'll put, I'm gonna have PDFs and the Google Doc available on my blog as soon as I've finished this presentation. I've got, they're both teed up. The blog posts are ready to go. I just, if you put your slides up before the presentation, no one comes, right? You bet, thank you. Thank you.