 Okay So we're gonna be talking about securing open stacks underside. I'm Eric Windish. I am a principal engineer at cloud scaling and Lead of open stack development I've been developing since cactus doing platform as a service since 2002 infrastructure service since 2006 and Was engineering lead for KT second generation block storage, which doesn't actually really apply to this talk very much So good news is We're the last session of the day, and I only have 40 slides So we're gonna probably finish up a little early And then you can all go home or drink some beers So we want to accept that we will fail to secure our applications And we have to secure the on the architecture underneath There's a lot of things that we can do to improve the security of open stack itself as As best we can we have to assume that we will fail to do that. There's a certain amount of risk in in that Yeah, I already said that Yeah, even faster escalation We accept that someone will succeed in gaining more rights than they should in our applications They must not be able to break the infrastructure We have obvious vectors users are going to be able to get in some ways API Actually, not necessarily any strict way they get into the API. Just that it's very public-facing. So it's pretty obvious vector RPC is pretty bad database is pretty bad we have potential for rain escalation and Rootwrap is actually a pretty nasty vector as well because it does to do we do a bunch of stuff as root so this all Messaging is it's just everything communicates to everything and There's no restrictions on that So we can and we execute code based on input received from the network From the database and the messaging bus. This is unsigned and clear text. There's no roles or authorization done on any of this You can't just secure that with TLS or SSL because That only gives you security to rabbit and it's not It's still untrusted over the network and well, I guess in rabbit and Q itself and potentially you have man-in-the-mill attacks as well Database some of our database is actually done over messaging Which we just evaluated is not yet secure and the database uses passwords. There's no PKI used on my sequel and Postgres both support Certificates PKI we don't use it. There's no capability in OpenStack to use that They could showed and will hopefully use PKI eventually There's the intentional limited escalation that we do in root wrap. So we need to do things with escalated privileges That's done as root. It's not done with system capabilities The input usually originates from the database or the message queue, which again are not secure We do use sudo to filter the input no granular stuff and We can break the hypervisor. We can have ring escalation You get jerks like this that go in the mortar and climb up the mountain and throw the ring in and Break stuff really badly for us Make your network secure and assume it isn't you have out of band attacks People coming in over your VPN and it's not terminated at the right place Or maybe they there's no firewall or they get through it. Whatever the case might be I'm using DNS as a risk of man the middle attacks and all sorts of nasty things. Don't use DNS or use DNS sec It's actually not that hard to not use DNS. It's kind of hard to use DNS sec. You should use it There's good reasons to use DNS, but if you're going to use it do it, right? SDN can be compromised and somebody can reconfigure your network. That's kind of a scary thing. I Know people will want to try and prevent that from happening So the whole point here is that we want to accept our failure to do all of these things We'd like to do all those things. We may not be able to Some things that we can't fix or presume that we will not Be able to fix at all or our supply chain. Maybe the government can do that because We can't necessarily trust that somebody is not going to do something at your vendor Someone in your transport customs your Personal services FedEx ship shipping and receiving Anyone any of these vendors people who could design your CPUs your systems integrators motherboard designers by us new EFI Developers rather designers manufacturers developers switch designers southbridge designers nick designers manufacturers You have to trust all of these people if you use text if you use secure boot you use TPMs All of these things can circumvent that And unfortunately, there's nothing we can do about it Be nice as we could I mean there are some maybe some extreme things you can do to prevent it But at the moment it'd be great if we can fix these problems, but we can't so we have to accept that We're going to fail here From our compromise is something we can actually kind of do a little bit more about but it's still pretty bad Ethan that controllers in particular are kind of vulnerable in January. There was an Intel EEPROM failure where you could actually kill network controller from where or kill the network Controller over the network by sending it a magic packet and it practically affected most of the Intel controllers out there today and that was really really bad and That's only one step away from an overflow that would have actually allowed you to overwrite the firmware over the network and Then all your text and all of your secure boot stuff in TPM is useless So what we can do to our best of our ability is to secure the underside And to do that we need to secure our boot process and our installation So this slide might be in the wrong place, but we want to probably talk about who wants to trust a little bit We want users to trust their provider and their provider systems Providers do not trust their users. You don't if you're a provider of your whomever is public provider HP rack space Data for any of these public providers that are coming out of Bluehost is a new one You don't want to trust your users They're just people that give you a credit card Users do want to provide trust their provider or unfortunately have to or presume that they have to But the trust doesn't go both ways So we need to convey trust Because the systems want to the providers want to trust their systems You need to use at a station Measure the state of your system verified by remote systems You want to for the provider to the user you need to have transparency Effective communication third-party auditing cloud all that kind of fits in there potentially and Effective communication users may actually also include at a station unfortunately To do any of that for the user we need as a provider as someone building a cloud and deploying a cloud We need to actually trust those systems that we've built and we've designed and we're deploying So we can do that through text and we can do this through you if I secure boot In unless you're using arm and then it's whatever to use an arm or AMD for that matter So to take measurements we can do it two ways we can measure then execute or we can execute then measure and Both are going to be useful, but one we We boot up your you if you if I measures your bootloader you execute the bootloader your bootloader measures your kernel and Then your bootloader executes a kernel or the other way around where you execute in the measure So after you've done all that and you've executed this code you can actually verify After the fact that you've executed good code using txt Which perform txt you do an SNR command that performs a soft CPU reset And it takes a measurement and you can test that with a remote service And it's important to note that neither of this actually verifies your data Verifies and measures your executables So you have to actually trust that your executables themselves that you trust or Verifying the data that they're using this is sort of an exit this actually from Intel and It kind of shows Measure and then execute and then you'd use txt on the other side and Could be a little better, but you would then verify and report that and attest that So on one hand you ensure you don't execute the bad code and on the other you verify you didn't execute the bad code and Both are desirable You don't want to There's something that will say you can just do the verification You can just use a dynamic route of trust and verify you haven't Executed bad code and there is definitely value in doing that But I'm of the opinion that one should never really execute code that they don't trust in the first place It's great to verify it after the fact, but you really as best as possible never executed So trusting your systems means trusting the software it runs Assuming that we trust all the firmware, so we're gonna do this through a secure unintended installation You can't whatever software that you're running Has to be installed. You can't trust that software unless you can trust the installation of that software And that starts all the way back at your bootloader So this is how you would do a pixie boot typically Your system boots up it loads the bios of you EFI you boot from pixie Pixie gets an IP address off of boot P and a TFTP server to download an image You download all the stuff over the network which for the sake of argument will presume is untrusted and Has been potentially compromised You download the bootloader you execute the bootloader the bootloader downloads the kernel from TFTP executes it kernel downloads the file system in an RD file that contains your installation media or medium and Loads it and executes code. That's in that file system. We depend Too greatly on network security If your network is not secure if somebody has compromised the hypervisor and Has compromised your network infrastructure You cannot securely install and Insitize new systems in your cloud and the systems that you insetize cannot be trusted to be secure so this is an example of something we'd like to try and do which is to do a secure installation and Your BIOS UFI UEFI has a pixie rom or it's I really a EFI Piece of code that runs in UEFI. It's signed and trusted Actually probably part of your UEFI installation I'm ready with and get the system. So Kind of good there That's going to download get information from boot P which is untrusted Which is okay because when we download that bootloader from TFTP it's signed and That signature is verified against a key in the TPM a public key in the TPM and Only after we have verified the signature do we execute that bootloader and Then that bootloader Will download the kernel and download the file system You're in a RD Verify the signatures on those and then execute them and by doing this You ensure that No one who has access to your TFTP server or your boot P boot P server or anywhere else in your network can actually Bring up systems running your software in your cloud or bring up any other system running in your cloud as long as your UEFI on each system is configured with your keys and no one else's keys And it's configured for secure boot Which is important because then you do just let you know you do need to have vendor support In general if you're doing this at scale you'll have to have your vendors remove Microsoft's keys first of all and install your public key into the SKS in the in the hardware and By doing that you can make sure that only your code can actually be booted on those physical machines that are in your Data center It should also be noted here that the kernel is that it's an interesting deficiency in Linux The kernel cannot actually verify the NRD itself, which is why some of the Ability to for instance bootload directly into the Linux kernel from UEFI Can't be used here and you need to use a bootloader that can actually verify the file system for you because Linux cannot There's just no feature Maybe Matthew Garak can help with that Just kidding Pixi network install So your hardware supports it secure boot can verify the bootloaders But doesn't verify so we kind of went over that already But right so it doesn't Doesn't verify the data so you need to have a bootloader you have to download a bootloader that actually can verify that NRD and only one out there today that seems to fulfill these promises is Ipixie Used to be formally known as Gpixie or etherboot. So you may have heard of any of these projects are all the same thing Yay renaming Another thing is it doesn't do PCR injection for TXT Which is kind of unfortunate So you do have to then just trust Ipixie did the right thing and the signature for Ipixie It you still do maintain your chain of trust, but you do have a little bit extra trust in Ipixie You might not have if it could do PCR injection. I believe that someone is working on adding that feature So I should also add that if you're using chef or puppet whatever you're downloading and installing from chef or puppet should also be signed or downloaded through SSL and packages should also be fixed so Sign your have you sign packages if you're using Ubuntu and red hat and so forth It's generally okay if you're building your own packages Make sure that you sign those packages and you're actually having those verified and you're not circumventing that stuff and you're not just use You can use self-sign key keys as long as you actually put your Certificate on those servers and in your installation process and don't just help to bypass the the signature checks and Do not ever trust Ipixie. It is bad. There's no verification of the signatures of Ipixie You can upload signatures for your packages But it doesn't verify it when you install with PIP. So Most people probably I doubt many people are actually doing installations of open stack using PIP and But if you are Someone can do an Amanda middle attack on you pretty easily. It's kind of ugly so It's actually a couple slides here on this and these are just some tools if you download the slide deck you can use these links I have some information on doing this stuff in QEMU and downloading Ipixie if you want to do this without actually having a whole bunch of specialized hardware Yeah, we're hiring so come work at cloud scaling So as I promised we'd finished early. I was just about right where I 18 minutes I thought it'd be about 20 and I'll open the floor for questions comments, please Walk to the microphone if you can questions none Hey, we get to go home Very very hard You This is one of those those security topics where it's really easy to tell people what to do It's much harder to do it The biggest problem here unfortunately is going to be to work with your vendors and get your keys into the systems and Get Microsoft keys removed But once you do that, yes, there's gonna be engineering work to get everything signed first of all and to do it also another important thing is a lot of this stuff is pretty new and A lot of the work only in the last one or two years by people like Matthew Garrett Now at Nebula from leader red hat Really enable a lot of these things to even be possible so Hopefully in the next one or two years. We'll see a lot more adoption of these techniques There's a lot of barriers Beyond that, you know the TPMs are generally not provisioned with any any certificates or keys in them and yes, and Provisioning TPMs on a large scale is Incredibly difficult because of this. It's a very manual process to enable them Microsoft does it No, they don't they don't come with their TPMs are not turned on by default because It's a manual and they don't come with The certificates everything burned in the TPM from the factory because that's something that the the box maker has to the board maker has to And it's there were privacy reasons I believe if you're running Windows 8 those systems have to they may be run in permissive mode many of them But yeah, they have to have secure boot On those systems boot may be on there, but I don't know how how extensively they use in the TPM As part of the verified process other than as probably just a key store Well, the SKS is seated with Microsoft's key So now granted not everybody's Microsoft not everybody can do that But if you have a good vendor relationship with somebody like Wanta, right you can get this but not every device has a TPM They they're doing it without so on some devices without a TPM I'm pretty sure Don't think that's possible. Now some of the TPMs now are in processor. I believe But I agree that the hardware part is the hardest piece of this the getting systems and The keys in your in your hardware. That's the hardest part. Absolutely remote attestation is also a big challenge It is verify, but then to actually have a you know because that requires a PKI infrastructure and So I didn't want to get too much into the application layer Beyond this and there is things like oats that are now integrate with open stack and one can look at that It's it's it's conceptually. It's a really good idea The challenge Sure, thank you Brian Brian Payne from Nebula. I just wanted to say that we have been working Quite hard on doing a lot of what you just described As mentioned, this is very non-trivial to do deployment is challenging Implementation is challenging as we go down the path It becomes increasingly clear that no one has really done it before Look like they should fit together and they just don't yes, and so we're working very hard to fix them But since we're talking about today, and this is an open source conference I just wanted to put out there that if there's others that are working in the space that are interested in the space happy to collaborate happy to Discuss, you know how we can as a community make this kind of thing happen more frequently. Thank you There's no more questions than I think we can break Awesome. Thank you