 Hi, my name is Denny, I'm speaking with Together with Sage today about storage security in a critical enterprise OpenStack environment. I'm working for Deutsche Telecom, mainly as a Ceph developer and also on contributing to OpenStack and, yeah. So let's give a short overview. We will talk about what we do at Deutsche Telecom to have a secure NFV cloud, what are the requirements and after that about attack services we have identified and in the other parts about counter measures proactive and reactive and at the end the conclusions. So what we try at Deutsche Telecom is building an NFV cloud for our telco core and we have for that a data center design and have two types of data centers. One is a BDC, it's a backend data center, there are only a few in this example in Germany for example and these are more classic data centers. We have high SLAs and for the infrastructure and for the services so including fire departments and all that, the usual stuff and on the other hand we serve services that are very critical containing also private and customer data and therefore need this high SLAs requirements and on the other hand we have the front end data centers. We have in this case many front end data centers but they are very small, they are near to the customer to provide the services as near as possible to be fast and they have very low or let's say lower SLAs and what we do is what we expect they can fail at any time so the services itself is running in these data centers are responsible for providing the service and switch over. The services are spread about many data centers and have to handle the errors in the infrastructure. As you can imagine on a telco core we have very high security requirements, therefore we build our cloud with multiple placement zones in this case as security zones and we have there for example the classic stuff like the DMZ or the MZ and security zones and management zones. If you want to have a deeper look into that there is a use case from the telco working group describing what we are doing so due to the placement zones we need separations for compute network and storage obviously. Each of these zones can use storage but may don't use it so it's open but we expect for this setup to have storage in each of them. The setup needs to protect us against various attack vectors and the product or the rollout is enforced and reviewed by the security department so we have to convince them that the setup is secure. Where do we have a connection between OpenStack and Ceph? On one hand we have the object stores S3 or OpenStack Swift interfaces that setup may make use of Keystone for user authentication. On the other hand we have Cinder with LibRBD clearly and Glance as in which store which may uses RBD or may uses an object store in the back end. Clearly we have Nova to run the services and the hypervisor and on the other hand we have Manila there it's a little bit unclear which component of Ceph will be used between Manila and the Ceph of S. We will find the solution and discussing that. What components of Ceph do we have? On the red block we have Rados that's responsible for all the handling of the cluster, enabling the racks and responsible for distributing the data and backfilling if something is failing. On top of that we have three main components in Ceph and that's the Rados gateway that's providing a S3 or Swift interface to object store and we have RBD a block device to provide block for VMs and on the other hand we have Rados that's providing a distributed file system. The main problem is how to do the separation between the security zones in this case especially for storage. On one hand you can simply build physical separation and build for each security zone as own Ceph cluster that would mean in our setup we have at least 20 front end data centers and that would mean a high number of clusters we have to maintain and that's obviously need a lot of hardware and a lot of maintenance effort and would be taking us a lot of flexibility that's provided by Ceph. The alternative would be to use Rados pools which is more flexible and would allow us to use our hardware especially since we have only the front end data for example are very small so it's much more efficient to use and the question for our security department in this case is clearly will provide us a pool separation the same security as the physical separation and that's all the problem we have currently and the idea is to have for each security zone an own Rados pool and limited access to these pools depending on the security and key management available in Ceph itself. On the open stack side we would use availability zones to build these placement zones and obviously we have three main components you have to handle and this one is CINDA. In CINDA we plan to use our own back end for each pool with an own authentication key to provide the zone and the problem here is that in general we missing an open stack policy a way to force a hard mapping between the availability zone and the storage back end in this case. Currently it could be done by accident that somebody is matching the wrong volume to CINDA availability zone and the other problem or component is in this case glance and there we have the problem that there is a lack of separation between the control and the compute storage layer so if you have multiple back ends and only one glance and stunts then you have obviously a component or a VM that is running and have contact to the complete or storage cluster and that is something we don't want. Manila we currently don't use or plan to use in production because simply Ceph as itself is not a really production ready but maybe in the future we could alternatively use RBD with NFS to provide child storage. Yeah what which attack services we have and there is a case of Rados gateway the tenant VM has only access to the gateway it has no access to the storage components like the OSDs or the monitors from Ceph. There is no direct access of the customer in the VM to this part of Ceph but due to the fact that it's all HTTP or HTTPS it's a single API you can attack that's still a problem. For RBD that's yeah let's say more secure at the moment because you have the hypervisor in between and there is no network access from the client don't care about from where the block device is coming and don't see the network in this case but the issue here is as seen more than once the hypervisor is not it's still software and software is not 100% secure so somebody could break out and would have on the host direct access to everything. From the host perspective as soon as KVM or whatever hypervisor you use compromised attacker has access to the neighbor VMs on the same machine and it has access to the local authentication keys for Ceph and therefore it has also access to the public network and the OSDs real demons that run these public networks so that would be an attack vector in this case and you might think you could handle that with a firewall or deep package inspection but from our perspective that's partly impractical because of the use protocol so the most traffic going through Ceph or the network at the end is binary blocks and you can't filter that easily so and it would also have an implication to performance and costs in this case and since NFV is a lot about performance on the network side that would cost out too much so the issue at the end is that the Ceph itself need to be resistant against attacks and the problem is that C or C++ is a lot harder to secure than Python in this case and the problem is also if you have only one security flaw in one of the demons you have it in a complete cluster so you could always attack the complete cluster in this case. From the network side the problem is may that the traffic between the cluster and the client is not encrypted so interior Snifr could read all traffic that's going over that and could maybe change at least read all data maybe recover and write something. The session itself between the client and cluster are authenticated so the attacker can't run in the middle attacks or impersonate a client or server in this case that would be considered safe and we have for sure the number of servers and there you could submit as an attacker large or many small IO and take the cluster out this way so you may have to use QEMU IO throttling to mitigate that. On the other hand you could also open a lot of connections that bring down the cluster or block something and yeah for sure you could crash the cluster if you find the flaw. So maybe there are also other expensive features in self-ordered interface that could be used to run a denial of service attack. So what can we do against that? In general for the deployment the usual stuff in general applies meaning have strong keys and make sure that nobody is getting into your machine in general. I don't want to go too deep in that because that is something you always have for each of the machine also for compute. On the network side it's crucial to run all ways separated cluster and public network. Don't give any compute node or any component that don't need access to the cluster network any possibility to connect and you always have to make sure that you separate your control nodes from other networks because the control nodes have may keys and that are not supposed to be used outside and opens new attack vectors and yeah for sure you don't expose these networks to the internet to the open internet and if you have to run in the data center traffic then you have to encrypt it and very important for us is we don't run any hyperconverge infrastructure is very obvious that if you have multiple security zones you don't want to do that because you run in problems if you have for example to migrate or shut down and compute node and bring parts of the cluster down and you have always risk to mix something up and also don't mix the control nodes between self and open stack. For the radar skate way we have a way to for the radar skate way the problem is that you have an easy target I mean that's HTTP and it's an arrest interface there are ways to attack that. You have a web server and you also can attack that and what we do is usually you have direct access from the tenant VM to the radar skate way through an IP that is not part of the public or the cluster network. What we do is we run a small proxy appliance between the radar skate way and the tenant VM in this case to protect the radar skate way and the appliance has a separated network and it's SSL terminated and within the proxy we run a web application firewall in this case more security to make sure that for example the user can only access or send an API call or arrest call only to for example his user and his packets and also filter other stuff out or you have usually there and this is in service from the infrastructure in this case and it will be run in a secured or managed zone that is not part of the other zones that the tenant is running so and it's very crucial that you don't share buckets and users between tenants. You can share it with the tenant but not to the outside. So I hand over to Sage. Hello. As Denny described there are only so many things you can do before you actually have to rely on the set itself to enforce the security that it's reporting to provide. So the mechanism that we use in CEPF to do this is called CEPFX. It's actually a plugable infrastructure but CEPFX is the one implementation that we actually use and the overall design is modeled after Kerberos that's sort of integrated tightly with CEPF and designed to be highly available distributed so forth. So the key ideas are that the monitors are the trusted key servers. This is similar to your Kerberos KDC that you have in a traditional environment. So they have copies of all the keys for the demons and also the clients that are accessing the cluster and each key associated with it has a capability that describes what that key is authorized to do so which pools it can read and write from and what operations it can do on those objects. So overall what CEPF provides is mutual authentication of client and server so the client knows that the server also has a copy of its secret key and the server knows that the client has a copy of the key that it has stored so you can avoid man in the middle in that case. It also gives you an extensible framework with these capabilities so you can describe in plain text different types of services and different ways to restrict access, at least within the overall framework and you can prevent these man in the middle attacks. What you don't get from CEPF is secrecy so the authentication protocol is smart so that a sniffer can't recover any of the keys used but it sends the actual data to get sent in plain text so you can see the reads and writes that are going by. So sort of the takeaways from this are that obviously the monitors have to be secured since they store all of the cluster keys so you want to put those on separate hosts and make sure they're hardened security wise and so forth. You also have to be really careful in managing the CEPF keys, how they're distributed and so forth. So in this case we're going to have a separate sender key for each of the different security zones and each of those sender back ends and so the VM has that key you can only access that particular security zone obviously and you restrict the capabilities that key can actually do. But sort of on the other side also you have to limit the administrator's power too so if you have random employees who are responsible for administering your cloud you can grant them access to administer the CEPF cluster but for example not be able to actually administer the authentication keys to grant other keys access and so forth and you can revoke access that way. So these are existing features that you can make use of and obviously you want to be just very careful with how you distribute those keys so if you're using you know puppet modules that are for something that's distributing those keys to the actual nodes where KVM is running and authenticating and so forth you want to make sure that only the keys that are around but obviously there are some things that we also want to do as well. So CEPFX was modeled after Kerberos in part because we should sort of piggyback on the authentication and crypto that they're doing and so we can sort of use a known good model but code has only been reviewed by CEPF developers so one of the key things we're doing now is bringing in external security exports to do additional review of that code to make sure that it's sort of doing what we think it's doing. On the open side you need to do the same thing to make sure that the deployment modules are distributing the keys properly and actually not copying them into bad places and sort of across the board we want to improve the quality of the documentation so that people who are actually deploying these clouds can actually see clear descriptions of what the best practices are and how security capability should be defined and so forth because it's not obvious but even assuming all that works you know CEPFX software can have flaws so how do we how do we mitigate that so one of the first things we want to do is sort of reduce to as much as possible the flaws that we have in CEPF itself so one of the key tools we use to do that is through the use of static code analysis so this is software that reads your code and understands it and tries to find bugs one of the main tools we use for that is Coverty which is sort of one of the one of the best in the industry they have this great project that they posted for a couple years now that scans open source projects and it's all posted online it's all for free so CEPF has been doing this for two years now there are many of those by Danny here who's worked sort of tirelessly to do this white listed a bunch of sort of non-issues there's some outstanding issues that we still need to address and there are some other tools also that are open source there's CPP check and Clang and LLVM have tools as well that can do this there's also runtime analysis so we use tools like Valgrind and our automated QA that looks for use after free and buffer overruns and all that good stuff to make sure we're not doing anything stupid that's done wonders for first ability across the board but there's more that we can do so one thing that we need to do is reduce the backlog of low priority issues in Coverty so that we can separate out all the noise there's a bunch of issues in all the test code that we don't care about but it makes it hard to see the real issues so cleaning all of that up we want to have automatic reporting of SCA regressions when people submit a pull request currently we do scans every couple of days and so you catch it pretty quickly after it's merged we want to catch it before it actually makes it into the tree and we're near release and as part of that process just overall improve the awareness among the step developer community of the impact of their code changes to security so that we can sort of prevent these design issues and code flaws before they happen this sort of human element of just being aware and being careful is critical so sort of the next step is actually going through and having a sort of a trained security professional go through and do penetration testing and stuff and actually you do review the code and you look at how the algorithm works and then you actually go and try to subvert that security by injecting bad information and trying to test all the corner cases in the security code and so forth that's a very high cost approach and sort of challenging so you have to have somebody who's trained to do it and you have to have guidance from somebody who actually knows the step code and to make sure they're actually spending their time well but it's a necessary part of the puzzle buzz testing sort of goes along with that where you actually use a computer to generate random inputs and just spray it in interfaces and make sure see if you can make it crash or misbehave and overflow buffers and so forth but there are other things that we can do that are sort of much easier and that we're working on right now one of those is just simply hardening the build there are all these features in the kernel and the build tools that make it much harder for people to exploit sort of traditional buffer overflow type vulnerability so fpine, fpic randomized location of code when it's loaded into memory so you can't sort of predictably know where the code the pointers are going to be trying to subvert a program fortify source validates format type information into some other sort of compile time checks stack protector strong compiler feature that implements all this stuff so that if you try to blow out a stack frame which is sort of a common method of attack it'll detect that and cause a segfault instead of letting you sort of do bad things rel arrow is the indirection table when you load dynamic code it marks that read only at a very early stage so that you can't dynamically rewrite code later in time and of course when you're doing all this you want to make sure that implementing all these features doesn't make everything go slower because that can be problematic in a cluster but in generally there's one of the nice things is that in enterprise distributions a lot of these features are sort of enabled by default in the build tools so some people running except today already have all this stuff enabled we're working on getting it enabled for everybody else to make sure that it's all sort of well handled and so forth but even so you know you do everything you can to try to and some of them are going to happen anyway so how do you mitigate the breach if that actually does happen so one of the key things that we're doing now is making it so that all the set demons don't run as root they run as sort of just a set user that has much limited permissions so they can't actually root the box this work is pending for infernal such as the next community release coming up in a couple months another big tool here is the use of mandatory access control so tools like SE Linux and app armor that further restrict the things that a process is sort of allowed to do and known to do in a normal course of action so if you stray outside that it will raise security alerts and cost failures and so forth and that's also sort of in progress right now at least on the SE Linux front I assume that there's app armor stuff going on as well there are some other things you can do as well some of the set demons that are that are lighter weight and less resource intensive you can just run in DM so even if you break out of the demon you don't you know have a fully privileged root account or whatever on the host so that might work for the monitor and the radius gateway it's less feasible for the OSD because it's much more sensitive and needs to talk directly to hardware so you probably wouldn't use it there but a key thing is that we want to make sure that all the networks that are used for sort of doing actual client IO and actually not normally operating the cluster are separate from the ones that are used to manage the cluster currently the monitor sort of shares all of it on the same interface and we want to separate that so you can very carefully control the management activity which it needs to be blocked down much more carefully encryption obviously becomes part of this picture data at rest encryption is sort of the first piece of this it's almost less an attack vector as a management thing so the SEPH disk tool allows you to layer in DM crypt on top of raw disks automatically when you're provisioning SEPH to encrypt the raw block devices and the key thing that this is actually used for is just so that when a disk fails or if you want to arm a disk that disk and safely discard it without running it through a shredder or if somebody walks into your data center and grabs a disk they won't they won't actually have encrypted data but that's sort of a that's a different threat vector the challenge right now is that the key management that we currently implement is very simple we're just putting the keys in a on a different disk in the same host and so that the key thing that we're that we need to do is use better key management. Our current plan is to use a new project called Paterra which is a new key escrow project from the security folks at Red Hat and that looks super cool it's very elegant and much better than what we sort of had in mind. It is a new dependency but it's very lightweight and small and easy to use so that's the current path forward. We might also also use the have an alternative simple key management that just uses the existing monitors to manage the keys but I think we like Paterra much better so that's probably the path forward. So ignoring on disk there's also encryption over the wire as you mentioned before Steph is careful about not letting you recover keys but the actual data reads and writes are sent over the wire. So we want to protect the data from someone who's listening on that network. The other sort of thing you have to think about is that because the administration operations are on the same network as the regular cluster operations then if an administrator is installing a new key the fact that they're sending that new key over to the monitor to get installed that actually gets sent over the networking could be sniffed so we want to be careful of that so we can extend the current stepx protocol to add these per session keys and enable that by default either in situations where clusters want to trade security for performance or just in those select situations where you have administrative traffic that's sort of critical and not performance sensitive at all. Denial service of tax obviously our problem for lots of systems Steph has no exception here so today you can limit the load from an individual client by for example using key news rottling to some level but there's sort of a limit to what you can do on that side on the server side eventually you need to think about how you can harden it against people who are trying to use too many resources so the list of things we can do there are pretty simple you know you want to limit the number of connections that a single OSD can handle at some point it'll just start refusing connections instead of bogging itself down in destroying service you could ideally we'd want to limit that on a per host IP or client IP basis so that if a single attacker couldn't do that they'd have to mount a distributed denial service attack before they could be successful to raise the bar and ideally it would be nice if we could also throttle operations on a per session basis but that's a bit more challenging so that's another key direction we want to go in finally I want to just talk a bit about stuff because it's a bit of a different situation than the block and object services and the challenge here is that there's no sort of standard virtualization layer so with a block device if you have this virtual disk idea then the hypervisor is actually giving a lot of protection that's not true with the file system so you can use proxies through NFS as one approach there's this 9p or you can just have the tenant network actually talking directly to the store services which immediately gets to the point where you have to rely on stuff security but unfortunately doing the granularity of access control is much harder there because there's no simple mapping from this complex directory hierarchy to the names of objects and the rados capabilities that would restrict you that way so there's a lot of work going on in the CEPFS community to actually improve the CEPFS ability to restrict access on a per sub-tree basis or on a per client basis and so forth but that's all work in progress so things like route squash and restricting to sub-trees and users and so forth is stuff that we're working on over the summer and we hope to have in place for the next release of the release act of that for the next sort of iteration of stable deployment versions but so those are all sort of preemptive things assuming you actually are there is a breach you know how do you deal with it so one of the key things we did this year was create a new email alias security at CEPFS.com which is where any reports about security flaws and stuff can go this goes to a restricted set of CEPFS developers and it also copies all the distribution security lists from Red Hat, Canonical and SUSE they all get copies of this so that they can start their processes and so forth as a community we prioritize security backports and fixes of course that releases as needed to do so and we can send announcements out to the public lists I can only really speak about Red Hat stuff as far as what happens on the product side but it's you know this is one of the things that distributions like Red Hat are really good at so they have a dedicated team of security professionals who deal with these sorts of issues they drive the CV process and make sure that it gets escalated and that they do all the check all the boxes and do all the tests and so forth and then they can push this out to customers quickly so that the flaws are mitigated and you can see these these sorts of things in action when you have flaws like heart bleed and so forth the distributions really shine and dealing with those types of situations you can also just implement general practices that allow you to detect and prevent these breaches in the first place so you know things like people trying to brute force your authentication where they're just sending you random keys we can implement mechanisms on the CEPFS side that will sort of notice this brute force attacks and blacklist it so you have really good logging so you can actually tell this and raise alerts in your system that you have some number of failed attempts from this IP and that you should go investigate the other thing is that you know sort of the again the key management is the keys to the kingdom that's sort of the key to keeping all the safe in a paranoid environment you can have you know compare or audit the set of keys that are installed on the system to make sure that people aren't surreptitiously installing new keys and using that to subvert the system and so with that we have a few conclusions sure so in general at a high level we find that we have sort of reactive processes are in place we have the security notification email lists and we're sort of wired into all the distributions and we've done this a few times so we know sort of what how to react to security vulnerabilities when they're when they happen the hard part is all the proactive measures you know how do we actually prevent them in the first place again key the key is code quality and we were using SCA heavily and so forth to actually try to prevent these bugs and detect them when they get introduced we're working on doing unprolaged demons it's going to be in the next step release and mandatory access control like SC Linux and app armor to make sure that you do breach that you can't do further harm and things like encryption used selectively in places where it's really critical there's been some progress defining security best practices the key thing here is that we need to improve the documentation and actually so it's clear that if somebody is trying to set up a cloud and is actually thinking about security they have some documentation that will give you a good reference and I know what to do there but I think that one of the biggest takeaways is that this is an ongoing practice a process you can't just sort of fix all the issues and say that you know stuff is now secure and you're done and move on to something else is ongoing diligence by the community and users both users and developers to make sure that you don't introduce new flaws and that you're continually staying on top of things so everybody one of the key things that you can do to help is to get involved so security and open stacks obviously a hot topic there's a telco working group that is dealing with a lot of these critical issues and all the different projects have specific security issues that they're addressing and obviously in CEF that's no exception so you can get involved on the mailing list and IRC and we have these online stuff developer summits every three months where developers congregate to online to discuss what's going to happen for the next release cycle so I think that's it thank you very much I'm happy to answer any questions yes there's a microphone right over there if you don't mind I have one question I saw earlier that you're using DMCrypt for address the encryption what were the reasons for using DMCrypt over using already encrypted hard disks from a couple of different manufacturers mostly simplicity so DMCrypt will work on any architecture as I understand it there's a semi standard interface for doing encryption at the hardware level in the firmware but the question is just how do you interact with that firmware and make sure that you properly detect the support and actually do it the tooling isn't as easy okay thank you that would definitely be a nice project for somebody who wanted to improve the state of things so the question goes to Danny but Sage I'd like to hear your opinion you mentioned earlier in the presentation about separation through Rados pools do you or would you also advocate taking that down to the OSD through the crush maps and say separation by pool by OSD to take it a step further or is that unnecessary because it takes a lot of flexibility hard mapping that is something that is near to having separated hardware cluster physical segregation on the level of a complete cluster that is nearly the same at the end and it wouldn't bring that much benefit and takes only flexibility yep I would agree with that I mean even if you're separating it across OSDs if they're on the same hosts and you compromise the less secure OSD then you're on the same host and you can get out all the stuff that the other OSD is doing so at the time that you're separating across hardware you might as well be running separate clusters part of the reason I asked is because it's been advocated to separate OSDs across pools that that work on different workloads so I didn't know if you'd advocate that in security zones as well it makes a lot of sense for performance reasons doing different types of hardware, different speeds of disk for security the benefits are limited I would say I've got one question regarding network encryption have you given any thought to just using you know given that SAP is fully IPv6 enabled just using the built-in encryption of the network layer or why are you pursuing building your own that would be that would be one option honestly this isn't something that a lot of design engineering attention has been spent on just yet so I think that the issue with using like IPv6 IPsec would be that it's oh yeah I think we just need sort of a a network person to tell us what the best thing would be there alright thank you very much