 Good evening, Cloud Native Security Conference. It's an honor to be joining you for this first instance of this new event. I'm joining you from Boston. This is Red Hat's famed Westford office. Also, you can't see much of it and this is just a small conference room that we turned into a virtual event recording booth. But with that and without further ado, let's get going. Now, my marketing manager wouldn't let me get away from the concept that I need to introduce myself. So here it is very briefly. I had the privilege of spending my entire career in free and open source software. I was the Product Management Director for Ceph Thright Hat for the last seven years. I previously was the Ubuntu server PM at Canonical. If you remember 1404, that was my favorite release there. If you go back the decade, I was the Systems Management Tsar at SUSEA. The Dreaded Systems Management Tsar, actually. Some things I worked on on the slide, a bunch of different products, and I was the maintainer of MAN for about 10 years. Technically, I still am, I suppose, but I'm letting Colin take the lead with this version. It seems like the right thing to do. So one last thing, I have a book out on AWS operations by O'Reilly. That's why you see me surrounded by clouds, and that's plenty enough, I guess. Now, Rook, for those of you that are not in storage, Rook helps reduce the operational burden storage team faces, creating horizontally scaled self-healing clusters and Ceph. And Ceph is complemented by Rook making them increasingly self-managing, even self-scaling, you could say. Rook enables Ceph to deploy on Kubernetes with ease, enabling the benefits of containerization. Storage is just another workload running on top of the compute plane. Oron Bear Metal has an external entity for petabyte scale independently managed storage clusters serving different Kubernetes clusters and possibly other workloads. So as usual, we can make things abundantly complicated. The combination of the two is a quite sweet storage solution in general, and obviously perfect for Kubernetes, which is what it is designed for. Now, security practices harden a specific point on the infrastructure. Cherry picking practices without the model of the threat and the attacker carrying it out is not a viable strategy. The joke usually goes, if you want to protect from all possible threats, you need to turn the computer off, bury it in concrete, drop it at the bottom of the ocean. In other words, absolute security is not usable and probably unattainable. It needs to be defined in the context of a threat model. Are you facing script kitties or the GRU or the dreaded privileged insider and that we should all be worrying about? These are very, very different scenarios. That's the starting point for any security consideration. Now, let's dive in on, once we have the profile, what are we going to tighten and what can be tightened? Starting from the network. We can break the network in security zones and the public security zone is the easiest to understand. It's an entirely entrusted area of the cloud. It could be the internet as a whole or you could conceptualize it as just networks external to your cluster that you have no authority over. Data transmissions crossing this zone should make use of encryption. Note that the public zone, as I just defined it, does not include the storage cluster front end. What in SEF is called the public underscore network, which defines something different. The storage front end, which properly belongs in the storage access zone. The SEF clients zone refers to the network networks accessing SEF clients like the object gateway, the SEF file system or block storage. SEF clients are not always excluded from the public security zone. For instance, it is possible to expose the object gateways as three and Swift API in the public security zone. The storage access zone instead is an internal network providing SEF clients with access to the storage cluster itself. Finally, the cluster zone refers to the most internal network, providing storage nodes with connectivity for replication, heartbeat, backfill, and recovery tasks. This zone includes the SEF cluster's backend network called the cluster network in SEF. Operators often run clear text traffic in the cluster zone, relying on the physical separation or VLAN logical separation of the network from all other traffic. This would not be a valid choice if your threat model includes adversarial privileged insiders, for example, but it is a perfectly fine assumption for other models. And this is a good example of what I was just saying earlier. These four zones are separately mapped or combined, depending on the use case and the threat model in use. Now components spanning the boundary of two security zones with different trust or authentication requirements must be carefully configured. These are natural weak points in network architecture and should always be configured to meet the requirements of the highest trust level of the zones connected. In many cases, the security controls should be a primary concern due to the likelihood of attack at this point. Operators should consider exceeding zone requirements at integration points, which for a storage product is often easier to accomplish than for other types of products. For example, the cluster security zone can be isolated from other zones easily because there is no reason for it to connect to other zones. Conversely, an object gateway in the client security zone will need to access the cluster security zones monitors, port 6789, the OSDs, all of them on port 6800 to 7300, and will likely expose its S3 API to the public security zone, port 80 and 443. Let's move on to encryption. Server-side operators overwhelmingly choose to encrypt data at rest using the Linux unified key setup mechanism, better known by all of us as Lux. All data and metadata of SF storage cluster can be secured using a variety of DM crypt configurations. And almost all of Red Hat customers choose to use this facility. A security best practice is to locate monitor demons, the mons, on separate hosts from the storage demons, OSDs, ensuring anti-affinity of the keys and the data that they encrypt. There's results in physically removing, this results in protecting against the scenario where physically stealing a machine or removing the drives does not carry the keys with the part that was stolen. The object store gateway has additional capabilities, including encryption at ingestion time, the use of per user keys, as opposed to per drive keys, their rotation with tools like Vault, support for Amazon AWS, SSC, KMS, and many more things. One thing that I'm going to call out is the Department of Defense ciphers certified on their FIPS 140-2. These can be used throughout SEF as supplied by REL in appropriate versions. Network communication can be secured by turning on SEF protocol encryption in the Messenger v2.1 protocol introduced with the upstream Nautilus release and updated since. Here, clear text is preponderant because the networks where the cluster uses the SEF protocol are usually the trusted ones and they're usually physically or logically isolated from access. Comparability and overheads are the primary reason why backend protocols are not encrypted by default, but in most use cases, the performance impact is negligible for a properly designed cluster. Latency is shadowed by the network communication provided that you size the CPU core count to account for the fact that you're going to encrypt everything. So it's going to cost you a little bit in hardware, but you can do it without seeing a performance impact for most workloads. Looking at more specific protocols, the S3 service is usually secured between RGW and the S3 client with TLS on port 443. Also, plain HTTP on port 80 is obviously also an option depending on the nature of the data served confidential versus public. There is plenty of scenarios where RGW may be serving just web pages. TLS termination at an HA proxy endpoint is a special case where the link between HA proxy and RGW itself is in the clear text and needs to be located in the appropriate security zone. Standard network practices like firewalling individual nodes to only expose a clear list of ports apply, all the usual network hygiene things that you would normally do. Let's look at Rook specific things for a second. Rook can use CRDs to encode many of these settings like configuring trust certificates for RGW's web service. The Kubernetes secrets mechanism is not really secured by default. So that's another thing to account for. Base 64 is easily reverted and needs to be hardened with encryption at rest and access policies to containers using secrets. Base 64 is not encryption. Base 64 is just encoding. It can be reverted with one command. Rook supports at rest data encryption as we discussed earlier, same thing with in-flight self-protocol encryption not supported yet, but coming soon. You should in any event be using the cloud's extremely flexible software defined networking to segregate unencrypted traffic to private networks. Really on public cloud environment, you have no excuse not to be doing things that way. You can go wild with creating zones. Kubernetes user permission system applies to PVs. So permissions, quotas and all the rest, all of that comes from Kubernetes with nothing for Rook to do here, which is great. More interesting perhaps is KMS support in the CSI driver. This allows individual volumes to be encrypted with their own key, something that, for example, in block storage we would like to do, but we cannot do without Rook at this point. Not yet, we're working on it. Ceph ADM, Ceph Ansible and other deployment and day one tools use SSH to provide a secure command path for install and upgrade operations. This is common these days. It is the control channel popularized by Ansible for host management. The dashboard is usually not exposed to the world, but it needs to be reachable by the operator's workstation to be of use. Lastly, the manager demon supports the whole infrastructure and needs to be reachable on the storage access zone. Ceph use of keys protects the cluster from man in the middle attacks by default. A good practice here is to grant key ring read and write permissions only for the current user and root, with client admin user restricted to root only. RGW predictably supports the key and secret model of AWS S3 and the equivalent model for OpenStack Swift. The administrator's key and secret should be treated with the appropriate respect. Use administrative users sparingly to reduce risk profile as you would do anywhere else. RGW user's data is stored in self pools, which should be secured as we discussed previously regarding data at rest processes. LDAP, Active Directory and Keystone identity vaults are all supported. Operator actions against the cluster are logged and should be periodically reviewed as well as aggregated to your log management system if you have one and you should have one. Once data is deleted from a self cluster, it generally cannot be recovered for practical use. There are exceptions. However, RBD has a new facility called Trashbin where dynamic use of spare pool capacity can be used to retain deleted images until that spare capacity is needed or until a certain number of days is elapsed. Similarly on RGW, versioning of object stored buckets may result in deleted objects preserved as part of version history until they're deleted by a policy or by the administrator. Wherever user data retention is a concern, configure your data storage pools accordingly. Additionally, individual data blocks that use to constitute an object, file or volume, are typically still present on the persistent storage system until they are overwritten by use. You cannot overwrite a self cluster just by writing a lot of data to it. It's not going to work. So secure deletion is another question that we often see. And the easiest way to sanitize media is to do the right thing and use OSD encryption. And when you need to sanitize the media, you forget the key. And that's the most effective way to be able to discard drives safely without overwriting with a separate device or degossing or shredding or anything of that sort, which is usually not possible when you have to return them under warranty. Hardening options are highly vendor dependent. The following are some of Red Hat's choices. Other self distributions will, of course, vary. Worship with AC Linux on by default in enforcing mode. Of course, you know that AC Linux is sort of a religion at Red Hat, so that's hardly surprising. And we can make use of FIPS 140-2 certified ciphers as supplied by RHEL when configured to do so. Hardening of binaries is also of interest because of what I told you about demons sitting at the boundaries between security zones and being natural targets for attacks to cross those boundaries. I believe Red Hat's self storage binaries use SecComp, Pi, and several, if not all, of the ASLR options. But I didn't manage to catch my build master to confirm before recording this, so I'll have to defer this to the Q&A session if you are interested. In any event, these are worth exploring because it makes exploiting of overflows or that kind of binary attacks all that much harder. And on that note, that's what we have for today. I hope that I've given you some useful information. I'm very interested in hearing what your questions are and, remember, speakers are Pavlovian devices. So if you like the talk, let us know that you did. If you wanted more of something, tell us what is missing, what you'd like to hear. And I guess that's it for today. Thank you.