 Welcome everybody to the Open Infrastructure Summit 2020 talk by SecoStack on physical tenant separation, where we will evaluate concepts and proposed implementations. I am Laura Geisler, Jr. Product Manager of SecoStack. That is our agenda for today. We will speak about possible approaches of integrating physical separation concepts into OpenStack and the corresponding advantages and disadvantages of each approach. In addition, we demonstrate how an approach of not sharing OpenStack infrastructure on a compute host among tenants can still be integrated in the architecture without giving up the main idea of a cloud. But let's maybe first start with whom you are dealing with right now. Who's SecoStack? We are a startup located in Dresden and an OpenStack Foundation corporate sponsor. Our product SecoStack is a security enhancement project for OpenStack. As part of the image encryption pop-up team, we are contributing image encryption as a feature to OpenStack. Why is today's topic physical tenant separation? Let's come to the motivation behind. Virtualization is a good practice to isolate workloads from each other, which are running on the same host. To break out of a virtual machine is not that easy and requires knowledge of vulnerabilities in the hypervisor. Yet, it is not impossible. There are workloads which may prove it from being rolled out in a cloud infrastructure, such as access points for patient data in a hospital. Such workloads need a high level of security. You could say they would require an outbreak proof hypervisor. But hypervisors are complex components and vulnerabilities cannot be ruled out. And further, sometimes additional requirements or regulations make physical separation mandatory. If an outbreak happens, like you can see on the slides right now, the red project gains access to the host. In this presentation, we will treat the term project synonymous to tenant, as it is a logical group in OpenStack, which resources like VMs and volumes are assigned to, and access is restricted to resources within such group. So why is this bad that the red project gains access to the host? We do have a lot of information on a compute host, like passwords, access to other project networks, and so on. And the red project can read the memory used for the other project's VMs. Thus, red gains access to information handled in the yellow and blue instances. Let's say in the yellow project are just ongoing experiments with anonymous data, where blue instances also hold patient data. You can redo an experiment, but someone tampering with patient data may cost lives. So blue VMs in here definitely need protection, by still being able to be accessed from the blue project. The easy solution would be physical separation. We just put all blue instances on a separate host, or more if there are so many VMs. That would make it impossible for someone outside of the blue project to gain access via attacking the hypervisor. They are still vulnerable to some insider, but as a project owner you can add and remove access to this project and thus VMs. The downside is this is not very cloud-like and flexible. So we will have a look into some ways to achieve this. Approach number one, exclusive clusters. The first option is obvious. We just need another cloud, a private cloud just for holding the blue project and its workloads. As you can see on the left side, we have a few physical nodes in which a cloud solution like OpenStack runs. The red, green and yellow projects can start instances on this cloud, and they are not physically separated. Additionally to the compitos, they also share controller nodes and storage nodes. On the right side we build a complete infrastructure just for the blue project. That means we have physical separation between hosts where blue instances reside and hosts where yellow or red instances reside. But we also need additional controller and storage nodes to build this project's cloud. While this is an option for very large projects, it has too much overhead for smaller ones. Such an extra cloud also needs controllers, databases, networks just like the original one. This results in a need for extra hardware and software. Additionally, there will be a need for someone to operate this new cloud. This is way too much especially when the blue project in this case would only need one separate compute node. Let's have a look into achieving this without having to build an additional cloud. Approach number two, OpenStack Ironic. We can use Ironic, which is an OpenStack project that can and will deal with bare metal nodes. Bare metal nodes are those nodes which not even have an operating system on it, but will be initialized and installed via network. Now we do have one cloud with a lot of nodes. In the picture you can see the physical machines at the bottom. Two of which are currently compute hosts providing VMs to the green, red, blue and yellow projects. One of the physical machines which is not already configured as a Nova compute host is now selected and separated as a bare metal host to the blue project by Ironic. Subsequently, an image is booted on it and the system is made available to users of the blue project which are now able to use the entire physical machine. As the machine is physically separated, it does not come into contact with other projects' VMs on the other compute hosts. The blue project can choose to execute direct workload on it or configure a hypervisor for self-managed VMs on their own. This is already a cloud-like way to achieve physical separation. A project can have a single node for themselves if needed. It does not have to be the blue project. It would be also possible for yellow or green. And a project can split its workloads to reside either on the bare metal node or SVM on a shared compute host. Projects can have both kind of instances, such with strict separation necessity and those without. But there are downsides. Ironic only gives you a bare metal node with an operating system of your choice. If more than one workload, more than one former VM should be started on this node, it has to be handled by the user. That means if any kind of virtualization might be needed to separate workloads from each other, this has to be installed and managed by the user. Using one bare metal host for each workload might sound like a possible way to avoid the managing, but may result in a massive overhead if the workload is much smaller than the appointed host. Approach number three. Host aggregates and flavors. Another way to dedicate a single node to a project in OpenStack is using a combination of aggregates and flavors. We first have to create a new aggregate and then add a host to it. This compute host will be the host dedicated to our blue project later. You can see this in the first two comments on this slide. This will not be sufficient to have a compute host, for only blue, VMs will be spawned. We also need to add a metadata key value pair to indicate which tenant should only be able to spawn VMs on this host. This is done with the filter tenant ID equals project ID, where we add the project ID of the blue project. The sketch here of Nova will now check this aggregate and only allow blue VMs being spawned on the dedicated host. You can see how our cloud looks like after we executed the comments in the picture. The far-right compute node is in the host aggregate separation host, and only instances of the blue project will be spawned there. Instances of the red, yellow and green projects will only be spawned on the other compute nodes. In the picture you can see a new virtual machine that is scheduled. Yet, it is not sure where it is spawned. As we can see, it is not enough only using host aggregates. We also need a new flavor. For this flavor we have to set what's in the second comment. An extra metadata for the scheduler. When this new flavor is used, the VM will only be scheduled on a host which is an aggregate. With the filter tenant ID equals project ID property which has the same project ID. While that does sound difficult, the result can be seen in the picture. An advantage of using aggregates and flavors is that we use an open stack functionality to manage VMs. On the downside, it is a complex setup. The operator needs to set up aggregate and flavor for every project which needs isolation, so that each project needs to be known. Further, coupling isolation with flavor is not a good practice. It needs many flavors for differently sized VMs and is not a clear statement for isolation. Approach number four, flexible compute host isolation. Now having these kind of options we wanted to see whether we can implement something even more flexible and cloud like. A mixture of handling nodes was ironic and having the usage open stack like in the host aggregates case. We first added two new parameters for the compute nodes, separation host and project ID. Because we want to know whether a node is a shared compute node or whether it can be dedicated to a single project and to see whether it is already dedicated to a project, we need to know the project ID. You can see it and a CLI call to list all hypervisors, which is a list of all available compute nodes. Now that we have a separation host parameter, we need to enable operators to set host to be a separation host or not. This will only be able to be executed on empty host. So there is no way VMs of other projects will already residing on a new separation host. In contrast to the host aggregates, the project ID will not be set by the operator. It will be set at the point when a project wants to create a new isolated VM and thus allocate previously unused separation host. You can see the comment for changing a hypervisor to be a separation host in the CLI example. To be able to start a VM with isolation needs on such a separation host and other VMs never, we added a new filter to the NovaSketcher and a parameter to indicate isolation needs to the VM. The tenant separation filter checks VMs for the project isolated metadata and if it is present and true, it will only schedule this VM on a separation host. Other VMs will be scheduled on shared compute nodes. The separation host which is chosen by the scheduler is either empty or already allocated by the project creating the new VM. The comment shows such a create of a VM and the new metadata project isolated. In this example, the blue project wants to spawn two VMs. One is project isolated, the other one isn't and we do have two compute nodes. One is shared and one is an unused separation host. The VM without a shield should not be isolated. Thus, the scheduler will select a shared compute node, in this case host1 on the left and spawn the VM on this host. As you can see, instances of the yellow, red and green projects already reside on this host. The VM with a shield should be isolated and has the metadata project isolated. The scheduler will pick the compute host on the right to spawn the VM, because it is an empty separation host. The creation of the VM succeeds and the former empty separation host is now dedicated to the blue project. The blue project has now allocated the separation host as we can see in the CLI example. You can see that the project ID is not known anymore. As in contrast, two slides earlier where the separation host was not yet allocated. If the green project now also wants to create a project isolated VM, the scheduler will return that there are no valid hosts available and the creation will fail. The scheduler will try to select a host which is a separation host for the green project's instance. As the left compute host is not a separation host, it will not be selected. The right host is a separation host, but it is already dedicated to a project. The scheduler will check whether the project ID of the green VM is equal to the project ID which stands in the compute host's metadata. But this was the blue project's ID, so there is no match and the creation of the green VM fails because there was no valid host available. To conclude, with this flexible compute host isolation, we have the positive aspects of both. The ironic separation and the host aggregate flavor usage. But none of the downsides. We let the scheduler do most of the work and just add host parameters to reflect the state of a separation host. It is flexible as ironic. Every project can easily use such a compute host. And the OpenStack API can be used to manage instances, just like in the normal workflow and same as in the host aggregate case. Conclusion. So we saw in specific scenarios, virtualization might not suffice. Physical separation is desired or mandatory. We explored three existing mechanisms for separation, with varying degrees of complexity versus usability. Thereby, we introduced one new approach using Nova Scheduling. The flexible compute host isolation. Which is a flexible and OpenStack native solution. By that we saw how the seemingly a contradicting approach of introducing physical exclusivity can still fit into the cloud paradigm. Thank you, feel free to get in touch and happy Q&A with my colleagues Yusefine Seyfert in Markus Hensch.