 Hi, my name is Tsu Min Chen and I'm a Principal Software Engineer at Red Hat. For the past year or so, I've been working on the Elastic Secure Infrastructure Project, or ESI, along with other Red Hat engineers, as well as people from the Massachusetts Open Cloud or MOC. So what is ESI doing? Well, from our README, we want to create a set of services to permit multiple tenants to flexibly allocate bare metal machines from a pool of available hardware, create networks, attach bare metal nodes and networks, and to optionally provision an operating system on those systems. One key thing to note is that ESI does not compel users to a specific service architecture. You can put the pieces together yourself to support your own architectures and project goals. That's your goal, but what is ESI actually doing? To explain that, I like to think of ESI as more of an enablement initiative that aims to fill the gaps in OpenStack in order to fulfill its goals. So we'll work upstream with the established OpenStack projects to expand feature sets, we'll write custom code where needed, and we'll provide guidance and support to other groups working in this space. The best way to illustrate the sort of work that we've been doing is to talk about an ESI project that we're currently working on in the MOC. That project is an internal hardware leasing system. The goal is to replace the MOC's current internal hardware leasing system that uses complex custom scripts that are powerful but difficult to maintain or extend. The idea behind the ESI version of this hardware leasing system is that users contribute their own hardware to the bare mill pool. Those node owners have the exclusive use of their nodes. However, when the node owner is not using their node, they can put them up for lease, allowing the less see to temporarily claim the node and use it for themselves. At the high level, this is what the hardware leasing system service architecture looks like. In the middle of the DAW, it's a bare mill service where we keep our inventory of nodes. A leasing workflow allows owners to offer up their nodes while they lease nodes. Then, a provisioning workflow allows both owners and lessees to perform provisioning actions. They can provision their nodes or configure node networking. Let's dig a little deeper into the hardware leasing system workflows. We'll start with the creation of offers. Let's say that the bare mill inventory has five nodes, A through E, with nodes A, B, and C owned by owner 01 and nodes D and E owned by owner 02. Owner 01 is not using any of their nodes, so they offer them up in the leasing service by creating an offer for each of their nodes. Owner 02 isn't using node D, so they offer it up as well. However, 02 is using node E, so they keep it for themselves. Now the lessees come into play. Let's see L1 wants to use two nodes, so they look at the available offers and create a contract for nodes B and D. Lessee L2 only needs one node, so they create a contract for node C. No one creates a contract for node A, so another lessee can take it. And no one can create a contract for node E, since it was never offered up in the leasing service. Once the nodes are leased, the lessee can move into the provisioning workflow. Lessee L1 can use the two nodes they've leased, nodes B and D, in conjunction with the public network and the two private networks that they've created. They can perform a variety of actions, provision a node, attach networks to their nodes, and network actions will be reflected on the switch. Lessee L2 can do similar things, but they're limited to node C and the public network. So that's what we want to do. What steps have we taken to get there? To answer that, let's look at the partial requirements breakdown from the Halcyon days back in 2019. It's not a complete requirements list, but it illustrates a lot of the work that we've done. Let's start with the bare-middle service, or Ironic. We need to be able to add and remove nodes from inventory, something Ironic already does very well. We also need the ability to assign owners on lessees to a node, and for those owners on lessees to be able to perform limited API actions. Combined, this translates into nodes multi-tenancy, and it's something that was not supported by Ironic when we started working on the hardware leasing system, and it ended up being something we implemented ourselves. Next are the network requirements. We wanted users to be able to create their own networks and to potentially share those networks with another project, things that Neutron already supported very well. The MLC also needed ML2 Ansible networking to support the Cisco Nexus switch, and we discovered that was something we need to code ourselves. Finally, we needed the ability to create and manage trunk ports. After some investigation, we learned that Neutron did most of what we needed, but there were some pieces that were missing. The next requirement is a bit of a strange one. Simplified CLI commands. We'll talk about this more in a moment, but the gist of the issue is that the default OpenSack CLI is comprehensive but not always easy to use, so we decided we would need to give our users some simplified CLI commands. Finally, there's a leasing service, which we created from scratch. It needed to allow node owners to offer nodes and for lessees to contract a node for a given time period. Let's go into more detail about the work we did. We'll start with Ironic node multi-tenancy, which was the key feature we needed above all else. We started by talking with the upstream Ironic team about our requirements, and they were amazing at walking us through our ideas and implementation, helping us refine our spec and giving us feedback about our code. The implementation steps started with adding owner and lessee fields to nodes. Ironic actually already had an owner field for nodes, but it was purely informational. Next, we exposed those owner and lessee fields to policy. OpenSack policies determine which users have access to which API actions and can be specified in an easy-to-update configuration file. Before our work, all Ironic API actions were only accessible to Ironic administrators. We added two new roles, isNodeOwner and isNodeLessee, that determine whether the user making an API request is the owner of the node or the lessee of the node. With those in place, we can now update the other policy roles. For example, we can update baremail node update to allow access if the user is an admin or the owner of the node, and we can update baremail node setPowerState to allow access if the user is an admin or the owner of the node or the lessee of the node. Note that these policy roles are fully customizable by each individual OpenSack installation. There were a few additional minor tweaks necessary, but that was mostly it. We tested our code with single Ironic CLI commands, and then we tried something more complicated, using Mel Smith, which is a client-side Python library for provisioning Ironic nodes. What we discovered was that Mel Smith just worked for node owners and lessees with our code. Owners and lessees could provision nodes that they owned or leased, and cannot do so for nodes that they did not own or lease, which was exactly what we wanted. Onto networking. We started by adding support for the Cisco Nexus Switch into ML2 Ansible networking with guidance from the upstream networking Ansible team. They were great at helping us understand what we needed to do, and helping us merge our code, and that was that. Next, we tested trunk port support. We discovered that Neutron already supported the creation and deletion of trunk ports. However, we couldn't attach a trunk port to a specific NIC, so that's a feature we added, actually, in Ironic. Lastly, we found out that updating a trunk port didn't fully work. If a trunk port was attached to a NIC and then updated, the change would not be reflected on the switch. So we made plans to implement this feature, and we got around to working on it. We discovered that someone had been messing with the punch by a week or two, which was great. Simplified user commands. Why? Well, because if you use the default OpenSax CLI, figuring out how networks are attached to a single node requires two plus two times the number of NICs commands, so six commands if your node has two NICs. You also have to use the output from one command as input for the next, copying and pasting UUIDs. Then, you still have to collate the results of all those commands. But what we learned was that it's actually really easy to extend the OpenSax CLI. No one on our team had much experience with the CLI code, but within a few days, we had developed OpenSax ESI node network list, a single command that produces the output you see here. We've created additional CLI commands to simplify various operations, and you'll see a lot of them in the moment in our demo. Finally, the leasing service. We started by looking at OpenSax Blazor, which is OpenSax Reservation Service. However, at the time that we looked, Blazor did not have full support for provisioning on Ironic nodes. It was also tied closely to NOVA, something we were trying to avoid as Ironic can provision nodes on its own, and we wanted as simple of a service architecture as possible. Lastly, Blazor did not support policy-based node API access. You might be able to provision an Ironic node, but you couldn't allow a Leci to power-circle node. So last summer, with the help of a bunch of Red Hat interns, some of who are now full-time Red Hat engineers working on ESI, we developed ESI Leap, which is a pretty simple alternative. It allows resource owners to offer their resources and resource Leci to contract and offer resources for a specified time period. For Ironic nodes, contracting a node just means setting the nodes Leci attribute. ESI Leap supports a leasing of Ironic nodes but can easily be extended to work with other resources. Let's take one more look at the service architecture of our hardware leasing system, and hopefully you have a clearer picture of how all the pieces interact. One thing I really want to emphasize now is how disconnected our leasing and provisioning workflows are. Maybe you don't need a concept of Leci's or maybe you're fine with owners assigning Leci's themselves. You can do whatever you want there and still take advantage of all our work in the provisioning space with Ironic node multi-tenancy and simplified CLI commands. Okay, let's move on to our demos. First, we'll see a demo of our leasing service and Ironic's multi-tenant node capabilities. We're going to start with three users, each with a different view into Ironic's node inventory. The admin user sees every node in the inventory. The owner sees the three nodes that they own, and the Leci has not yet leased any nodes so they see no nodes at all. Now the owner has decided that they'll share Dell 2 so they create an offer for the node by specifying the Ironic node resource type and the Ironic node's UID. Note that the owner can also specify a start time and an end time for the offer, but we'll just specify an end time so the offer will be immediately available. A Leci can see the offer and when it's available. They can now create a contract for that node. After the contract is created, note that the availabilities for the offer are also updated. Other Leci's can still contract the node for those times. When we reach the contract's start date, the manager's service will fulfill the contract by setting the node's Leci field to the Leci's project. When the contract's expiration date is reached, the manager's service will also unset the node's Leci field and clean the node. Now that the contract has been fulfilled, the Leci has access to the node. They can also perform specific actions as dictated by Ironic policy. For example, our Ironic policy allows Leci's to power nodes on and off, but not to set or unset node attributes. Here are our Ironic policy changes. This is the generic node update rule which allows API access to only admins and node owners. And here's the set power state rule which allows API access to admins, node owners, and node Leci's. This Ironic policy is restrictive but still allows Leci's to do pretty complex things. For example, using mail Smith to provision a node. This operation takes a while, so we'll skip ahead in time. Provisioning is now complete and the Leci can now lock into their provision node. That concludes this demo. Next, we're going to look at our custom OpenStack CLI commands. The base OpenStack CLI is powerful and comprehensive but often requires a user to run multiple commands. When I run an ESI CLI command, I'll also tell you how many OpenStack commands will have been needed for the same result. We'll start with a node owner who can see the three nodes that they own. Let's look at how neutron networks are attached to these nodes by looking at the bare metal nodes and ports. Looking at this bare metal port, we can see that this corresponds to this node UUID which is still two, and there's no information in internal info so you can see that this bare metal port is not attached to any network. Actually, we can also use an ESI CLI command to get the same sort of result. In here with one command instead of three, we can instantly see that none of our nodes are attached to any networks yet. Let's verify that there are no networks attached to Delta 2 on the switch and we can see that there's nothing attached. The next thing we're going to do is create a trunk port. Let's see what networks are available. We're going to use these three test networks. We'll start by creating a trunk port with test one network as a native network and test two network as a tag network. This ESI command would take five separate base open stack CLI commands to run. We've created the truck port and we can see some summary information. This summary information would have taken four base open stack CLI commands. Now let's attach this trunk port to Delta 2. I'm going to specify the trunk port, the MAC address and the node. This command would take seven to 10 open stack CLI commands for the equivalent depending on what state the node is in. The network attachment is complete. Let's take a look at our summary node network list. Now we can see the attached network on Delta 2. We can verify this on the switch as well. VLAN 611 and 612 should be accessible and we see that that is indeed the case. Next, let's add another tag network to the trunk. This would take four base open stack CLI operations. Let's verify this update with our ESI CLI commands. We can see that test three network has been added as a tag network right here and our summary node network list also shows that same information. We can verify this on the switch as well making sure that VLAN 613 is accessible and that is indeed the case. Okay, now let's clean things up. We'll start by detaching the network from the node. This would take three base open stack CLI commands to execute. Let's verify that this update happened on the switch and we can see that the switch port is now shut down. Finally, let's delete this trunk port. This command would take five base open stack CLI commands for the equivalent. That concludes this demo. So what's next? Well, we're working towards a trial implementation of our hardware leasing system in the MOC with just one key feature missing which I'll talk about in a second. There are other scenarios where we might use ESI. A red hat colleague working on OpenShift thinks that ESI would be perfect for creating OpenShift bare metal clusters that can span and shrink on demand. There's also FlockX, a PhD project at BU. FlockX is a hardware marketplace where users can buy and sell use of bare metal nodes. We also have some feature development to do. We've already started to look at no attestation with KeyLime. The MOC also needs Cep IScuzzi support in OpenStack for its hardware leasing system and there's an upstream patch that we'll be testing out. Then there's improvements to the leasing service, reporting reservation limits so that one SCCAN leads all the nodes until the end of the time in the UI. Further information, the Red Hat Research Quarterly has an article about our multi-tenant Ironic implementation called isn't multi-tenancy ironic. So if you'd like more details about what we did, read all about it there. The ESI Get Repository is a good place to look at our code and documentation. And finally, you can reach us at any time through IRC on the free-node server in the MOC channel. Thank you for listening to this presentation. I hope you enjoyed it.