 And this is a agenda for this talk. First of all, I will talk about some of the security and reliability issues in ironic pre-juno and pre-kilo releases. And then I will go over some of the features in ironic kilo and idle driver that can be used to mitigate the security risk and enhance the reliability. Then Shu is going to take over and give a demo of those features. I'd like to cap my part of the presentation and discussion in 20 to 25 minutes so that I can have enough time, give enough time for Shu to complete his demo. Okay, so there's some security and reliability issues in the ironic, especially in the pre-juno and pre-kilo releases. So there's no ironic mechanism exist before to prepare a note for a clean start of deployment. So operators will have to manually configure the billboard or note. For example, they have to configure the ray configuration. They have to configure the phone work settings and they have to scrap the disk before they could put a note into ironic for deployment. And also it requires the manual configuration of hardware properties, such as the number of CPUs, memory size, and disk size. Those need to be manually configured. And the manual configuration process is really cumbersome in this Aero poem. There's also no segregation of management network from data network, which Deva covered in the previous session. Pixie driver use TFTP and TFTP protocol itself is not secure, so all data transfer will be in clear text. And also TFTP will encounter some of the packet loss and timeout issues when you use it in the larger scale deployment environment. There's also no validation process during the boot process, and so there's no mechanism to prevent loading of unauthorized option ROM, kernel, and boot-loaded. The good news is in Kiro release, ironic add a few features that can be used to enhance, to address the issues I just discussed. So in Kiro, ironic add the note cleaning features, which can be used to prepare a note from a clean start for deployment. We also add the highway property introspection function. So all the highway properties such as note, disk size, memory size can be configured by the ironic automatically. The idle driver also support a pixie list, virtual media deployment, which can be used as alternative for pixie deployment. Idle driver also supports secure boot, and I will give a brief introduction of secure boot and then talk about secure boot support in idle drivers. Okay, so before I move on, I just want to give a very brief introduction of idle drivers. Idle drivers are ironic parking drivers for HP Proline servers. They integrate ILO and Proline platform capabilities with Ironic, and so customers who deploy bell metal on a Proline server that can use this driver to enhance, to get extra capability and enhance their deployment experience. This features the idle driver support. We support pixie list, virtual media deployment driver, and this feature is available since Juneau. We also add the UEFI boot mode support and boot mode management in Juneau release. In Kiro, we add the note cleaning, highway and forward property introspection, secure boot, and boot option. Boot option means that a user can specify they want the bell metal note to be booted locally or network boot. If you're interested in more details about this idle driver, we have online document on the OpenStack review. So note cleaning. I like to think of note cleaning as something similar to the hotel room cleaning. So when a guest checks out from the hotel room, the room needs to be cleaned up, and everything needs to be put back in order, and the room key needs to be reset before you can give the room to the second guest. And so ironic note cleaning is... It does similar things for preparing a note to a really long state, so it can be used to deploy, already deployed. Kiro provides a parkable, ironic note cleaning framework. It allows vendors to park in their note cleaning function into that framework. So idle driver parking the note cleaning functions into this framework. So currently we support... We set our system run forward setting to baseline. You can reset secure boot key to manufacturer default, and you can clean the user's secure boot keys. You can reset idle and also squat the disk. We are working on great configuration and firmware update, and hopefully those features will be landed in Liberty. So note cleaning is enabled by default, but you could disable it for whatever reason if you want to disable it. It comes with a set of default priority, but the priority can be as configurable, it can be changed. If you don't want any specific step of the cleaning to be performed, then you could turn that step priority to zero, and then ironic will skip that part of the particular step. Okay, so for hardware introspection, we work with upstream, and we actually contribute the ironic API and CLI to do hardware introspection. And currently we can discover the basic property that required by bailment or scheduling, which including the CPU architecture, number of CPU, memory and disk size. The ironic will automatically add the node property for the discover hardware properties, and so user doesn't need to do manual configurations. It's easier to use than less error form. For idle driver, in addition to all the basic properties, we discover some extra features. So we can discover whether a node can support secure boot, and we could discover the firmware version for system firmware and idle firmware. We could also discover the server model for the bailment or node, and whether the bailment or node has any GPU devices connected to it, and also discover the maximum link speed that this bailment or node can support. These features, this property then, idle driver will automatically create the node property for this discover properties, and then user can create favors to, based on those discovered node capabilities, to press their workload on a bailment or node that can match their security reliability and resource requirements. So for example, that you could create a favor with a secure boot, you could create a favor with firmware version that has certain half fixes needed by the workload, or if your workload requires a lot of bandwidths, bandwidth intensive, then you could specify a favor with the big fat pie like a 10G network. 10G link. The idle driver did the introspection out of bands, which means it doesn't need to boot up and rent this and to do the discovery, so the discovery can be done anytime, regardless of the OSS running or not, and it's faster because it doesn't need to boot up and rent this to do the discovery. The idle driver also supports virtual media deployment, and if this driver is used, then it will deploy RemDisk and Kernel, it will deploy the RemDisk and Kernel through the idle virtual media over a separate management network. So there's no need to use TFTP and no need to set up pixie environment. The user image is retrieved over data network so to speed up the large image transfer. This driver can be used Swift, and so user need to turn on HTTPS with Swift or use SSL proxy with Swift so that the file upload to or download from Swift can be encrypted. We are working on adding an option to not using Swift, no need to use Swift, and so when the option is available, we will update that information in our document wiki. We have two kinds of idle virtual media drivers. One is the idle virtual media iSCSI driver. This driver will use virtual media to boot the ironic deploy Kernel and RemDisk, and then use iSCSI to provision the user image from conductor to the bailment alert. We have the idle virtual media IPA driver. Same thing, you use idle virtual media to boot up the IPA RemDisk, and then IPA will take over to deploy the user image from the bailment alert. The good thing about this driver is user doesn't need to prepare different user images. They could just use the partition image used by the pixie driver and idle virtual media driver will automatically create the ISO image from those user images, from those partition images. We have this image builder element and also IPA tools that user can use to build the ISO image for the deploy, RemDisk, and Kernel. Let me move on to Secure Boot. Secure Boot is a UEFI standard supported in Windows and many Linux distributions, including Red Hat, Sousa, and Ubuntu. Secure Boot doesn't require TPM hardware, and the concept of Secure Boot is it builds a chain of trust and validation. So every boot component is signed, and the signature databases, including the DV and DVX, which I will explain next, are embedded in the system. So DV is a list of approved sign-in lists, and DVX is a list of revoked lists. And so signature of each component is validated by the previous component in the chain before they are allowed to be executed. So they have to meet all the requirements for the validation before they can be loaded and executed. So, giving Linux as an example, the system firmware will validate the shim. The shim is the first stage of the boot loader. It's typically signed by Microsoft UEFI CA. And once the shim is validated, then the shim will validate the grab2, which is the second stage boot loader signed by the OS vendors. And then grab2 will validate the operating system signed by the OS vendor again. So with secure boot, you can use it to prohibit loading of unauthorized option ROM, kernel, boot loader, and some of the Linux distro even support validation of kernel modules as well. So it can be used to mitigate security risk from malicious firmware and OS attacks. Okay, in order to use secure boot, the program need to be signed. And so the hash of the program will be compute, and then it will be signed by the signer's private key. Then the certificate and the signatures attached to the program and this program become a digital signed program. During validation, the hash value will be calculated from the program, and the signature will be decrypted using the signer's public key, and those two match will be checked to see whether they match. Those two hash will be checked against each other to see whether they match. And also it will check against the approval list and the revocation list. And so only after all the check and validation is completed, the component can be either denied to load or be able to load if it meets all the validations. Why? Hold on. Sorry, I messed up the slide. Let me go back to the... Okay, so idle driver support secure boot. Thank you. So idle driver support secure boot. In order to use secure boot, you first have to configure secure boot node capability onto the bare metal node. And there's two ways to do that. One is to use the hardware introspection API and CLI and then idle driver then would discover that the node can support secure boot. And if it does, then it will automatically configure that capability for the bare metal node. When we come in, this is a better way to configure the secure boot node capability. For whatever reason, if customer want... If user want to use the CLI to manually configure the secure boot node capability, the CLI is also available to do that. Then user will have to create a failure, a nova failure for secure boot so that nova can select the bare metal node that can support secure boot to serve the workload. So the next step is to create the secure boot failure and then you use nova boot with this failure. When the secure boot failure is specified and idle driver will then switch the boot mode to EFI and enable the secure boot as needed. So there's no manual pre-configuration of EFI boot mode or secure boot enablement is required. We also have the disk image builder option. We add an option to the disk image builder to create a digitally signed deploy and user images. So in summary, we recommend user to use hardware introspection to prevent the node configuration errors. Then you could create the failure based on the capability discovered by the idle driver to better control where your workload want to place. What kind of bare metal node you want your workload to place on based on the workload security, reliability, and resource requirement. Use a node cleaning to prepare a node from a clean start for deployment. And also use the idle virtual media pixie driver to segregate management data network and use the secure boot to protect your system from firmware and software attack. So there's still a lot more work to be done so we hope that you could join us in ironic project to make ironic bare metal provisioning more secure and more reliable. And then there's a demo time. So Shu is going to use idle driver to demo the hardware introspection idle virtual media driver with secure boot and not cleaning functions. Hi. This is Shu Arunthinlukar here. I will be taking you through the demo part of the... using the ILO I SCSI driver. I will take you through the creation of the images then creation of the node for the idle driver and how do you discover the properties and then deploy a secure image. And also to tell you if the insecure unsigned image is used how the deploy is going to fail. So this is about creation of the images. We have used Ubuntu signed which is... provides a signed kernel and the bootloaders for image creation. Here you see that it is installing the signed bootloader and signed kernel during the image creation and it builds the ISO with those bootloader and the kernel. Similarly, the user image is created using the signed bootloader and the kernel and this is the image that we are going to use for the secure boot deploy. So now we are uploading all these images into the glance. We upload the RAM disk ISO. We will be uploading the cloud image components like kernel and its RAM disk and then attach those kernel and the RAM disk to the actual cloud image. So this four things completes the whatever we require to images required for the deployment. So these are the kernel and the RAM disk associated with the user cloud image. So now we create node using the Iskasi ILO driver which is just the virtual media for the deployment. So before we initiate a deploy or any operation we move it to the managed state so that we can discover the hardware properties that this node is providing to us. From the managed state we move it to the inspect state where actual out-of-band discovery gets initiated. It is inspecting and it will also detect the MAC addresses and create the ironic ports for them and also it will discover the mandatory and the additional capabilities that are available with this node. So these are the hardware properties discovered and you can see that secure boot true is the capabilities discovered which says that this node is capable of deploying a secure boot environment. And these are the nicks that are available on this node and corresponding ironic ports are created for the same. Before we initiate we move back the node to the provide so that it becomes available for the NOVA and it is available for the provisioning. So when we move it to the available state the cleaning operations will be invoked so that node is cleaned up and all the ILO is reset and things are ready for the deployment. Now the node has come to the available state and it is available for the deployment. So before starting anything we will create the flavor for the secure boot deployment. So we are using some basic properties like two VCPUs maybe 8K of RAM and 50GB of disk space and some swap side. So upon creation of the flavor for providing that extra capabilities like secure boot we need to put in the additional metadata yet we are putting that secure boot capability and we are saying we need this to be true for the node that gets selected for the deployment. So now we have a flavor which says the node to be provisioned with a secure boot enabled. Now we use that flavor for the deployment with the user image that we just created a signed user image. So this initiates the deployment process. The deploy ISO is attached to the virtual media CD-ROM and the node is undergoing a boot. So we use Floppy, virtual media Floppy for passing the auth token and the other ironic API URL and this CD-ROM contains the actual deploy ISO that is going to deploy. So the initial deploy starts in the UFI boot mode. So it brings up the Ungrubman menu for the installation and the node is booting using the deploy ISO. So here is it gets the DHCP, IP and it talks back to the ironic with the URL and now the actual user image is getting flashed using the ISCSI. So upon completion it is changing the boot mode again to the UFI and turning on the secure boot on the node to on. It will reattach the boot ISO that will be used for the during the final bringing up of the user image and the node goes for the another reboot. So it's a virtual media URL as the boot ISO attached to it. So here the initial boot starts with the UFI. We are not at mode into the UFI secure boot because that will happen at a later stage wherein it has to bring up the signatures and load those signatures into ILO. So it detects ILO has detected it and it is going for another reset and it will bring up the server now in the UFI secure boot mode. Now the server is booting in the UFI secure boot. So it is bringing up the boot partition of the user image and now the boot has completed and we have got a login screen for the Ubuntu user image that we had given. So this is the IP that was provided to this node and it has come into the active state. So we can check the state using the Ironic CLI also and it is inactive. Also we will check that node is pingable. So this basically completes the deploy in the UFI secure boot mode for that node. So we will see how the cleaning happens. So we are terminating the node. This will cause the deletion of the instance. During the deletion we are disabling the secure boot on the node so that if the next request is non-secure boot still this node becomes available for the provisioning and other cleaning steps are invoked on this node which are configured as part of the node cleaning and the node is moved back to the available state for the next provision. So we will see what will happen if we use a user image which is not signed and try to deploy using the UFI secure boot flavor. So we are going to use the same flavor. I have pre-uploaded an unsigned image and we will just quickly see how the deployment handles unsigned image. So we are trying to launch the instance and we will be using the UFI secure boot as the flavor and we have an unsigned image. So again it is going to go through the same process. Attach the virtual media CD-ROM and virtual media floppy with the deploy ISO and the auth token and the ironic API URLs. So the deploy has started in the UFI mode. So it is flashing the user image onto the node just as it did in the last occurrence. And after that it is going for a reboot. So initially it will start in the UFI boot mode. Now ILO has discovered that it is a secure boot has been enabled for this node. It will try to reset the server and bring it up in the UFI secure boot mode. Now the user image is getting booted up. But since the kernel is not signed it is going to show that it is unsigned and the boot is failed over here. This is the way it will not allow any unsigned images to be booted if secure boot is enabled for the node. Thank you. So any questions? I am not able to hear you. Yeah, so the signed images are the images wherein there is a shim. Shim is signed using the Microsoft certificate and then the kernel and also the shim contains the Linux distro certificate like for example Ubuntu. Shim will be signed by the Microsoft and the shim will contain the canonical certificate and then the kernel is signed by the canonical certificate. So when the image starts booting up shim is the bootloader that comes up first in that it has a Microsoft certificate and the ProLiant servers are pre-loaded with the Microsoft certificate, SUSE certificate and the Ryder certificate, UFI certificates. It will validate those. It is one of them. It will bring up. The shim has the canonical certificate and it has been validated by ILO. So it is a chain of trust that will come into play. But when we have an unsigned image that means that shim or the grub is not signed by anyone and when it tries to bring it up in our case the grub was signed because it will use the grub from the deploy ISO and we had a signed deploy ISO but the user image was not signed. So the grub came up which was validated by ILO but when it tried to bring up the kernel from the user image which is unsigned by any signature over there. It did not find means it may be unsigned or has the signature which is not part of the shim and then it says it's an invalid signature and boot fails. Any other question? Able to sign my own images? Yes. It is possible to sign your own images then you will have to sign the grub by yourself. There are steps to sign and all the kernel components you can sign. Then you have to upload that public key of that signature. Signature into the ILO, that is possible but today we don't have the automatic way or the programmatic way to upload the signatures but you can manually upload that signature and deploy. Have you resolved the problem of segregating the management network from the data network? So when the bare metal instances provision which network it is provisioned on? It will be provisioned on the non-management network that is the user network. Is there an isolation problem? Security problem? For example, if you go into a public service provider a public cloud service then you would isolate the formatted tenant service you have to separate. So this is a work underway to be able to assign a different reading for the ironic management network and the tenant network. So that work is working progress. One more question. Can this bare metal instance be used just like another VM saying you can join a virtual subnet or it can be put into as a member of ELB and utilize other L4 to L7 network services? Are these functionalities implemented? Not that I'm aware of unless there's any other people in the audience know the answer. My understanding is that I only come to use a flat network to support VDN from the top of the switch layer so you could second get the traffic starting from the top of the switch network. All right, thank you. No, your question is ILO required to be configured? Yes, the ILO network needs to be preconfigured and when you create a node we actually pass the ILO credentials that is what is the administrator login and what is the administrator password IP is part of the node creation itself. Is there a question there? No, there is no security manager so this all goes into the ILO firmware. There is no certificate manager separately to be installed so ILO firmware takes care of all the certificates and you have to upload these certificates using the ILO console. Correct, so ILO firmware is working on providing the APIs so that we can programmatically upload these certificates into the ILO. So probably in the next version we may be able to do that. Once it is available we can use that in the Ironic as well. But for all that matter the Linux distro images if you are not using custom images then you don't have to do anything. These three certificates are factory uploaded into the ILO about Microsoft, Reddit and SUSE certificates. Question over here. Have you guys done any work with peripheral firmware? Do you either design your nodes so that I don't know if you are aware of the work but for instance there has been proof of concepts of key loggers being done from GPU firmware or being run on GPUs that are out of the scope of what measurements you can do. Have you guys either designed your nodes in order to avoid components that have peripheral firmware or have you found ways of verifying those? Actually I did not get your question fully. So peripherals such as GPUs or networks or hard drives carry firmware with them and it's possible for an attacker to overwrite these with malicious stuff. So some examples have been the NSA attack where they overwrite the hard drive firmware and serve up a malicious boot loader at boot. Have you guys done any work with ways to address that? So the ILO firmware is also a signed firmware. So HP also uploads its own signatures into the firmware and it is one of the first things. When anybody wants to upload any say malicious firmware he will not have the HP certificate to sign that firmware. So I'm not talking about the system for where I'm talking about. You're talking about option ROM. So you're referring to option ROM, right? So the adapter, the storage adapter, network adapter. Yes. So it is the option ROM and if the option ROM is signed then Secure Boot can actually validate that signature. So you're saying Secure Boot can validate peripheral firmware like GPUs? GPU. Because they're only connected by like a PCI bus so I don't know that there's... I know you can validate the network devices as well as the storage devices. For GPU devices I have to check. No, PCI devices, it does validate. The firmware validates those devices and that is the one of the things that is addressed using the UFI Secure Boot. Because you can have the rootkit installed through the PCI devices but it gets handed as part of the UFI Secure Boot protocol itself. Okay, thank you. A lot of good questions but is there a flow documentation of how this actually would come about when it does get the certifications for these systems? So the PCI thing is documented in the UFI Secure Boot pack itself that it gets handled through the... as part of the UFI Secure Boot pack. Right, but I mean just kind of looking for a flow or a workflow of how it would actually get done. Oh no, I'm not aware about that part how that handle gets handled in the firmware. Because we get to understand it. In terms of when you provision, you try to put this forward and you're like, okay well what do I got to do next? As well as the certificate management, you know we're talking about a single but as you start to do scale how that gets deployed we get to understand that system that comes up. Yeah, those are good inputs where you will consider... Just a nice little diagram. Yeah, that part we can do. Yeah, surely we can do. Thank you. Alright, thank you.