 So, hi everyone. Good morning. I'm Naina Jain. I work for IBM Linux Technology Center out of IBM US and I'm mainly leading the secure boot effort for open power systems. Today the topic is open power secure boot host OSP management. This is a disclaimer just for I'll just flash it for a second or two and then we can just go ahead. Okay the agenda is we'll start with some background then we'll get into the high level overview and actually the details after that and then there's a debug and recovery thing and finally we'll end with summary and patch stages. I would like to give the acknowledgments to my whole IBM LTC security team who had been working and contributing to this work and our collaboration with IBM Security Research Group and IBM Enterprise Power Security Team. Okay so let's get started. Well there have been enough of presentations over the secure boot around the years so I'll just give it's basic definition to get started here. So the secure boot it actually implies that only authenticated firmware and operating system are allowed to be executed at all times. So it involves authenticated firmware that means there are two things which are involved as part of secure boot. One is signature verification and another is key management and these key management is required to handle the keys which are used to verify the signatures. The signature verification part there was a presentation on it we had given last year in Linux security summit Europe 2018 and this year so today's presentation will mainly focus over the key management part. Okay so the high level overview this is just again to get into the flow will not get into details again the details of this this has been discussed about in different presentations in different conferences. So let's get started here secure boot open power secure boot if you look at the block it starts from here the main hardware the core root of trust which is starts from sb and it loads host boot is key boot is key root host source. So this is actually divided into two parts there is a firmware and then there's a host source so the whole secure boot on open power has two domains one is firmware secure boot and once the is key boot which is also called this opal it loads the boot loader we start with the host source secure boot. So today's focus is mainly on the key management part of the OS secure boot that is for the loading of host source one of this important thing to remember here is that the boot kernel in case of open power it is actually a pretty good application running running on a Linux base kernel. So we use a kexec here and then it loads the host source and the key management and the signature verification are handled differently for some firmware secure boot and the OS secure boot. So we'll be only focusing on OS secure boot. Okay so what is key management? So let's say a key management is basically managing keys in a cryptosystem that involves creation protection storage and control over the usage of the keys. So as part of this it goes to different phases where you start with key generation and installation it's a storage where do you use it and then how do you destroy it in case of revocation or expiry or rotation if you need to and what are the currently existing mechanisms as part of secure boot key management so they are like different but in context of secure boot. So there is like currently we have UEFI key management and then there's the shim which does its own so if we just see it into specific properties like the variables which have been defined for the UEFI are like pkkk dbdb the thing is by default it is bit in a controlled way and you can reset it later but by default it comes as a controlled thing and the shim has mark list and mark list text and shim mainly takes care of the OS secure boot and that is once you get into the bootloader but the UEFI actually is the same for the firmware and then to the OS. The key hierarchy in case of the UEFI is you have the highest-scale authority pkk then which is from the OEM then kkk and then the dbdbx which is for the OS verification and in case of shim it is the password based. The tools which are used for UEFI tools and then shim it is mockutile. The key containers where the way you store the certificates or the hashes in case of UEFI this UEFI signature list that is ESL and in shim also it handles the ESLs. Authentication of key update the UEFI has the UEFI authentication variable header which is like it embeds the detached pkcs7 signature format and in case of shim it is password-based and for the key updates you may or may not require physical presence for UEFI but depending on how you do it but in case of shim you need physical presence because you need to enter the password when you are into the mock manager for enrolling the keys. So this is what exists currently so now let's see what is on the open power and the main goal is to keep as much as compatible as possible keep it secure keep it flexible so that it can be extended with ease of automation. So let's go through this journey of key management on open power and see if we are able to achieve this goal or not. I'll break this key management into the three layers so there is a user who has to do the operations like the sysad means the distros and all and then this to get the user interface is exposed by the kernel which is the second layer which provides also an access between the user and the firmware and the kernel is the one which is also going to use it because the boot loader is kernel-based and then there is a firmware which actually does the key management uh the key updates processing it is the one which is controlling the the the handling or controlling or handling the keys so the key initialization its verification and storage part is with firmware so let's get more deeper again breaking it down into further components the firmware we need to store the keys to make them persistent so here comes the key storage this is the persistent storage the persistent storage are two here there is a pinor and the tpm the thing is we need a secure storage to ensure the integrity of the of the keys tpm provides the security and pinor provides the enough space which is not there in tpm so we combine the good properties of both of them and try and provide the secure storage this storage is accessible via firmware which is the iski boot in case of open power iski boot or you also call it as opal and it handles it provide access via opal apis to the kernel and the user interface and it does all the handling of the key updates key manipulations who can access who cannot access etc and then uh this is the kernel which provides access for the user space and again as i said it also uses it and manual is the user space the ellipses which i've shown here are available at runtime and the rectangles which i've shown here are actually the boot time the components which actually does the processing at boot time the one thing which i want to specify here is the most of the key of this processing we are doing is an iski boot not in the kernel why can't we actually make the kernel to do all the processing the thing is the hardware the actual the key storage which is accessible we we wanted to keep the exposure limited and block this so access to it as early as possible because the tpm security which we do is is by locking it so we didn't want to expose it to the point where the users or the networking gets enabled or the file systems gets enabled and all so we wanted to lock it access to this as early as possible and that is where this keyboard this reduces the exposure so that is where it's done at the firmware side okay so we will take a top-down approach we will start from like assuming a user wants to start with setting up the keys and how does it go through this the whole process goes so key generation and installation okay uh i need a tool to generate the keys what should i do so which ones should i use so support uh asymmetry keys it's a signature verification mark asymmetry keys rsa2048 and sha512 the container for the keys that is the certificates and hashes is tianoco derived efi signature list so we use esl as a container for the keys the updates when you sign the keys are for the update the structure which hold this signature as well as uh new data is uh again tianoco derived which is efi variable authentication to descriptor we use the authentication to descriptor as it is based on time system and it helps protecting against lipid attacks the tools are uh we have taken the minimum utilities from the efi tools and ported them to the power and they are called as the sacred tools now so the three utilities which we have ported are search to efi siglist which converts from certificate to as an esl uh and then the sign efi siglist which signs it and the hash to efi siglist for converting hashes into a efi container this is how the whole structure looks like uh when you so if using these utilities i generated a certificate i generated an esl and i generated a signed command this is how it will look like it will have a header structure authentication to which will have some of its metadata and then a pkcs7 detached signature this and at at the end appended with the new keys esl which you want to actually get updated onto the database so i have now this command with me which i need to give to the submit to the system or install onto the system now here comes before uh i install ms allowed to install or who should be doing the installation so the comes the there comes the point of authorization and who is authorized to do what we the the the variable names are if i like uh we continue to use we use pk k ek dba dbx the pk is the highest security authority it has the authority to install the intermediate keys and it can be more than even for the secure boot the kks are the intermediate keys who has the authority to install the db or dbx for the signature verification the db and dbx are the keys used for kernel verification kk can sign the db or dbx both that's so the thing to note here is each esl hand uh source multiple certificates and one key can sign one esl and it does not sign in sign individual certificates in the esl so it is we currently support replace mode and not the append mode now again the point here is the the question can come why why did we decide the three-level hierarchy we could have decided the two-level hierarchy it could have been pk with db alone and it could have been k ek with db alone there's no need of pk so but if you see this uh assuming take an example uh there's an organization and who have there's a there's there's one organization and it has multiple departments and each department has a sys admin and even a department can have sys admins in the shifts working so so if there's only a single pk then it's the organization which has to handle take care of assigning authorities for or handling the dbs for everyone if there is no pk but there is a there are multiple kks then there is a complexity who authorizes whom and when you do there is also complexity of who has actually installed the db or who has installed the dbx so how do you keep track of all of this so this defines this gives an optimum level of managing the hierarchy at the organization level you have the highest department level pk authorized who gives the systems and assigns kks which can be sys admins in each of these departments and then these sys admins are allowed to install the db keys and this is admins even if they are working in the shifts they can install the db or dbx keys so this is how this hierarchy becomes an optimal level to do to achieve what is required as part of organization key management thing uh now another aspect of the flexible key authorities this is what as we say is that there might be different cases where i want the way i want to install it i might be a sys admin and i want to install my own linux kernel i might have got something shipped but i want to install my own linux kernel so in that case i have i generate my public private key pairs and i sign the kernel with my db key and then i have my kk so i i can sign my db with my kk and then install it that is one case there might be another case where i'm the sys admin but i've got the i have to install a distro package a distro is and i would have got a key from them so how do i install that so in that case i i use that key as a db key but i sign it as a sys admin with my kk and then i install it and there can be a third case where the system comes with the shipped pre-shipped db pk kk and db and then in that case they want to have their own keys installed so it provides a flexible key authority and it's there's no central system control and it provides them options based on the requirements you want to satisfy so this is how the design for the key hierarchy and what how many levels are optimum has been decided okay so i got the keys i know i need to install it and what's the interface available to me so there are two interfaces we have expressed as part of the kernel one is device three so the properties are of the internal key management status are read only properties which are they're exposed via device three uh the OS secure enforcing property so we call it the interface as a sec where you'll see i know the sec where the OS secure enforcing property tells whether the secure boot is enabled or disabled the second thing is for submitting the keys uh we have read the c surface interface which allows you to do a read or write of keys and they're exposed via sys firmware secure verse and it can be enabled via config secure c surface the patches for it are in the work of progress which we are trying to get upstreamed so the kernel has got the keys via c surface and it needs to give it to the firmware so the next step is kernel firmware interface and that comes as part of the interface which we have implemented called the sec where operation it provides a pluggable interface get get next set uh power pc base platform can implement uh this interface to provide to use the c surface and for different platforms since currently the secure boot support we are doing is for power nv systems we have implemented opel apis as the interface at the iski boot level so now at the firmware level so this is the kernel interface for the firmware but now firmware also has to expose so that is the opel api interface which and that work is being done by arik richter from my team and that is where the kernel starts talking to the firmware kernel had given the request or the key update command to the firmware no other firmware do it so it has to handle that command uh so there comes now storage uh the pinor storage so i said in uh we have two storage pinor and tpm the tpm provides the security and pino provides the storage uh um i mean space more space pinor is divided into two parts one is for the keys and one is for the update command so whatever key updates we have to do that is stored into the update bank and into the update bank partition um the key store is maintained as two banks so that if you need to roll back because of any failure reasons you can do that and they are there is also an in-memory cache maintained by the firmware itself which is available at runtime so that you i mean the the during the boot time the keys are loaded from the persistent storage to the in-memory and that is what available at runtime again this support is like the storage driver support is like a pluggable so it can be extended to support different storage devices uh for now it is implemented for pino and tpm combination the tpm driver at the firmware level is taken from ibm's tss which is by kenneth goldman who is from ibm security research and it has been modified to use mbit tls so so we we got the uh uh command we stored it into the update bank and what next so the next thing is uh yeah before the next thing so there is a storage in journals this is the pluggable book which i was talking about and um if you want to add an addition an additional additional storage device which is different from pino storage these are the interfaces to be implemented so that is how it is made is flexible or extensible okay so now once you store the key update command there's a need need of reboot you reboot the system and then when the reboot starts before update processing so when the reboot starts and the system comes back to the firmware opal ski boot level it starts the processing of those key updates and the component which handles this is called the backend component for the key management again the work is being taken care by eric richter uh what is this backend actually it provides it defines the supported variables the signed this structure or the signing algorithms for the keys and it also provides the definition of the i mean what type of authority or the authentication is allowed or the key hierarchy is the secure boot modes so it basically it defines what the user space tool should be doing and how user space tools can know about this is by you looking at the device the property which is available the sacred backend the backend compatible flag can help the user space tools it it lets users space tools to know what type of the signing algorithm or the modes or the the structure the container structure is being supported uh this is where the backend comes into picture again the backend is flexible tomorrow if you want to change the support for the signing algorithm or we want to hand a different container or different way of handling these modes then it can be done by because as they are provided as extensible uh you need to update the compatible flag to represent what you are providing the thing is uh it it you and and the driver should provide these functions pre-process process post-process invalidate the one which we are implementing currently is derived from tianoco reference implementation edk2 and that's the and it looks because like we we are processing the esl and the authentication header so that is how we call it so now though when the backend comes into it gets loaded it uh it it leads the update commands from the each of from the update bank and how they get processed if it is a pk and if it is the first time pk it gets sold but if it is there is already a pk existing in it that means it has it should be very favorable by the existing pk so presence of pk defines the secure boot mode if it is there is no pk the secure boot is disabled if there is pk secure boot is enabled the kks are verified via pk and the db or dbx is very favorable via pqk in case the secure boot is disabled no verification is done if the secure boot is enabled the pk csv structure from the authored is extracted and the new esl data which has been submitted gets verified against it we have the we are using mbtls as the crypto library at this key boot of firmware level but the but the mbtls does not provide pk csv support so we had implemented a limited support for it and we are going to open source it we are working on it so we are going to submit the request for the mbtls group and contribute to that uh so this is how the key updates processing gets done and if the signature gets verified it gets stored into the key store partition of to the pinot now so let's get that is where comes apart of storing into the memory making it persistent so what does that involve so now comes apart of the protection of the key storage or the protection of the key database so the kks or the db or dbx which are intermediate authorities they are stored in pinot but the highest security authority which is pk that is stored in tpm to reduce the attack surface again then and the pk is single then the integrity of pinot is maintained by we store the hash of the key store of the pinot into the tpm so there is always a check done whenever the system loads it always checks the integrity of the pinot key store with the hash into the tpm before actually loading it into the in memory and if there is a mismatch it it it considers it as something something wrong or something malicious or corrupted and then it does not load the key store the steps to maintain the pinot secure by tpm are so when the key store has to be made persistent the first thing is does is it writes the hashes of the key store the in memory key store to the tpm then it makes the key store persistent to the pinot and then it right logs those tpm envy indices so whenever i was saying is storing to the tpm i meant it is stored into the envy envy storage of the tpm and then it right logs them so that it after that nothing can be updated into those indices and it can be unlocked only after reboot so by the time the opal or the firmware now loads the bootloader and which now has access even the network access or the file access the tpm has already been logged and there cannot be any modification done at the key store so this is where it the point comes off that we are actually doing the the processing at the firmware level and not the key corner level and once it gets everything is locked the locking is done by the platform authentication property of the tpm so this is where how the key storage is made secure with the combination of the pinot and the tpm so i have got my keys a process updated is stored into the pinot they are also available in memory but they are actually this always was being done so that bootloader can use these keys for the kernel signature verification so let's come to the key usage there are two things of the key usage one was the update key updates verification itself which we already talked like pk verifies kk and kk verifies dbdbx so variable bank actually verifies the update bank the second is the kernel verification and this is where the we did all this so that the bootloader can verify the host source so how does the kernel gets the keys which are actually at the firmware or which are actually handled or controlled by the firmware so it is the same oppel api interface which explains from its back at the for the ccfs interface for in context of ccfs interface so the kernel keyring code it calls oppel api interface it loads db keys to dot platform keyring and the dbx keys to the dot blacklist keyring the dot platform keyring is what was being introduced by us in the last season time and this is the one which is being used by aima so the for the signature verification we use aima and aima uses the platform keyring it uses it's we define the arch specific policies some of this was being done as part of signature verification and they are being upstreamed this support of the loading the keys in power pc for dot platform and dot blacklist can be enabled via load ppc keys and this is how it looks like this dot platform and the blacklist coming to the blacklist currently we support the hashes of the blacklisted binary for the signature verification we use appended signatures and not extended attributes parts of the aima because the extended attributes are not available for all the cases like for the netboot and all so we use the appended signatures and the blacklist binary when i say it implies the hash of the binary without the appended signature so that is what the support of the blacklist is here so this is in summary the verification flow so kernel called opelepia is got the keys from the firmware stored it into the keyring that is the kernel memory and now how they use it so if the secure boot mode is disabled none of the verification gets done but if the secure boot mode is enabled enabled then the keys are loaded db keys are loaded to dot platform there is a policy arch specific policy defined which says our kernel check kexec kernel check aima flag is check blacklist that is check for the blacklist before doing the appraisal and the template is mode c template which is for appended signatures and then before doing the appraisal appraisal it looks so aima looks at the policy so it knows that it has to check the blacklist so it checks against the dot blacklist and if the binary is not found this blacklisted it actually does the appraisal and if the appraisal is successful it loads the host source so this is how now the key got used i and so we have taken we started from the top and we also taken the uter and we are back to the user space where the host gets loaded uh what is the case now when i want to delete my case because for a case of revocation or the expired and also how do we do that so you will delete db db exit kek by submitting a signed update with empty data if we have a valid pk still existing then it can be also be deleted by submitting signed update deleting a pk means disabling the secure boot so it's it's possible to delete pk only if it is validly present into the tpm if pk is detected as corrupted or its private keys lost or pk is reworked or if we want to change the ownership of the system then in all these cases we have we can only delete it via physical presence or we can already only modify via physical presence and the key rotation is basically okay it's it's basically doing the full cycle that you generate the new key you delete the old key by submitting an empty day is signed empty data and then you install the new key and how do you do any logs error logs or recovery or debugging it so uh open bus systems the petite boot which is the application and then does the boot loader part it has its own ui you can see the status of the secure boot onto the ui uh we are starting with the minimum interfaces onto the ui side but then over the time we plan to make the key management and everything from the ui interface for now it is available from the ccFS um any opal error logs can be viewed via sys firmware opal message log the kernel error logs can be seen from the sys log the device t properties can be looked at to see the secure boot state whether it is enabled or disabled the device t properties have the properties for status and update status which tells whether the update processing was successful or not and the key manage key management in the the sq boot got successfully initialized or not the key uh uh the pk if the pk private is lost then it can be recovered via physical presence i mean you clearly then then you install the new one but you you need physical presence in that context and so the key takeaways for open parking management is system owner is in full control no central ca currently supported modes are like setup and user which in context is like enabled disabled and we will add the audit mode support in the future it provides ccFS interface for user space we have ported minimum efi tools to the power and it's they are called the secret tools uh the support is like for efi like variables pk kk dbo dbx uh it accepts cianoco derived esl and efi variable authentication to descriptors it requires a system reboot for the update processing and storage is secured via maintaining integrity checks with the tpm the physical presence is required for the recovery of pk and it's supposed to replace mode as part of the initial starting and not the open mode so these are the key takeaways so let's see did we achieve our goal and we said that we need to we want to keep it as much as comfortable secure flexible and ease of automation so the variables which are used are pk kk db dbx the key ownership is flexible there is no central sca and there's no default uh controlled like thing so it is open flexible the key hierarchy is pk kk db pk to kk and kk to dbo dbx but if somebody wants to keep it minimum and just a single authority then pk to dbo dbx is also possible uh the tools which are used are secret tools and they write from efi tools the container is efi signature list the authentication key update of key updates is done with the efi variable description authentication to descriptor and the updates do not require physical presence until you actually need to recover the pk or clear the pk without authenticating it so that is the only place where you need physical presence so it can ease the automation for the organization so that is what we said we want to give as much as compatible secure flexible and then easy for automation and this is the patch setters the work has been going on for the ski boot mail into the ski boot mailing list the embed tls for the pkcs 7 support the kernel to get the ccfs interface and the arch specific policies and the blacklist support for blacklist support from the ima site arch specific policies for power and the loading of keys for power the signature verification related patches have been accepted already the appended signatures had become part of it recently v5.4 so there is the work had been ongoing sometime and the user space interface the secret tools that is something work in progress and we will get it like yeah we have it working in ported and power but yeah like more publishing it these are the general references there is a development GitHub repo we are maintaining for now to try it out but the patches are still like in the process of upstreaming yeah any questions questions if not questions thank you