 Hello, good morning, good evening. My name is Reinhard Bündgen, and I want to talk to you about hardware using hardware security modules to protect your block devices in Linux. I'm at IBM, and I'm leading there a group that does crypto development for Linux on Z. And this will present some of the work we have done. So some formal slides, well, I'd like to acknowledge the work of my colleagues who helped us and of the crypto setup development community without whom this effort wouldn't have been the success it is right now. So before we start about encryption of disks and how to secure the keys of encrypted disks, I want to talk with you about attacks. And I typically distinguish two kinds of attacks. One sort of attack, so-called online attacks, attacks for which the attacker must include the system that is to be attacked. And therefore, the attack is probably observable in contrast to an offline attack where data is attacked not in the running system, but while this data is outside of the systems, be it on a disk outside, be it in transit. And such an attack is much harder to observe, and it's connected with much less risk to the attacker. So if we look at a standard compute environment, there are areas, the virtual server, the guest, for example, where you're running your application, that can be attacked with online attacks and everything. But most of the attacks points are actually offline attacks, being in the hypervisor environment, being on the storage area network or the storage server. Well, how do we protect against online attack? Well, Linux has quite a lot of good means to do that. It's all about access control. There are the authentication mechanisms, passwords, multi-factor authorization, SSH keys. If possible, avoid root logins, and you should set up your firewall. That is business as usual. On the other hand, authorization. Well, there's a good old Unix read-write execute for you, the groups, and the world until the very elaborate things like Linux security modules that allow you to control the authorization at a very fine, granular level. And last but not least, don't forget auditing all privileged operations because if someone intrudes your system, then that's your only choice to find out what actually happened. Sorry, that was the wrong button. Well, and with access control, you see we can protect some of the tech points in our system. Now, what do we do about offline attacks? Well, access control doesn't really work. Well, there is access control possibly for the storage. So maybe some level of access control to the sand. But in a hybrid system, you might not trust that access control. You don't typically do have it under control. So the best thing you can do is to encrypt your data. And cryptography is really great. It can protect your data, proving provenance, integrity, and data confidentiality. And hopefully, if you remember one thing from your Crypto 101 course, it's Korkoff's principle. Modern cryptography depends on algorithms being open, well known, well scrutinized by the whole research community. And the only thing that is to be secret is cryptography keys. And well, that means when you protect your data using cryptography, you must protect your keys. They become the most critical pieces of your data. And you must somehow protect them from not getting stolen, from not getting a success somewhere. But we get to that later. Now, the picture looks like this. We do end-to-end encryption of data, meaning we encrypt our data that we want to send to a disk inside our application, or at least inside our operating system. And then the encrypted data is sent all the way through whatever storage area networks to the storage server. It's encrypted there. And whenever we use it, we read the encrypted data and only decrypt it inside our server. That is what we call end-to-end data encryption. And it means that actually, together with access control, all our attack points are closed, secured against. Well, the most popular way in Linux to implement end-to-end data protection for your data address, for your data disks, is the DM crypt. And DM crypt is actually based on a kernel block device driver, a logical kernel block device driver. Let's have a look at the details. So it's the very bottom of the file system stack. Sorry, I should take a step. We have the physical block device driver that actually talks to the to the hardware. And on the very top, we have the application that issues our file system commands via the virtual file system. The virtual file system calls the specific file systems. Say, for example, X4, the request may go through the page cache or may not go through the page cache, depending on how you opened your file. And then it will reach the block device. The block device in Linux can be structured into multiple layers of logical block devices. Logical block devices are typically used to implement logical volumes, rates, multi-path volumes. And one of the possible layers is actually DM crypt. That is an encryption layer, which when reading data from a layer below decrypts the red data and puts the encrypted data to the layer above. And when writing data from a layer above, it encrypts the data and pushes it down to the layer below it. So if you look at the picture, all data in the stack that is below the DM crypt layer is encrypted. All layer above the DM crypt layer is in plain text. Now, for those not using DM crypt on a daily basis or configuring DM crypt on a daily basis, I hope most of you use it more or less implicitly if you are having a Linux desktop. Here is what has to happen. If you want to encrypt a block device, and I'm here on the left column with labeled plain format, then you have to provide the information on how to encrypt the disk, which is a key, it's a cypher, and a few more parameters that you want to have. What the kernel then constructs is a logical block device structure that is a device node located in DevMapper. And that can be from which a file system can be mounted. That block device in DevMapper already provides you with plain text data, whereas the physical disk would only provide you with encrypted data. So problems with that is at every time you open the device, you have to provide the key material and the cyphers. You must not forget which key belongs to which disk. And somehow you must find a place to securely store your key. Some improvements to that situation have been done with the introduction of the Lux or Lux2 formats, in which case all the cypher information and the keys are stored in a header of the physical disk. And since that information includes the keys and the keys must not be disclosed, the keys are then protected by a passphrase, actually by a key derived from a passphrase. Now, at open time, all you have to provide is the passphrase of the devices that you want to open. And then tooling like Cryptsetup will automatically read the header, extract the keys and the cryptographic material, and actually open the device. That has reduced the problem a little. Now we know which keys belongs to which volume, because each volume contains its own key. But we still must remember which passphrase to use with which volume. And we must find a place to securely store the passphrase. Sorry. OK, we have keys or passphrases that protect our data. Now, how do we store these things? We can now securely store our data almost anywhere, provided we find a place where to store our keys. We can store the keys or the passphrases in a file. Then we have to encrypt that file better. Where do we put the encryption key for that file? Well, you see, we are running in an endless loop or some kind of catch-22. We can put the key into a server, into a key server. Then we have to authenticate against the key server. How does authentication works? Well, the one who wants to authenticate needs to know a secret. And he has to place the secret somewhere, catch-22 again. So there is one solution to it that works for PCs or laptops. You can enter the password or passphrase interactively. And hopefully, nobody watches you while doing so. So it's a good way to do things on a laptop, on a personal computer. It's not a good thing to do on a server or even on a server farm if you want to operate this automatically, because then you do not want to always interactively enter passwords. Then there are additional problems. How to protect keys from being stolen from my main memory? How do we avoid creating dumps that contain keys? So quite a few questions to which there is a technical solution. The technical solution is called, and now we come to the term in the title, a hardware security module, an HSM for short. HSMs are special crypto hardware devices. They may be plugged in. Sometimes they are network attached. They are special crypto hardware. Typically, they are tempo-proof or tempo-resistant, and they contain at least one un-extractable secret. Typically, this is a master key, this secret. And this master key can be used to wrap other keys. And these other keys, called secure keys, can be used inside an operating system. They can be placed inside an operating system. They can be stored anywhere, because nobody can do anything with this key unless he has access to the very hardware security module. And in order to use those keys, you actually have to send, say, your message you want to encrypt or sign together with the secure key to the HSM. The secure key, the HSM will then interpret or unwrap the secure key, use it for the cryptographic operation, and give you the result back. So it's a very secure scheme. So once you have an HSM, it's very secure. Many HSMs are certified according to the standard FIPS 140-2 with a level 3 or higher. And the advantage is, well, you can more or less carelessly store your keys wherever you want, provided you don't give anybody your HSM. On the downside, well, you need next device for this, which comes in with a cost and which must, to some extent, be managed. We will see it one management operation later. And for fast ciphers like AES, typically all symmetric ciphers, HSMs tend to be slow, because they're required to do an IO. If we go back one slide, you'll see this is a card, it's an IO card. So we have to go through an IO bus or even through a network to actually reach that card. And that is quite a bit of overhead for a fast operation like AES, in particular, if it's hardware accelerated, which is the case on many modern processors. Well, on the IBM Z and Linux 1 platform, we have a similar function like the HSM function inside our firmware. So for each virtual server and the lifetime of each virtual server, the firmware generates a master key that is hidden in the firmware and that is not accessible to the operating system, neither to the hypervisor nor to any guest if you're running a virtualized system. And keys wrapped by this master key are called protected keys. Protected keys can be used like secure keys before, only that our CPUs, that the IBM CPUs, understand how to use protected keys. So the protected keys can be used to perform operations in the CPU using hardware acceleration for cryptographic operations. And therefore, protected keys are very fast. They come with a downside. They have only a short lifetime, so they do not survive a reboot of a machine. And therefore, they're not worth storing anywhere. So what do we do about that? Well, luckily, at least for the IBM platform, the HSMs can cooperate with the firmware and it's possible to generate a secure key on an HSM. Secure keys live very long, as long as the HSM lives, the hardware lives. And then whenever you boot your system, you can transform your secure key into a protected key. That is something that our crypto adapters and our firmware does under the hood. And once you have your protected keys, you can use it until as long as your Linux system is up and running. This operation is actually implemented in a kernel module, the so-called P-key module, that allows you to support different variants of secure keys. It generates random secure keys and random protected keys. And it can also transform secure keys into protected keys. Next, we have added to the Linux kernel a module called PAS, F390, that implements a new cipher. But that's the least thing a cryptographer should do, invent a new cipher, and we didn't. PAS is actually implementing AES. The difference to the AES cipher in the kernel is that it uses a different data structure to represent keys. Namely, it uses the secure key data structure and not the standard clear key string that the AES module in the kernel uses. So how does it work? Whenever you initialize a cipher, for whatever reason, these ciphers representations are called transforms in the kernel crypto framework. Then you provide it with a secure key and at initialization time, the secure key is transformed using the P-key module into a protected key object. And from then on, that cipher instance uses for its encryption operations and decryption operations, the protected key object and calls to the CPU directly. The PAS module supports different secure key object types for different HSM modes that we have. And it supports a list of modes of operations like ECB, CBC, the counter mode and the XTSAS mode. Well, a short note on secure key objects and secure key concepts. Secure keys are typically not only encrypted keys but the secure keys typically are connected to additional attributes. An HSM typically restricts what a key is used for, whether it can be used for key wrapping or making or normal encryption. And almost all secure keys I know come with a specific attribute that I call a master key verification pattern. A master key verification pattern may be a hash of a key, it may be the result of encrypting a zero block or whatever, it's an almost unique reference of that key without disclosing actually the key value. And it's used by HSMs to detect whether a key object is actually valid for that HSM or not. Same thing exists for protected keys. Okay, we were talking about PAS, the current framework allows for being FIPS certified and the FIPS 100-2 standard actually requires that before using a crypto module at power up, it must perform and succeed a self test. And that is one of most likely the key reason why the kernel crypto subsystem supports self test in the kernel, which is a large ever growing set of tests and they cover lots of corner cases so they're very intensive. And if you start the kernel in FIPS mode, that FIPS mode triggers the self tests on all the registered cipher mechanisms. And once you if a cipher doesn't have a self test and if a self test even if it has a self test and it fails, then the kernel will even panic. Now for PAS, we have added self tests and while the maintainers of the kernel crypto insisted that we reuse the existing self tests for PAS module and therefore we had to provide a way to transform the self tests that use a plain text keys as input to transform this plain text keys into protected keys while running the self test and therefore we use one of three different modules to import a plain text key into either an HSM or convert it directly into a protected keys. Which of the three methods is used depends on the hardware that is available, the HSM type that is available and whether you have configured your system to support the import of plain text into a protected keys directly or not. Anyway, our P key module will find out which of the three modes are available and then do the according transformation. If none of the three methods is available, then typically the system isn't in a state that you can use a protected keys anyway in a reasonable way manner and the kernel would be panic, would panic. Now with that, DM crypt can use the PAS cipher. What you have to do is just to provide the PAS cipher as a cipher argument and you have to provide a key object that together with the key object size that is large enough to contain a secure key and then DM crypt with PAS works out of the box. Provided of course the PAS module is loaded. A few comments, it's worthwhile on our platform at least and with PAS to use 4K sectors with DM crypt because that gives you some performance boost and you must be careful with the IV generation methods. There are, well, Lux recommends, for example, to use the SIV IV generation method for AS with CVC that unfortunately relies on using the same key that is used for volume encrypted and that doesn't work. So you must not use a IV generation method that depends on the volume encryption key. Luckily, the most popular and most recommended cipher to do disk encryption with this AS XTS with plain 64 doesn't have that limit. Well, I say things work out of the box that is true for the plain DM crypt volumes and remember for the plain DM crypt volumes we had that problem that we somehow need to remember which key is associated with which disk otherwise we would get some unexpected results and therefore we developed a little tool to maintain a key repository. The tool is called Z key and therefore we call the repository also the Z key repository. Well, it's open source, of course, and I'm done MIT license and it allows you to generate DM crypt volume encryption keys and store these keys with the, with the data of the associated volumes. So here's a little example of key generation call. The different options provide you with all the parameters you would typically put into the crypt setup open call or into a lux setup call and you certainly can list of the keys that you have and you will see all the attributes that you have provided for all keys you can also validate whether the HSM required to interpret the key is available and actually you can, out of the data in the key repository you can generate entries for ATC Cryptop and you can generate crypt setup commands that will actually open the volume with the according keys. Here's a short summary of all the functions that Z key features, generating keys, listing keys, renaming keys, removing, copying keys, changing attributes, of course, exporting a key to a file or importing a key from a file. Then there are some DM crypt related functions like generating crypt hub entries and crypt setup commands and HSM related functions. So we have support for different secure key types or different HSM types. We have support for multiple HSMs per key. So if you want to configure your system to have say two HSMs such that you have a backup HSM then Z key will automatically check the consistency of these HSM checking whether both HSMs are configured with the same master key because otherwise you might run into trouble. Well, some of the secure key types can be converted into another key type. We have the validation function that I just talked about and there's a re-encipher function that is a function that you need whenever the master keys on the HSMs get changed and that is something I will talk more about in a few minutes. So how does the PAS cipher and next Lux2 work? Well, it didn't work out of the box with Lux. We had to wait for Lux2 and actually for not the initial version of Lux2 but a later version of Lux2. Currently I would say it's safe to use like Cryptsetup version 2.1 and what the Cryptsetup team did for us is first of all it separated the cipher that is used to encrypt the blocks on the volume from the cipher that is used to encrypt the key slots. You remember in Lux the key is stored in the header and that key is encrypted by another key derived from the PAS phrase and that letter encryption is now different from the encryption used to encrypt the blocks of the volume. Then there is something called an unbound key slot that are key slots which may contain keys that are currently not valid for usage. Then there is support for something called Lux2 tokens. These are spaces inside the Lux header where you can put some specific piece of data and they enabled the use of DM Crypt with 4K block sizes which helped us to improve the performance. With that you would store secure keys read by the PAS phrase generated key in the slot which means the effective volume encryption key is read wise once by the HSM master key and a second time by the key derived from the PAS phrase. However, the PAS phrase in that case need not be secret. It's not relevant for the security because it's a master key that does the actual securing of the key. It's one of the few cases where CryptoFort tells you allows you to use the password 1-3. And write it wherever you want. There is one little catch. Cryptsetup doesn't know how to generate random PAS keys or random secure keys. To do that, you have to do that. You do that outside of Cryptsetup. For example, using the Z key tool and then you provide the secure key using the minus-minus-master-g-file option. Even so, Lux does the association of keys to volumes. There are a few goodies the key provides for Lux2 volumes so you can provide each key entry with the DM Crypt volume type, either plain or Lux2. And for keys that have this Lux2 flag, you can generate Lux2 Cryptsetup entries like in the example here. And you can generate Cryptsetup commands to format Lux2 volumes. Yes, Cryptsetup Lux format with the type minus-minus-2. Well, here's the same picture as before. The only difference being that now instead of a secure key instead of a plain text key we have to provide a secure key and instead of AES, we write the string PAS. Same when doing the Lux2 format and that resolves the problems we have. We now have using Z key, we can associate which volume belongs to which keys or vice versa and using secure keys we do not have a problem storing our keys anywhere. With the Lux2 format, we still can have to remember which password we used but we do not have to hide the password somewhere and we can use something that is easily revembrable like for example 1-3 because the password no longer is essential to provide the security of the keys. Well, now comes the hard stuff. An HSM has a master key and maybe every now and then you have to change the master key. These are rare operations from the customers we have they don't like them too much but sometimes there are regulations that require them to change master keys and that is likely in the order of the magnitude of at most once a year or every two years or maybe even less often. But what happens if the master key of an HSM changes? Well, all secure keys become invalid and therefore you have to do something you have to transform your secure keys that have been wrapped with the old master key into secure keys that have been wrapped with a new master key. On the left hand side of this slide, sorry on the right hand side of this slide you see a scheme that shows you how this can be done on an HSM. So HSMs have registers for the master key the current register contains current MKE register contains the currently valid and live master key and there's another master key that may hold a new another register that may hold a new master key. So assume the current master key is the yellow one then all secure keys that are yellow wrapped can be used. In order to change the master key it's a two-stage process. First you load the new master key say a violet one at this point in time you cannot use any violently wrapped key any key that is wrapped by the violet new master key. The only thing you can do with the new master key is you can re-encipher or re-wrap the old secure keys such that they become wrapped with the new master key. Once that is done you actually set the new master key making the new master key the current master key. At this point in time you're only the violet the keys wrapped by the violet master key are usable and the keys wrapped by the yellow master key are no longer usable. Well this is one scheme there's a few variations of that scheme in the market but I think that is the essence on how master key change works. Some HSMs allow you to recover from one master key change but then they wouldn't allow you to recover from two master key changes in the world. Now what do we do with CDM Crypt if master key is changed? Well first on our Z key repository we have two commands one for each phase that allow you to re-encipher the keys stored in the repository that are associated with a specific HSM so first you do the re-wrapping the re-enciphering and once you have actually set the new master key on your HSM for which you typically have a different control for our platform you actually have to go sometimes to a different workstation that is in the actual HSM console and after that is done you have to complete the step and do the re-encipherment replacing the secure keys that worked with the old master key by the ones that now work with the new master key. Then you're done with the re-enciphering operation if you're using the plane mode in this case. If you decided to use Lux too then of course you do not only have keys in your Z key repository but the keys are in your volumes and you have to replace and re-encipher the secure keys with headers of your volumes and that has to be done for every volume and don't forget the one volume that currently is offline to do that otherwise you might have a problem. So what do we do? We have written a new tool called zkey-setup first we recommend that you use the data that is written in the Lux2 token of the current key then you have a re-encipher operation that writes a re-enciphered key into an unbound slot and only if the master key has been set and is now valid you use the re-encipher complete function to actually copy the newly re-enciphered master key from the unbound key slot into the actual key slot and then you can actually use the new master key on the HSM. Note on the system since once the volumes have been opened we are using only the protected keys and not the secure keys this can be done while the volumes are being used without interrupting any I.O. operation on the volumes. Okay, let's come to an end. I've told you how we made an HSM usable for dmcrypt and how we integrated this into cryptsetup and even made it usable with Lux a good question to ask us can other projects benefit from a similar way to treat Cyphus for HSMs and clearly candidates would be filesystem encryption, the dmcrypt integrity checking and possibly the IPsec if you somehow use pshar keys. Note the key negotiation is typically done outside the kernel and not supported by all HSMs. What is required for a project to have a chance to deal with things secure keys like in the PIS case first of all Cyphus must be configurable preferably by the configuration should be doable in the kernel using the Cypher names like dmcrypt does well we need to be able to provide key objects and specify their size and that should be the object size not the security strengths and note there might be unusual sizes of secure key objects sizes greater than 32 bytes which is the maximum that we have today for plain text keys and last but not least a key generation must be doable from outside the module or the according module must have a specific plug in for HSMs, sorry hitting on the wrong button. As for the key development is not complete yet we are always working on usability enhancements for example currently we do not have a passphrase attribute that is something we want to add one of the next things we want to add and if you have an idea what might be missing we are keen to learn your input clearly it would be interesting to integrate Z key with external key servers and provide a good authentication yes an HSM based authentication to these key servers and maybe even Z key can be used for other platforms Z key itself one idea I loosely have is providing a pkcs11 interface and allow keys or passphrases to be wrapped by pkcs11 secure keys clearly in that case they would be unwrapped into a clear key and then the clear key would be stored into the corner but at least you would have a chance to store the keys in a secure way and only unwrap them if you have access to the HSM for which might have a pkcs11 interface yes and with that I want to conclude my presentation let's summarize that today's Lincoln's Kernel and DMCrip tooling allow to use HSM with HSM protected keys to encrypt volumes I've shown you the example that we implemented for our platform this allows you then to store DMCrip keys on unprotected media and at least the initial DMCrip key must somehow be stored on unprotected media you cannot currently protect the boot partition and that's where you would put the key or the passphrase for the root partition and with a secure key or protected key you never ever disclose the plain text of the volume keys not even during the operations of the system and with that I'll open it to questions let me check through the Q&A so question number one I want to store a large amount of data with Redunas and not sure what it is is that an HSM and encryption volume manager, LVM and encryption at the same time can I get a flexibility of LVM without having to unlock every physical volume when the system boots I'm not 100% what you mean clearly you can combine LVM and DMCrip that works we actually suggest encrypting the LVM physical volumes because then you can use PVMove to transparently set up encryption and change the encryption keys but without ever opening the volumes for DMCript I don't see a chance you can alternatively encrypt the upper layers the logical volume layer that is possible too but then you cannot use PVMove trick to re-encrypt your data let's see for question two SED, FIPS-enabled SSDs and hard disks play any part in this secure ecosystem at all not really unless you want to double encrypt end-to-end encryption and the assumption is you do not know where your disk is it might be outside of your system you can transport between the operating system and the disk storage if you have this storage that encrypts itself then the pass between the operating system and the disk is jeopardized and if it's a sand it's pretty open and again the key used to encrypt the storage must be maintained somewhere and it's not really under your control with SSDs it's hopefully put somewhere in a place piece of hardware but with an external storage server you just don't know so okay is a different protected key generated each time the HSN sends one to the CPU or is it always the same on our platform the protected keys are generated by the firmware whenever the system is booted what you have to do is you generate a secure key and transform it into a protected key therefore the secure key is sent to the HSM the HSM negotiates with the firmware of the current virtual server how to re-encrypt the secure key into a protected key and the protected key is then valid for the lifetime of the virtual server meaning until you boot the virtual server or until you actually you could also migrate the virtual server to another system then of course a new protected key is required and PS knows to deal with these situations it will then try to re-encrypt the secure key that it still has next question how is access to PIS keys managed in a multi-user multi-container system okay PIS protected keys on our platform are unique to virtual servers so different containers running in the same virtual server would share the same master key of the protection key so there would not be key isolation between two different containers running in the same virtual servers unless you use constructs like kata where each container runs in its own virtual server in that case each kata container has a has protected keys that are not usable by another kata container multiple users for multiple users it's there is no separation dmcrypt is a scheme that works on system level and is managed by root and I think that is sufficient I have a slide on that on my opinion on that in the appendix if you download the slide you find why I think root level system level encryption by root is sufficient at least in non-cluster system let me scroll down what is the sand oh sorry storage area network typically a fiber channel connection can you recommend any low cost hsm cards I'm afraid I cannot can pkc as 11 keys be used for other devices over the air to perform secure boot a secure boot I'm afraid you do not mean the boot integrity feature secure boot here but using a boot disk or a root disk that is encrypted not easily what you could do is have somehow your passphrase encrypted by another key that is another hsm key and that hsm key could have a pkc as 11 interface pkc as 11 is actually an api description that describes how you can talk to an hsm it's not a format description of keys the actually the protected key pis and peaky modules were introduced before kernel 5.6 yes I learned that we are close to a hard stop what was introduced in kernel 5.6 was only the self test and I'm afraid we are here close to a hard stop I hope I get access to the I uploaded the slides so you should have access to them and I hope I even could answer all your questions so thank you for attending this session and you see my email address on the slides in my bio so if you have further questions feel free to ask me bye bye and have a nice day have a nice evening depending on where on the world you sit