 Good afternoon, everyone. I'm Russ Coney, Product Manager in the Real Business Unit working with emerging technologies. I'm Javier Martinez, software engineer working on the bootloader team. So if your machine fails to boot, probably I am someone to blame. And what we'd like to do today is talk about a hardware road of trust and specifically about doing useful things with the hardware road of trust. This is our second year. We're back. And last year we were talking about technology. We were talking about the neat things that can be done with some of the new technology. And we were basically here with the message, it's real this time, trust is. Well, we're back. It's real this time and we've got some solid things to show. Last year we were doing technology preview of the trusted platform module in Fedora. We were showing a preview of the network-bound disk encryption using the hardware road of trust again in Fedora and talking about wonderful future. So we've heard talks like this before. So what's happened since then? Since the last F-Comp, network-bound disk encryption with TPM 2.0 support has been shipping in Fedora in Rails 7.6. And will be included in the future version of Rails that cannot be named yet. Some of the recent developments are we have added measurement of the boot path to the system. And we'll be talking about that both why you care what it is and how it works. We actually have customers using this. There are customers using the capabilities of the TPM 2.0 to protect secrets, variety of secrets, notably keys of different parts. But there are also some people even using the advanced ceiling capabilities. One of the traditional challenges with using the hardware road of trust has been that they've been very difficult to program and you have no idea of where to get started. So we're going to help you with that. We're going to be talking about signing with TPM 2.0 and these slides are falling off the bottom so we're missing some of the other really important points. So maybe we've been changing. So as he's updating the resolution, a little bit of background would be useful. TPM, the trusted platform module, has been around for about 12 or 15 years. There are two versions of it. The initial broadly basal is TPM 1.2 of which billions are shipped, hundreds are used. So there's some excellent capabilities in TPM 1.2, but they're almost theoretical because they are so bloody difficult to use. TPM 2.0 builds on the foundation of TPM 1.2.0. It provides additional capabilities, notably crypto agility, but it also has a set of characteristics that make it dramatically easier to use and dramatically more useful and we'll be talking about those in here. As part of TPM 2.0, there are two user space libraries that have emerged. The early work for user space library to work with TPM 2.0 was done by IBM and was a IBM development. The trusted computing group, TCG, the developers of TPM specifications, including 1.2 and 2.0, have developed their own user space specification, which we believe will be the standard user space interface for working with TPM 2.0 and this has been implemented. To give them credit, Intel has done the majority of the work and implemented the TCG 2.0 user space specification. We are shipping both the IBM TPM 2.0 user space and the TCG Intel TPM user space in Fedora and in Realm. However, our primary focus in developing tools and applications is using the TCG user interface. So in summary, the focus at Red Hat around doing useful things with TPM is based on TPM 2.0 hardware and API and the TCG user space. Another interesting note, since this may actually fit in with the theme of ease of use, is that there is a fairly widely used interface called PKCS11. So PKCS11 is intended to provide a platform independent interface to a variety of security tokens. So it's trying to provide a standardized interface that can be used across a range of devices. Basic capabilities would be implemented on smart cards, on hardware security modules. If you want to hear, have a UB key, UB keys and TPMs. It has a fairly basic set of capabilities that address the use case of creating, modifying, and deleting keys and tokens. This would include things like RSA keys, X5.509 certificates and so forth, and is very widely used by certificate authorities. So this is useful and there's some work underway to provide a PKCS11 interface to TPM. Now this would be to a subset, relatively small subset of TPM capabilities, but a very useful and important subset. Also note that the PKCS11 work is building on top of the TPM user space, the TCG user space stack. So it's not a question of whether to use the native TPM API or to use PKCS11, which is the best choice for the specific problem you're working with. An example of this is that you can have something, a user application, GNU TLS as an entirely non-random example. And the PKCS11 interface would allow you to use the same interface to talk to smart cards, to talk to the GNOME key ring daemon, and to talk to trusted platform modules. So it's one of the things that is being worked on to try to make TPM more useful. Now, if anyone is interested in actually developing software that uses TPM to, we have a cheat sheet for you. Traditionally, the most difficult space or place getting started with using these things is where to get started. What do I need to set up? What do I need to initialize? What do I need to interact with? What are the demands? How do I know if it's working? What do I do when I'm pounding my head against the brick wall, trying to get something that should be obvious to work and it's just not cooperating? We hear you and there is work that as a side effect is really benefiting people who need the capabilities of TPM and don't know quite where to get started. We're going to be talking about network bound disconcorrection and specifically the Clevis module. And it turns out that it's almost a side effect of the problems intended to solve. Clevis is a crypto framework with some very useful capabilities. It has considerable flexibility. We've seen more about that. And a number of the Clevis functions have been implemented to use TPM as well as the Tang server for the back end and other work is being explored for the future. So what this means is that the Clevis code base is a really good starting point for working with TPM. Clevis has gone in and implemented the most common functions that you would be interested in working with TPM. So there's code sitting there ready to be stolen. I mean there is a reference implementation that you could look at to determine exactly what is needed to get these things up and running. Now in addition to working directly with TPM, Clevis is a good reference point for working with other parts of crypto such as the Jose specification, the Java web encryption, Shamir secret sharing and we'll be showing an interface to UDISCs too. So there's quite a bit of good stuff in there. github.com slash latch set slash Clevis. Javier, what the hell is a TPM? Yes, as Russell said last year we talked in detail about the TPM to software stack, different layers and libraries that it compones. So this year we are not going to cover that but instead show some of the applications of TPM, some apps and projects that are using this software stack. But still for people that are not familiar with the trusted platform models, I'm going to briefly describe what are the building blocks of these models. So TPM too are crypto processors. So they have a crypto processing unit to create random numbers, to generate key for different ciphers always. And also they can do encryption and decryption by using the keys created by the TPM and also have stability to calculate hashes. It also has a lot of memory that can be used to store transient objects because the key created by the TPMs are not stored in the TPM but you have to load it. We are going to explain more in detail about that. So that memory is used to store those keys. And also it contains the platform configuration registers that are a set of registers that store hashes for things that have been measured. We are also going to talk a little bit more about PCRs later in the talk. And finally it has some number Latin memory that can be used to persist objects in a secure way. And this memory also contains the seeds that are the root of trash of the hierarchy. These seeds are created when you utilize the TPMs and all the primary keys are derivated from these seeds using a key derivation function. And this primary key never leaves the TPMs and are the root of trash. So what are the features that TPMs are? As we said it's able to generate random numbers, it can create keys and do encryption and decryption. It can also be used to identify the machines in a secure way. It can also store secrets in a secure way and it can be used to measure the integrity of the system and to attest its health. That has not been tampered with. TPMs come in different flavors. There are hardware chips that sit on an actual bus like SPI and S4C. There are also firmware based TPMs. For example the TPM in this machine that we are presenting is a firmware based TPM. It doesn't have a real TPM chip. As a side note, everyone that has a laptop that's shipped in the last 05 to 8 years, you do have TPM. If you do have a laptop that's shipped in the last 4 or 5 years, it's almost certain to be TPM 2.0. Part of the reason for this is that Microsoft requires TPM 2.0 for Windows 10 certification. So any system that is Windows 10 certified will include a TPM 2.0. You may need to turn this on in the BIOS, but laptops and other client systems are pretty much guaranteed to include TPM and server platforms increasingly. That's true. One of the features of the TPM is the ability to store secrets. You can encrypt and decrypt data using the keys created here. As we mentioned, the keys are not stored in the TPM, but are stored outside the TPM, but protected by a primary key that never leaves the TPM. That means that you can have as many keys as you want, which was different with the older TPM specification where the keys were stored in the TPM. All the crypt operations happen inside the TPM, so are protected by the hardware chip. Another feature of TPM is the ability to seal data. Sailing means that you encrypt something and specify a policy. The TPM is only allowed to decrypt the data if that policy is met. We are also going to talk about sealing more later. Something to keep in mind is that the TPMs are not crypto accelerators. In fact, they are very slow by design, so it should only be used to encrypt small secrets like keys, for example. This is an example of how to do encryption and decryption using the TPMs by using the TPM tools project. This is a project that is a set of command line tools to interact with the TPMs. The TPM tools project uses the TPM to software stack libraries. It is a good starting point to wrap your head around the programming model of the TPMs. When I started with the TPMs, I had to read a lot of specifications and it was very complex to understand how to operate. The TPM tools are useful for that. By looking at the interaction test it has, you probably can find the use case for a TPM that you are looking for. They are also a good reference code to understand how to use the TPM to libraries. In this example, we first create a primary key. Then we create a key that is going to be used to encrypt the data. Once the key has been created, the key has to be loaded into the TPM so the TPM can use it. Then we can have some plain text to encrypt and use the TPM to RSA encrypt command. That encrypt data in the TPM using the key that has been loaded. Then we can also decrypt the data by using the TPM to RSA decrypt command. All the key operations happen inside the TPM so the encrypted data is protected by the TPM and can only be decrypt in that particular machine. As you said, it's not hard to use the TPM, but still there are a lot of steps involved. There are many properties that have to be specified when you create the primary key and the key and so on. We try to do these a little easier to use. As Russell mentioned, we add support for TPMs to the Clevis project. The Clevis project is an automated decryption framework that contains several pins. Each one of these pins implements a decryption policy and also specifies a key store for the keys that are created by Clevis. Clevis also has a set of command line tools to decrypt and encrypt data using these pins. It also has the ability to bind a LUX volume to a pin so you can automatically unlock your LUX volume if a certain policy is met. It has several unlockers to do this. There is one for the drag-through model so you can unlock your LUX volume in the Internet fast. It also has a UD2 plugin and also a command line that you can use. The Clevis looks unlocked if you want to unlock a LUX volume explicitly. This is how Clevis works. You need to provide some data to encrypt a pin, for example, TPM, and a configuration. Clevis has a reasonable default but every pin has its own configuration if you want to tweak the parameters. Clevis takes that information and generates a JSON Web Key that is used to encrypt the data. The encrypted information and information on how to retrieve the key are stored in a JSON Web Encryption. Then the key is protected by a TPM. On the encryption, you provide to Clevis a JSON Web Encryption. Clevis takes the information about how to get the key from there. And also the cipher of the text and retrieves the key and decrypts the data. He is an example of how to use Clevis. It is a single command. You tell Clevis that you want to encrypt something which pin to use a configuration. In this case, it is default. It generates a JSON Web Encryption. If you want to encrypt, you provide to Clevis a JSON Web Encryption and eat the encrypts. I am going to show here. I encrypt something and then I can decrypt. Clevis returns the... As you can see, it is very easy to use Clevis. You can also bind a look volume. We said to pin. You just need to install the Clevis track code module. Then generate your IRFS. It contains the Clevis and the TPM's two libraries. You bind the look volume and if you reboot your machine, the TPM is present. It will unlock the look volume automatically. We showed now how to encrypt and encrypt data using TPM's. We mentioned that you can also seal information. Now, Rasmus is going to explain how sealing works and how the PCR's are used for that. This is something that we are just starting to really use. On the one hand, it has been around for a while. On the other hand, it has been used. A lot of that is because the implementation in TPM 1.2, for a variety of reasons, was very difficult to use. The implementation in TPM 2 is much easier to use. What is the measurement in sealing? Measurement is basically creating a hash of something. A hash of firmware, a hash of the configuration, a hash of the boot loader, a hash of the kernel. Something else. So far, pretty straightforward and good clean fun. The TPM has a set of platform configuration registers, or PCR's. You can take a hash and you can write it into a register. Okay, that seems kind of obvious. There are a couple of things here. You don't write it into a register. There is a TPM operation called extend. You provide the hash value and you extend that into a register. Now, when you come to the next hash value, you don't overwrite the register. The extend operation will take whatever is already in the register. Do is kind of a glorified concatenation and reduction. But it uses crypto operations to produce a new value. So, if you have, oh, let's take a random example. How about a kernel and five drivers that you care about? You can hash the kernel in each of those five drivers. And as you do it, hash, extend, hash, extend, hash, extend. You end up with a PCR containing a single value. That is the result of the specific measurements and the order that the measurements were taking it. The extension operation is done entirely inside the TPM. And this can be used as part of more complex operations called ceiling. So, the thing here is that this can be used in a number of rather interesting ways. It is dependent on the contents of the things that are being measured and the order that the measurements are taken. It is reproducible. If you have the same set of things being measured and you measure them in the same order, you will get the same results. This is interesting because you have things that are common. For example, you can have a measurement of a BIOS and that measurement will be valid for any installation of that BIOS. The BIOS maker can publish the measurements of that BIOS and this can be used to determine if the BIOS has been modified or corrupted. So it can be anything from a single file on the system to universal things like a BIOS. So, Tavir, do you want to go into the PCRs or do you want me to do that? No, you are going to do it. So, the interesting thing here is that there are set of 24 PCRs. Some of them are very low level like the BIOS or the BIOS configuration. Is anyone looking at PCR number two option ROMs and wondering what that is all about? So, do any of your systems have a disk controller or a network controller in them? So, what is happening is that we can measure all the firmware on the different devices in the system. One of the attack vectors is advanced persistent threats, root kits in the BIOS of peripherals. We have the ability to check the BIOS and the peripherals and determine if the peripherals themselves have been modified or corrupted. We can come down and we can do the boot loader, the master boot record. We can check, oh, state transitions and wake up events. We can measure the operating system and there are registers available to use so you can build the custom applications. Okay, this may sound vaguely interesting, but it also sounds like there would be a lot of work to implement it and a lot of opportunities to mess things up. A couple things. PCR zero through three are extended by the hardware in the TPM. The low level hardware and the TPM measure the key parts of the system. So, you now have a mechanism to determine if your BIOS configuration and device firmware has been modified or corrupted. Peter Jones has been implementing extensions to the shim and boot loader to add additional measurements such that the shim and boot loader are now providing measurements for the shim and the boot loader for the system certificates, kernel and grub commands. So, we now have as a standard part of booting the system a set of valuable and useful measurements, PCR measurements being done for you that are available to use. More information on it, github.com or hbootshim and you can get the details on that. So, let's tie these two together. We talked a moment ago about encrypting and decrypting secrets using the TPM. With ceiling, we can seal the secret operation with a set of PCR values, which means that the secret will only be decrypted if those PCR measurements match what was used. So, TABIR, this might be a useful place to show an example. Yeah. Previously, we showed how to encrypt and decrypt that I think clever, but you can also, and we use the default, can you see there or should I make it bigger? So, we use default configuration for the TPM to pin. This time, we will pass a PCR ID argument so we will say we want to encrypt but also we want to seal against the PCR number zero. So, we generate the JSON web encryption and we can decrypt this with our problem, but if we have a value in the PCR register zero with a hash. So, if we change that hash, for example with an extent operation, TPM won't be able to decrypt the secret anymore. So, let's use the extent command to extend some hash, for example of a test string, and then we have a different hash in the PCR zero. So, this time if we try to decrypt our secret, the TPM will not allow us to do that. So, Clevis wasn't able to get the JSON web key because the PCR policy that was used to encrypt wasn't met. So, measurement and PCRs are only one part of the puzzle, but we also need something that's called attestation and Russell is going to talk about that. Okay. So, have any of you taken a Linux system disk from one computer and moved it to another computer and run it? Or have you cloned a disk to get a disk that will run on another system? Well, if you're using Clevis with LUX, and particularly if you're ceiling, you can, you have set it up such that the disk will not boot on anything, but the system it was originally configured on. And that it will not boot if someone has messed with the BIOS or has turned off secure boot or has done a number of other things. So, this provides a very strong capability for locking down systems. So, in some case here is basically we just showed seal and then check it. So, good starting point. It does have some limits in its flexibility. The next thing to do is to begin providing comparison of the PCR measurements outside the direct ceiling. This can be done either by having a list of good values on the system. So, basically BIOS manufacturer would provide a list of the PCR values for different BIOS versions. So, you would be able to upgrade the BIOS without breaking your system. Useful. If the list is kept on the local system, this is known as static attestation. But it also, there is a broader one, which is take the various sealed PCR values, make them available outside the system and allow a external server to determine if this set of PCR measurements is valid or if the system has been compromised. So, there are several systems that either exist or are under development for implementing dynamic attestation. Some examples of this would include the KeyLine project, the Intel OpenCIT system, and the StrongSunk One package has some fairly extensive capabilities here. So, there are a number of things that you can do directly with PCRs and PCR ceiling. There are also some systems that begin to build on this. Okay. Signing. Everyone here knows what signing is. It took me a little while to figure it out. I'm somewhat slow. The people that know me can well assure you of that. So, signing. Signing means you take a hash of the data, you encrypt it with the private key, and then with a public key, you can verify that the data was signed with a defined private key. And that, since it's the hash, that the data in the file has not been corrupted or modified. So, good stuff. We use Signing all over the place. It's really the foundation of much computer security. So, the thing that you can begin to do with TPM is you can do the Signing operation in hardware. So, the Signing operation is entirely inside the TPM, inside the security processor boundaries, and can only be done by that one specific TPM. So, TPM Signing gives you the ability to prove that a file was signed by a specific system and that you do not have the situation of someone being able to copy your private key off someplace else and start signing things. Now, traditionally this has been somewhat challenging to do using TPM. Javier, has that changed? Yeah, now, here we have an example of how to do Signing. Again, you've seen the TPM tools. So, you create a primary key, like in the previous example, then you create a key. You load the key. That's the same as the last example. And then there is a TPM tool sign command that can use that loaded key to sign the hash of some data. And that gives us a signature that contains the signed hash. And since only the private portion of the key is protected by the TPM because it's wrapped by the primary key, the public key can be loaded in any TPM. So, another user can use the TPM to load external tool that allows to load keys that were not created by that particular TPM. So, can load the public portion of the key and use the TPM to verify signature to check that the hash for that data and the signature matches. So, we have some demos of interesting upstream projects that are using the TPM to software stack. We already showed some examples in Clevis, but we have another one. And also, there are two projects that are under development. They don't have a release yet, but we, on our test, are working pretty well, even when we don't have a release yet. One is the TPM2 TSS engine that is an open SL engine for TPM2. And the other one is the TPM2 PKCS11 project that provides a PKCS11 interface for TPM2. So, the demos we have for Clevis is to automatically unlock a LUX volume using the UDIS-2 support. For the open SL engine, we have a demo to show how a private key of a cert can be protected by a TPM. And finally, for the PKCS11, we will show how an open SL private key can be protected by the TPM. So, as we mentioned, Clevis allows to bind a LUX volume to a PIN. So, in this example, we have an external USB drive that will be unlocked automatically by Clevis. So, I'm going to plug in my computer. And even when it has a LUX volume on it, it will be unlocked automatically because the Clevis Unlocker was able to. So, we see that it has been unlocked and mounted in my system. And I didn't have to provide any passphrase for that. So, this is, for example, useful for backup solution if you don't want to enter your passphrase every time you want to copy something. But if someone steals your disk, they won't be able to automatically unlock the LUX volume. Only in your system because it's protected by a key that has to be loaded into your TPM. So, as Russell mentioned, you can also seal your LUX key using a PCR policy. So, only, for example, only if my secure boot configuration is the same or if my firmware didn't change, unlock automatically. Otherwise, ask for a passphrase. So, I can unlock. So, to be back up for one second, this specific example, it is going to be checking to see if the system firmware is unmodified, if the option ROMs or the device firmware is unmodified, and if the system is in secure boot mode. So, a lot of things that you can check, this would be a simple yet very useful capability that would give you a strong indication that you're booting a secure unmodified system. So, the next example is to protect a private key of a certificate using the TPM. So, the TPM, the OpenSL engine for TPM has this TPM2 TSS GenKey that generates RSA key using the TPM. So, then you can use OpenSL and specify that you are using the TPM2 TSS engine to generate a certificate signing request and a certificate and so on. And then you can, for example, use the OpenSL server tool to start. We only have five minutes, so I'm going to do this very quickly. So, here we have a private key that is a PEM key with a label TSS. So, OpenSL knows that it has to use the TPM2 engine for that. And we will start the server. I'm a client here. And we see that they are able to establish a TLS connection. The client is using the certificate that is associated with that private key. So, if someone steals your private key, the attacker cannot do, for example, a money in the middle attack. Because that private key can only be used in that particular machine. The last example is to protect SSH private key. So, using the PKCS11 interface, you can generate a public key associated with your private key and you copy it to the server. And then you tell SSH, since SSH has support for PKCS11, you specify that you want to use the LEP TPM2 PKCS11. And, see, I can establish, I can authenticate using that key. And you can see here that the client offers a PKCS11 public key and the server accepted key so you can authenticate using your key that is protected by the TPM. So, again, if someone steals your private key, the attacker won't be able to access that machine because it can only be used by loading into your TPM in that machine. So, it's another level of protection for your secrets. So, yeah, we show some examples how to use TPMs and what these modes are useful. And now Russell is going to talk a little bit about other use cases. Actually, this is almost a plea because I think we've demonstrated that there are some useful and powerful capabilities with a hardware root of trust today that we're beginning to do useful things with this hardware root of trust and that it is actually quite easy to use. It's more dramatically easier to use than it's been in the past and we've been demonstrating with even these early efforts that we can provide significant levels of hardware backed security that are transparent to the user. Transparent to the user. Hallelujah. Maybe people will actually start using this stuff. But as I said, we're just scratching the surface. These are some things that we're looking at for potential uses of TPM. I put this list together a few weeks ago so Javier has already moved beyond me. I need to take some of these things off the potential list. But there are a lot of other things out there. So if the hardware root of trust backed security would be useful for any of the projects that you're working on, we encourage you to use it. Any and all the upstream projects building on TPM can of course use additional assistance. We would love to have people working with it. And if you start getting into these things, there are a set of resources. A really good starting point if you want to know the details of this is a Practical Guides TPM 2.0 250 page book which is available as a free electronic book from A-Press. There's more information available from the Trusted Computing Group. The user space tools you have listed for the IBM and for the Trusted Computing Groups. Clevis and the work that Javier and the other engineers are doing with network bound disk encryption. And the work going on with OpenSSL and PKCS11. And by incredible coincidence, as the out of time sign came up, we got to the point of, do you have any questions, comment or feedback? Entertaining? Informatics. Regarding the support of TPM in GPG and how maybe is there a possibility to use it with the GPG out of the box? Have you looked into this? No, I don't know if there are some efforts in that. I have to check. So with this, would you be looking at some questions like, hardware read trust support in PGP? No, I would like to check my pilot key, pilot GPG key and... Oh, absolutely. With the simple instructions we have here, we've shown you exactly what you need to do to take a secret such as a GPG private key and store it into the TPM with or without ceiling. So you can do that today. I do not think it will work with GPG. Yeah, I do not think it will work with GPG. GPG doesn't support any OpenSSL engines. It doesn't support any PKCS11 modules. It's just using their thing. So the two things here are, you can use TPM outside of PGP. The simplest case would be with a script. And this is where you should point out that that's an ugly and incomplete answer. And I would agree with you completely and say that a GPG implementation such as PGP can be extended to use this capability. And I have to admit that that work has not been done. Yeah, I don't know if there are efforts already to do that. Because what you said, if you encrypt the key using the TPM, you will store the key at some point unencrypted in your file system. So we need proper support in GPG for that. Good idea. Is it common to have support for TPM on a system without secure boot? Is it related? No, it's not related. You have secure boot and trusted boot. There are separate things. You can connect it for example by using measurement. You can say, I'm going to seal this data if my secure boot configuration didn't change. For example, if a secure boot wasn't disabled or the keys replaced it, for example. But there are separate things because secure boot is active. If your binary is not signed by one of the keys in your of trust, that binary won't be executed. But TPMs only measure the good components and it doesn't take any action for you. You can use the information in the PCR state to say for example, I want to encrypt this. But the TPMs don't do nothing by themselves. TPM in this case extends secure boot. And there are a couple of obvious attack vectors on secure boot. One of them is turning secure boot off. That's kind of hard to defend against. The other is to add new signing keys to the mock list. So both of those would allow you to boot with an un-signed per. What you can do with TPM is that you can seal it both to check that secure boot is enabled. And you can also seal the signing keys to make sure that no one has added a key to the approved signing key list. So it's able to extend secure boot. But it's almost to turn secure boot off. For example, in power platforms you have trusted boot because you have the TPM there. But you don't have secure boot. That's not dependency between you. I think last question because we run out of time. How do I prepare for TPM where TPM will do the dice and then you have to recover data? The question is, how do I prepare for my TPM module dyne for my system that operates the TPM module dyne? TPM is not a backup. So you need to make sure that your data is backed up outside the system. And one of the things is you have a mechanism and approach for placing data in the TPM initially. You would need to be able to create that data. So if you have keys only in the TPM, that is a risk of hardware failure. And since we're getting the hook over here, thank you very much for attending this session. Hoppy or not, we'll be just on the other side of the board and delighted to continue this conversation for as many hours as you can. Thank you.