 So, thank you for coming in. I'm Nikos Moryanopoulos and I'm going to present a state of cryptography in Fedora. Who am I? I work for Red Hat, for the Red Hat Crypto team. We supervise a lot of components which relate to crypto, both in Red Hat Enterprise Linux and Fedora. Myself, I contribute to open source software, to free software and Fedora, of course. So, let me move on. What this talk will be about? It will go through the major initiatives in Fedora cryptographic support. It will also go through briefly a few important future directions. And why is it useful to go through all these major initiatives? It's because we can learn from our mistakes. There have been mistakes down there. And we can also see our current status and current direction that we have taken. So, let me start from the beginning. And the beginning is the early days of Fedora and Red Hat itself. I split it into two major periods. The early days is a period where community and Fedora and Red Hat on one side were on different, were separate. On one side was the community doing one thing. On the other side was Red Hat and Fedora doing a different thing. And how did we end up there? Let me start telling a story. So, Red Hat was entering the business world at the time. We were the outsider. We wanted to start into the financial sector into highly regulated industries. We were looking at certifications like common criteria, like FIPS, and we wanted to get there to make open source software pass all the certifications that commercial software at the time had, proprietary software at the time had. So, in a way, we managed. Open source entered highly regulated sectors, demanding sectors, the financial sector quite successfully. And the questions that are quite today, everything open source is very widespread. And how did we manage this? Through a project called the Fedora consolidation, crypto consolidation project. I don't know how many of you are familiar with that project. If you have participated with it, may I ask how many? One, two, four. Okay, I will explain a little more about this project. It goes about a project to consolidate all the crypto to a single one because we saw there were many crypto libraries, crypto backends in open source. Maybe way too many at the time. We thought at the time. And it made sense to consolidate them in one so that we can easier certify it. Certification was a very expensive process and Red Hat was very small at the time. We wanted to consolidate it in one. But anyway, we wanted to change the crypto landscape with bulldozer. That's why I selected the bulldozer there. We had one bulldozer, so we didn't succeed. So what was happening Fedora at the time? Red Hat was focused on NSS, which was the selected library to replace everything. Fedora was kind of following behind Red Hat on that, considering NSS as the primary choice. But everything else was actually building up without anyone paying attention. OpenSSR will be updated. NUTILA is the same. GNOME, everything else, was using their crypto libraries. But no one was paying attention, at least in an organized way from the perspective of an operating system. So all enhancements were upstream driven, were entering Fedora anchored, coordinated. And we ended up with a very complex landscape, like these rocks there. It had a lot of rough edges. It didn't have a feeling of an operating system. The second phase of that story is the reconsideration phase. At that time, Red Hat and probably most of Fedora's contributors had realized that the consolidation project failed. And something changed then. And let me go again to the end of this phase, because Fedora, at least in my view, and Red Hat were closer to the community. Why did that matter to us? Because we don't want to only pull from the community. As Fedora, we are an operating system. We want to also contribute the community to improve backends in a way that they fit the operating system, that they are suitable to be used in the operating system. We don't want to just open a cell in Fedora. We want to open a cell in a way that we can use it consistently with the other libraries in Fedora. So it's much easier for us to work together with the community rather than in parallel. So how did we get there? So we realized that we have a lot of crypto libraries and we have a lot of diverse crypto libraries. So we have to make them work well together. That was the first step, the realization. What happens after realization? You have to face the facts. And the fact was that Fedora was very ugly at the time in terms of crypto. If you know for the internet public infrastructure to work, you need to have trust anchors. Trust anchors are installed in the operating system. If you go to Windows or to any other operating system, they are globally. In Fedora at the time, they were installed with specific components. Java had its own trust anchors, OpenSSL I think we had its own trust anchors, Firefox its own, and so on. So what is the situation then? You run an application, it works, because it's linked with Java, and it cannot connect to another server because it's not linked with Java. So you have a situation where its application uses its own set of trust anchors. Then it comes the question, how do I install a company-wide certificate or a certificate that is only valid for my LAN? How do I apply Blacklist globally in the operating system? We couldn't do any of this stuff at the time. So the first project that was towards unification of the libraries at the time was the shared system certificates across the operating system. So it was a project about using the Mozilla Trustor as a central trustor for everything in Fedora. And in order to make it work, we had to contribute to several, sorry, several upstream projects. And we cannot make it work as it is in the picture, in the picture, a central trustor used by every other application, every crypto backend. How does it work? Something like this. If you have a Fedora today, you can write trust list, and you get the list of trust certificates that exist there, the list of trust certificates that are used by all applications. In this particular case, we see the Amazon root CA certificate. And how do we add a new certificate? Again, with the same tool, trust anchor certificate name, the certificate file, and update CA trust to propagate this trust to everywhere from Java to other applications when needed. So that was about certificates. Then let's go on, let's move on. So this is the goddess of chaos. Her name is Eris, and this was the second part, the second major issue that I was seeing during the reconsideration phases. When you have multiple systems using containers, virtual machines, running Fedora, let's say you have 100 containers running Fedora, and you see 100 connections coming out from these containers. Can you predict what would be the TLS settings that would be used for these connections? Can anybody answer? That's true, you cannot predict, and you don't know if they would be secure, you don't know if they would be insecure. That's why I selected this goddess of chaos. It was impossible to predict anything. So we had inconsistent settings, not only on TLS, but with SSH, with IPsec, depending on the library application you'd be using, you may use totally different settings. That's when the Fedora crypto policies project started, which was mainly a safety net and will protect all applications from going under these settings. It will provide a baseline for applications, crypto settings. How did it work? It had three global policies, so it was focused on simplicity, only three policies to make it easy to understand. The default would apply by default to every application. Future was more conservative for networks that don't necessarily need to talk to the internet, but they need to be protected for five or ten years. Maybe five is a more reasonable target. Legacy is if you are primarily deploying Fedora in an environment where you are talking only with very old systems and deployed ten or fifteen years ago, and you could select between these three policies globally. How would you do that? You would run these commands, update crypto policies, so you would show which policy is now Fedora, and you can even switch it using the same tool. Does it apply to whole Fedora? No. It applies to a large part of Fedora, however. It applies to major crypto libraries. It applies to Kerberos. It applies to SSH. To OpenSSH. It applies to DNSSEC if you are using Bint, Python, Java, Perl. All the major libraries and languages are being covered. That relates also to the question that was raised in previous talks. What is the core operating system today? Because we didn't have a target to cover. We just were trying to cover everything, and that was pretty much impossible. Fedora is simply too large. I will get back to it later. As part of this policy, we needed to work with a lot of upstreams because the global settings in Ocean were not available to OpenSSH a lot of time. It was not available to NSS. It was not available in UTLS. It was not available pretty much anywhere. We had to work together with all the upstreams to introduce this in Ocean. Today, in Fedora 28, it works well. In Fedora 29, it's going to work better, but we are getting there. What was the next step? The next step is something different. It was about smart cards. If you are familiar with smart cards, there are two competing open source drivers. There is the Kulki driver, competing in quotes. I will explain why later. There is the OpenSSH driver and the Kulki driver. The Kulki driver is something developed internally in Red Hat. I think it ended up in Red Hat through an acquisition, but maybe I'm wrong about this. It's a driver which covers the Kulki card, which is part of a Red Hat product, and the US government cards. OpenSSH is a community-based project which targets pretty much everything under the sun, but didn't have support for these cards that Red Hat cared about. Fedora was actually shipping two drivers for an overlapping list of cards. In the end, we had to go there. We wanted to go there. We wanted to have only OpenSSH, which was a viable community project, and move our kind of proprietary drivers because although they were open source, there was no community behind them. It was a developer at Red Hat only. We contributed our drivers back to the community. We worked very well with OpenSSH guys there. I don't know if I don't see someone from OpenSSH, but maybe I'm wrong. So in the end, now in Fedora 28, we have an OpenSSH driver covering all possible smart cards that you may need in Fedora. Furthermore, something that I'll skim briefly, but I think it's quite important, it's about hardware security modules. Security modules are very similar to smart cards, and once we optimized the driver for smart cards, the next step was optimizing, making more user-friendly the interaction for smart cards and hardware security modules. They usually operate under the same API called PKC-11. So a big enhancement that came in Fedora 28, it was not announced, it was under the hood, was that ModSSL had much better support for hardware security modules via NTNPKC-11, which is now called OpenSSLPKC-11 in Fedora. So you can set up a patchy and have it use hardware security module, a TPM or something else where you cannot extract the keys already on Fedora 28. I don't know, have you ever used hardware security modules? Are you familiar with hardware security modules? Peter is. Maybe most of you are not. The hardware security module is something that takes the keys out of your PC and moves them to a chip, and in a way that you cannot extract it back, but you can use it. That's quite important. So you can run a patchy, have your keys in the security module, and you have a hard bleed attack on OpenSSL, but your keys are still correct. You reinstalled the operating system, and your keys have not been stolen, so you don't need to regenerate old PKI, you don't need to regenerate anything. Let's go a little to the future directions of Fedora. Maybe not that long, but I'll go through a few challenges that relate to crypto. So TLS 1.3. TLS 1.3 is a new protocol. It was designed for four years in EF. It was driven by browsers mostly. It comes with a very significant performance change. You can have connections in one round trip, or even in zero round trip. That's kind of important for systems where you need it to be very interactive, and it comes with better privacy. The question is, can we have it in Fedora 29 or later? There have been some trials already. We believe we have very few regressions, because the protocol doesn't map 100%. We have some reservations there, but probably we're going to say it sooner or later in Fedora landing there. The experience we have with browsers is also encouraging. So most likely you should expect it in Fedora soon. One other challenge for the future is, we have let's encrypt. It's an easy way to get your certificates. I think what is the nice challenge for Fedora is can we have an operating system where you don't need to care about certificates, where certificates come automatically by the operating system. You install your service, and you have a certificate already there. I think it's a good idea, but at this point it's only me. So if you are interested, I'm really happy to discuss about it later. My main point for the future directions is this, actually. Fedora is buffing. We are continuously adding software in Fedora, but we have no process to deprecate any software. Once something is added, it's over. It will exist there forever as long as it compiles, and there is some maintainer assigned to it. That's a very low bar, because the software may compile, but threat models change over the years. Ten years ago, the threat model for crypto was a remote attacker. The threat model today is a remote attacker who has access to your CPU, who has access to the cast of your CPU. So having crypto software for ten years with no updates most likely means you are today insecure. Let's see what it means for Fedora. I skimmed through my Fedora system. These are the crypto libraries we are shipping that I could find in 20 seconds. Most likely we'll have ten times more of these. That's a very large security perimeter, and on the operating system level it's an unmanageable situation. Do having that many crypto libraries really help the community? That is not related with the consolidation. It's a different question having one. We don't want to get to the situation where we say, okay, that's one crypto library we're using this, but is a thousand crypto libraries good for Fedora? Everyone from these libraries most likely believes their library is the best. Their own solution provides the best countermeasures for attacks. However, we see very little collaboration all over. We see that more new libraries that just come and implement whatever is there continuously. We see Amazon developing a new library instead of helping the older libraries to become better. We see Facebook creating a new library instead of helping all the libraries become better. So how do we go from there? Do we add all these libraries in Fedora? We also have several open source, but not community-oriented solutions, meaning we have a company which open source is what they are doing. However, they don't really accept contributions from the community. If you try to work with them, if it doesn't suit what they are doing, it's over. So having too many libraries maybe doesn't help. Let me put it in contrast with REL, which is a commercial operating system from REL Health's operating system. It's a fraction of these libraries that is achieved. That's because you cannot guarantee the security of so many libraries. The question or the comment was that we have libraries which come as a dependency, not necessarily because they thought it was the best library for the task, but because it was depending on something else. Yeah, I don't have a good question. We have seen this question in Enterprise Linux, and the answer was we don't bring these dependencies if they are known to be insecure. It would be very nice to have something automatic there to detect or at least to flag these components as inactive on upstream. It can happen always that you can have an inactive upstream for two or three years, and then they are back developing. But when you are 10 years inactive, maybe Fedora shouldn't be sipping a software that is inactive for so long. If it becomes later, maybe we should consider it. This is the reason I'm actually making this presentation to underline this problem that we are. Probably Fedora security could also know if a particular library has got a good security crack or not, that there are a lot of crypto li-libraries which are kind of dead for the last two years or three years. Yeah, that could be something, and there is even a better tool than Pelk. Pelk is an internal tool that we used to identify. Crypto, please. Yeah, it does solve the problem because probably the problem is a lot of those libraries got there in this way, so this stops me from getting into it. Yeah, I just brought it as a dependency. Okay, you have a question? The question is that we have a lot of crypto li-libraries that are language specific, so Java has its own crypto library, and Goldlang the same, and I guess others the same. What's my take on this? I don't think we can answer in general. Languages which are safer, so maybe it makes sense to have crypto there because it provides an advantage. Rust, let's say, is coming like this, that it's a safer language, you can have crypto there, and you will not have all the buffer overflow, you will not have... Yes, this is correct, and the point is that maybe the language is safe, but maybe the crypto is not. That does not necessarily mean we need to follow that. OpenJDK, for example, provides hook, and you can use NSS for Java. RHEL was doing it for quite some time, so you would use Java, but you would use underlying the NSS implementation for crypto. So you reduce the security perimeter. I believe Goldlang crypto can do the same. There is already a patch from Google that I can use boring SSL, and I am not sure if I can announce it, but we are also looking at using Goldlang with OpenSSL. So... Yes, yes. So we can kind of reuse what we have there. I mean, if you have already OpenSSL in a system, the security perimeter is fixed, and if you use OpenSSL for Goldlang, even if you introduce problems by doing that, even if OpenSSL, for example, has a flow, it will affect Goldlang, but it will not extend your perimeter. You can still fix it once. You don't need to fix flows in Goldlang and in OpenSSL. I think that's a good idea. I am not pushing for it. I think sometimes it makes sense to have a separate crypto implementation, especially if you have a safe language also in the terms of timing resistance or memory access resistance. Though I am not sure we have such language yet. So in some cases it may make sense. In general, I think it is better to reduce the security perimeter. So let me move on. I think there was a discussion here where the questions I have here were answered, but let me move on. Can we stop that puffing of Fedora? I think we should make it easier to eliminate dead software from Fedora. We should make it automatic, so that no one is involved. It was suggested before, I don't think anybody should be involved. If software has not been updated for 10, 15 years, it should be out automatically. It doesn't make sense to keep it in Fedora. It's a very long time. And what I say under it is maybe contradicts what we were suggesting before. We should allow new crypto software to enter Fedora. As long as we respect the policies, for example, that we have set up, we don't want to make Fedora stagnate. We want to have the new OpenSSL or the new Nautilus in Fedora when it gets designed. If it is used, if it is a nice library, if it follows the crypto policies, let's bring it in Fedora. Let's not stop it. We should make it easier to remove something that got in, rather than make it harder to get things to Fedora. The comment was that it's good to make people aware that this problem exists, so that when something enters Fedora, we can review it. The thing is that I don't believe anybody would review it. I've been involved in a past in Fedora. The Fedora security team, I've never seen it functional. I'm relatively new. I'm only four years in the community. If it does, it could be a nice thing. If you can review an implementation before getting in crypto implementation, that would be good. Review even if it is superficial, just to see if it's a good language, if it's a style, if it's good development practice being followed in the development of the library, that would be good. So, as you say, I don't believe we are at odds. I wanted to underline that we should allow a new crypto to enter Fedora. We should not make it overly difficult and I would like to finish with that slide. Thank you very much for following. I like that picture. But I think it's a good picture of Fedora and crypto. So, thank you. Let's start with the second. Post-cantum cryptography is a personal decision. Yeah, sorry. The question was about post-cantum crypto. Is Fedora doing anything about it? And whether PELC, which is a RERHA tool, will ever be made available to Fedora? I will start with the second about post-cantum crypto. Both RERHA and Fedora do not believe we are doing something in an organized way. That's kind of a problem, industry-wide problem at least for RERHA, post-cantum crypto, because we don't have solutions at the moment. We have some algorithms being presented as post-cantum safe, but we have no proofs. So, we just know that these algorithms, there is no attack with a quantum computer, but we don't know if they are really secure, and that's not enough to start implementing in operating systems. We have also indications that quantum computers are getting close, but we don't have a practical quantum computer that can attack the crypto we have today. We just know that maybe in five or ten years, we will be there. We have that feeling. Depending on who you ask, you may get a different timeline, but we believe in ten years, the problem will be there. So, what do we do as Fedora? We wait. We wait until standards are established for post-cantum crypto. We have an algorithm that is widely accepted and then we will try to bring it in Fedora and maybe instead have to contribute to upstreams to bring that algorithm to upstreams. Now, about PELC. PELC is a tool that automates discovery of crypto in packages like RPMs, but it's not a very good tool. I don't think we should bring this to Fedora. There is a... Because it has very... And it has many false... false positives. So, you start with a very long list of crypto that was invented because idea, for example, was mentioned somewhere in the code and it thinks, oh, idea is implemented here. So, there is a better tool developed in Red Hat for that reason, but maybe you can get that made open source and donated to Fedora. That's something we should probably discuss offline. Please. So, the question is about the trustor, the certificate... about certificates in general, whether we can share it between containers between the host system. With the trustor, we have currently a solution to share them across containers or across virtual machines. Actually, Flatpak is using that. P11Kit implements the trustor in Fedora. So, P11Kit allows you to forward smart cards in... between systems. So, the trustor is being implemented as being the interface of... the trustor is actually like the same interface as a smart card. So, you can forward the trustor read only to a remote system. So, you can actually have the same certificates, the same trust anchors as you have in the host, for example, in containers by forwarding P11Kit. However, that will not work with the specific certificates, as you said. Because we don't have in Fedora or any other links, that this can be seen as a container for your certificates or for your keys, a key ring. We don't have... They used to be GNOME key ring, but it's not so generic to see it as a system key ring. It's being phased out anyway. So, we have an answer for the trustor. We don't have an answer for other certificates. Any other questions? Please. Yeah, that's a nice question. It's OpenSSL-PK11 has a very bad description on its packets. And the question is, should I install it on my system? Yeah, the nice description, I will encourage you to open a bug against OpenSSL-PK11 and say, I don't understand what it is about. It makes sense. I mean, we should provide good descriptions in Fedora packages. Now, going to the point, what these packets do is, if you have a hardware security module, if you have a patching on your laptop... No, usually, if you run a web server, you usually know you are running the service there. And if you want to use a hardware security module, you also know because you have bought it. You bought the UBHSM or you bought a nitro key or something that can be used as an HSM. You plugged it to your server, and then you move your keys there, and then you make a patch and use these keys. So, it was maybe in the past targeted an audience which was very into it. They knew what PK11 is. Maybe today that HSMs are being more easily available. Maybe we need to make them more attractive, more easy to understand. So, you say PK11 in general doesn't tell anything to you as someone who may potentially want to use a hardware security module. If you don't use a hardware security module, you don't need it. If you don't use smart cards, you don't need it. If you use smart cards and hardware security modules, you need it. So, you can make your life easier with how you specify the object to the smart card with how you find the driver to the smart card. I don't believe you will receive anything because the dependencies are recommended by Apache and by applications that may use it. So, it will be installed automatically unless you remove it manually. So, thank you. Over time. Thank you a lot. Bye-bye.