 So welcome, everyone, to a talk where we are going to discuss a very interesting topic on managing trust anchors in a cloud-native environment and kind of going back a little bit back to the future. We'll look into how trust anchors been managed historically and how we can do better in a cloud-native environment. But hi, everyone. My name is Eli Nesterov. And it's such a pleasure to present here. I still remember a very first KubeCon that I attended in San Francisco many, many years ago. And it's very small. So it's always good to get back on stage and talk about some cool stuff that's happening here. So I've been long time as a community. I happened to run one of their world largest deployment of Spire, scaled beyond million nodes. And in 2020, I don't know if you've answered their pandemic, we've been writing a book about this kind of infinite regress problems in security and how Spifi inspired us to solve them. The book called Solving the Bottom Turtle. So if you want to read it, it's completely free. You go to spifi.io slash book. And you can get an awesome PDF in there. And I'm also co-founder and CEO of a company called Spiral, where we build in workload identity platform based on specific specification. So getting back to 2020, I've been thinking about the trust. And I kind of get to this idea of identity trust paradox where we're just starting with very interesting observation of we as human beings usually tend to grow trust between each other over time. And surprisingly or not surprisingly, there is lots of different researches how it's happening. But in many cases, the trust between human beings started on some kind of maybe close to zero, maybe higher depends on reputation and other things. But over time, it usually tends to go up north and east. Different researchers related how it's the trust erosion between human beings happening. But I found it's interesting that in digital world, it's a little bit an opposite. So if you think about, hey, here's my server. And I stand it up today. And I connected it to internet. But in a year, you probably will trust this server less than today if you didn't touch it. Because you didn't do any software updates in there. And if it's been connected to internet, probably someone find a vulnerability already on it. So there is kind of idea of erosion of trust over time in digital world. And the same related to digital identities. We used to issue certificates for our raw servers for like three years and for two years and for one year. But it was like a few of them. And we manually put a private keys in there. But you still kind of, over time, you have less trust in this certificate than a private key. Because if someone compromised that server, maybe they already own the private key. So we get to, if you use a lots and creep these days, issue certificates for 90 days. So your trust go in this up and down to these digital identities over time. And I found it's very, very interesting compared to how we human beings trust each other. So we're going to talk more today about managing trust in a cloud native world. Because we don't have a single server instance. Now we have a bunch of containers, pods, and everything, or virtual machines running everywhere. And I'll start by looking into how a browser and operation system manage their trust anchors and what it actually means to trust anchor. But at the end of the day in our digital systems, your managing trusted boils down to managing trust anchors. So what is a trust anchor? And RFC 5914 defines it as an authoritative entity represented by a public key and associated data. So there is a public key and some metadata about it, but usually have more than one trust anchor in your system. And often it calls like trust store. And you can think about it as just basically a list of trust anchors or public keys. So the probably most popular trust anchor list is called the VOPPKI. It contains, I don't know how many entities there, but I've seen recently, I think Ryan Hurst tweeted, that's 85 trust, VOPPKI trust anchors cover 99% of all the certificates issued on the internet. I don't know how long the list in your browser, but you can probably open up, depends on your operation system, or if you use a Mozilla Firefox, it has its own built trust store. So this is probably the only browser, at least like from most popular that I know that support its own trust anchors store that is completely independent from operation system. But the rest of the browsers, like if you use Chrome, or Safari, or Microsoft Internet Explorer, they will be using trust anchors that's managed by your operation system. So they don't have their own. And trust anchors have really this trust store has an interesting property, because users can add or remove the trust anchors in there as well. We need a way for them to manage it. So most of the browsers rely on operation system. So which operation system has its own? This is like example of how macOS look like. And as I mentioned, users can add or delete trust anchors in there. One of examples could be your private company, ECA. But what if this, well, PKI, one of this, I don't know, let's encrypt them or get compromised. How you all know about it, you'll probably get an operation, your operation system update with this trust anchor removed there. So if you go to macOS has a list on their website where they publish a list of trust anchors that's delivered with an operation system, you can find the same on your somewhere in user slash library. There is also HTML file with all the trust anchors. That's your version of macOS support, like with all your added trust anchors in there. So you can compare and find. Maybe it's the same or it's different. Maybe someone else, someone put something that should not be in there. And I found it interesting that Windows also have disallowed certificate list. So it's kind of, they have a model of allow list and a block list as well internally. But there is also Linux and different distributions have their own idea of where the trust anchors, well, PKI trust anchors should be stored. In most of the cases, like any open SSL based software, we'll be looking into EDC SSL certs. So this is where most of the trust anchors will be stored. And I think Red Hat has its own view of this. And here's a pass for basically CA bundle, where every trust anchor on that type of system is stored. So in order to get your trust anchors on Linux, you need to install CA, their certificates package. This is like update of this package will deliver basically new CA certificates in there. And because most of the containers based on Linux distribution, so basically this is the same way how you get your trust anchors in there. And if you, I don't know, if you use some curl container, and it usually comes with CA certificates package so you can go ahead and do a curl google.com, and you won't get in any error in that case. And oh, that was loud. Yeah, so you can see in Dockerfiles this line where you do run a CA certificate package where you do, I don't know, install of it to get all your trust anchors. But some of their container distributions, like distrelas, for instance, doesn't come as a package system, so we need to copy their whole directory in there for your software. And it depends if your software is using OpenSSL underneath or something else, like an mzlnss. This will be looking into different location. So this is about WebPKI, but how do you deal with your company private PKI? So I don't know. We have our own version of JIRA or a GitHub running internally, and obviously we want it to be covered by TLS, but we want to use our company private CA. So now how do you distribute this root of trust among all the machines that users have to connect? Or if you have your production or DAF or a staging environment that needs to use this WebPKI, how do you distribute this trust anchor now? And there's different systems that's built over time. Maybe also want to use client certificate authentication for users that's connected to JIRA or, I don't know, any internal system. So there is for distributing private root of trust among their user machines like a desktop, there are systems like an MDM, or for users there is a different certificate enrollment protocol that's been built over time. But there is also a need like, hey, I'm building a Java application. And it also needs to trust my company private PKI, so how do I do this? I need to add it to Java key store, which is also the fun process to do. And if you build in something, I don't know, very exotic, like a software that's using Mozilla Stack underneath, it will be using an SS component that's stand for, I think, network security services that basically comes with its own trust anchor store and you need a way to find a way to add your private CA in there. So this is like in general, but what do you do with the containers? And I slightly mentioned this, that you see this kind of line of adding CA certificates packages, right? So this is what you do in most of the cases. An example is like if you're using Alpine, you'll do something like APK CA certificates. But for your company private CA, during CI process, you need to modify your Docker files and have some example later to copy your company trust bundle into the container, and then you need to run updates, a certificates command, so to propagate it. So you have like multi-stage process of adding in there during your CI system. And I found it interesting, and Dan Lauren wrote this kind of simple utility called insert. You can just run it, and you can point into your company trust bundle. You'll just download the container image and it add this bundle inside the trust anchor store. Internally, it can also push this updated container back. So in most of the cases, that's what you need. But with Java, you need like a separate process that built the Java key store file, and then you need to copy it inside a container as well. So this is like how you do it during your normal CI process. So here's a quick example for Alpine, right? So you're adding your VOPPKI trust bundle in a third line, adding your custom CA. This might be your company private certificate authority. And then it's basically adding another command to add it. So that's what you do during CI, but CI process is like propagate very slow, right? So it's like how you can, another option to do this is to do during runtime. And how you can, for containers, add the trust bundles. And one of the easiest solution could be you just do volume mount. So you can do volume mount for containers that's running just like on a node. Or if you're running on container on Kubernetes cluster, you can do mounts through CSI driver as well. So just mount the whole volume with all the trust anchors that you need to have in your system. But it's still kind of, you need to invent this machinery, you need to build this machinery, like how this bundle will be delivered through which means through your config management system or through GTOPS, whatever process you're following. There is also a way for Kubernetes to use a CSI driver that will mount this volume automatically. There is an interesting project from Jetstack called Trust Manager that's basically automated all this process for you. I didn't use it. It's still in development, but it's showing this kind of promising step of automation toward you don't need to reinvent the wheel, basically. Another option is to, it's interesting, and there's a screenshot and in the slides later, there'll be a link to the article from a small step where they basically suggest in a solution where you, your container use this init script to download the trust bundle from trust anchors from some endpoint that's you presumably control and there are some additional checks to do. I just found it interesting and added in here so we can check it out later. But in general, how much you should worry about managing your Vope PKI trust anchor? How much you care, right? Because probably you need them in your container only if your software is talking to some endpoint that's protected by Vope PKI, right? So instead of like, hey, here's like Slack and today their API is using, I don't know. I don't know what they're using, but let's say Aletsyn Crypt and you are adding on the Aletsyn Crypt trust anchors into your container, tomorrow if they move to, I don't know, Cloudflare, you'll need to update again your certificate. So instead of you just add everything, like the whole Vope PKI. But do you really need to worry about if something of this certificates in Vope PKI get compromised or expired because what is the potential mechanism of exploitation in this case? And the only one I can think about is like if there is a man in the middle attack on these containers, but it's also need to be a very targeted attack, right? So you know that I'm talking to some API and that API is protected using some CA and that CA is compromised. And in this case, I can just do some man in the middle. So it's like very complicated and not very a patternistic attack. But there is also a vector of I found a way to compromise CI system. And I know that this director is using to build a trust anchors in my golden image, right? That's after that's used by the whole CI system to build everything. So if I find a way to sneak something in there, then it means I can do man in the middle like on everything in, think about it like SolarWinds potential attack type of thing. So it depends. And as an interesting utility that basically can scan any of your container image, and it will show you all their expired certificates. In it, it's using a combination of, I think, Mozilla, Anasas, List, and something else to verify against. I wanted to show the screenshot with their Nginx latest, but it was like, I don't know, 20 or 30 expired certificates. So I run it against Chingard curl. And it still has like a five expired certificates because of the downstream dependencies, right? So it's using Alpine and Alpine using Mozilla, Anasas, List. So instead of using directly Mozilla, Anasas, there is some lag, right? So this is why it's, in many cases, important to know your dependencies for all your images because, as I mentioned, at the end of the day, most of the Linux distribution rely on Mozilla, Anasas, List, but the speed of propagation of these updates is different for different distributions. So it's like your Debian and Alpine speed of getting updates from Mozilla, Anasas, List will be very, very different. And maybe you want to do this directly because it's just like a one file in Mozilla directory that contains the whole list of trust. Another known database, I think it's from Mozilla, Microsoft, and Google built a database of trust anchors for the internet. But it's, I don't know, I spent 15 or 20 minutes looking into the list of all the trust anchors. You can query it, but you can't get a list from Mozilla. It's much harder. So all this kind of talking about dependencies basically to lead to thought on what if tomorrow one of certificates or private keys for them get compromised? Do you know how long it will take for your company to update all these images that's used in there? I have no idea. And usually the speed of updating this through CI system takes long time to propagate. So we need to rebuild all these images. We need to rebuild your gold images or gold images. Then the rest of the dependencies. And then you need to update everything at a runtime, like anything that's like running on my, I don't know, Kubernetes clusters in production. And if you're working in a large enough organization, the release cycles could take, I don't know, weeks, sometimes months to propagate this. So it's very slow. And how we can do it better? There is probably half an answer for this. I don't have like, hey, here's a solution. But SPIFI solving not only workload identity problem, but it's also solving trust bundle and trust anchors propagation, but specifically for SPIFI. So what is the SPIFI? And SPIFI stands for Secure Production Identity Framework for everyone. There is a specification. It's like one of their CNCF graduated projects. It's basically a standard that describe what is identity, what is the format of it, what is the SPIFI ID. And SPIFI ID is basically a URI-like string that goes into SPIFI verifiable identity document, which is basically X509 certificate or dot talking. And the biggest part of the specification is a workload API. And workload APIs are not local Unix domain socket, which any workload can talk to, who understand how to talk to workload API. And when I talk to workload API, it can get multiple things like its own identity. It can get a trust bundle that define what this workload should be trusting in this environment. And it's all very kind of end-to-end automated. So SPIR is like one of SPIFI implementation. This is an open source implementation. And in general, any SPIFI implementation consists of two things. There is a server, and there is an agent. An agent is the one that is basically providing this Unix domain socket with workload API. In SPIFI world, workloads doesn't ask for specific identity. They just literally ask for the identity. Hey, give me identity. I don't know who am I. And then it can use it to talk to other workloads. And a bundle is a part that basically helps to manage the stress. So I'll show a quick demo. I have this simple SPIFI implementation. Let's basically take these two parts of agent and the server and it's kind of one binary. It's exposes this Unix workload API. And my workload is a back-end. Let's talk to this workload API and it will get an identity. And I'll use a browser as a front-end to show what the identity is. And so first I'll show what the SPIFI and how this identity look like. And then I'll show another quick thing, how you can use this workload API not only to distribute SPIFI trust bundle, but how you can extend it to distribute, I don't know, your company root CA certificates. So I have this thing that's called the SPIFI CLI. And we're going to run it. And it use this configuration file, which I don't want to explain and go into details. But the most important part is, like, hey, we have this supplemental bundle path that has multiple certificates. So it has, like, three certificates in it. And so now we have this SPIFI server and agent running. And our workload will be this demo application that's front-end listening on a local house. So if we go here, we can see what is the SPIFI job as we look like. So remember, I talked about the SPIFI URI. So that is how it look like for their job as we did. And we can show their X519 as we did, which is just basically X519 certificate. And it has this Sun URI thing with exactly the same string. So this is, like, the string that can be used for your authorization systems beyond an authentication. And this thing's called the SPIFI trust bundle, which has digital certificates as well for this trust SPIFI trust domain. And this is an extension that I added. So there's, like, three certificates here. They just basically, this is human readable view of them. So we have this AcmeCorp, say, one, two, and three. And what I want to do now is I have this another file that's called bundle short. And it's only half, like, a two, say, certificate. And what I want to show is, like, OK, let's say these certificates get compromised. So I want to update our bundle distribution. And it will, at a runtime, propagate it pretty quickly everywhere. So I'll just need to change their configuration here to point to this short bundle instead on a server side. I was lazy for a demo, so I do not watch the changes on this file. So I'll just quickly restart the server. But now if we request back-end, we'll request workload API. Hey, give me the supplemental bundle. It will have a two-certificate. So it's like, basically, if you have a fleet of machines, it will propagate everywhere pretty quickly. So that's it for I'm running out of time. OK, so quick takeaway to wrap it up quickly. So if you build an insertised system, build your golden image and propagate it. And automate every scene, ideally in a runtime, and use SPIFI. And you can even extend SPIFI to propagate your images very fast. So thank you. I'm run out of time, but I'll be around if you want to ask some questions.