 All right. Welcome, everyone, to Cloud Native Live, where we dive into the code behind Cloud Native. I'm Taylor Dolezal, head of Ecosystem at CNCF, where I work closely with teams as they navigate their Cloud journey. Every week, we bring a new set of presenters to showcase how to work with Cloud Native technologies. They will build things, they will break things, and they will answer your questions. In today's session, Richard and Citurum have joined us to talk about workload misconfiguration, the number one security threat when using Kubernetes. This is an official live stream of the CNCF, and as such is subject to the CNCF code of conduct. Please do not add anything to the chat or questions that would be in violation of the code of conduct. Basically, please be respectful to all of your fellow participants and presenters. With that, I'd love to hand it over to Richard and Citurum to kick off today's presentation. With that, please take it away. Okay. Hi, everybody. I'm Richard Cullins from Jetstack, Product Marketing Manager. Thanks very much for joining this chat. The idea here really is to, as you may know, Jetstack is the primary maintainer of CERT Manager, a very successful open source project that is used to automate X509 certificates for Cloud native workloads, and that particular project has huge community traction. We're here to talk about that, obviously. In particular, we thought we'd go into detail about where we see some of the potential security vulnerabilities that affect the way we see enterprises using Kubernetes. There's particular areas of information and research that we've identified and looked at more recently around that seems to identify that workload misconfiguration is a very general way that you can describe how vulnerabilities manifest within Kubernetes, but we feel that there is a particular context in relation to significant misconfiguration that we can talk about in detail during the time we've got here. But I'm now going to hand it over to Citurum, who's going to say hi, and he's going to take you through a demo that we hope you'll find interesting. Citurum, over to you. Excellent. Thanks so much, Richard. Hello, everyone. Sitaram Ayer. I am Senior Director of Cloud Native Solutions at Jetstack, a benefit company, and today I'm happy to be here to talk about some of the things that we've been doing at Jetstack, and also expand on some of the use cases scenarios that you use typically with CERT Manager and some of the add-ons of CERT Manager and how we can truly, truly build that zero-trust architecture more in the context of securing your mesh identities and how you sign those mesh identities utilizing the CAs or the reorganization CAs. Today, what I really want to do is walk you through, I'm hoping that you're all seeing my console, but really walk you through some of the things that you can really do today with CERT Manager and some of the add-ons, and I'll introduce those add-ons as I go through that. But at a very high level, the way I want to approach this is to think about any applications that are deployed in your Kubernetes cluster and potentially, let's say you have Istio as well as a service mesh that you have rolled out. How do you sort of, one, make sure that your ingress gateways are secured using a certificate that is approved by the organization? That's one piece. The second piece is also, how do you ensure that all of the workload search that are issued as part of your mesh workloads are also sort of secured and issued via an approved CA from the organization? When you think about it, very often that ingress gateway cert is essentially a publicly trusted certificate because more often you do have applications that are used via a browser or some sort of a mobile client, things like that, and mesh identities typically sort of are internal to the workloads or the internal to the organization or internal to the cluster. So very often you see them signed by an internal CA or a PKI infrastructure that is managed within the organization. So what I really want to do today is to break it up into sort of two pieces. One, where you set up and manage that aspect of securing the ingress gateway itself. And then the second piece essentially is to ensure that all of those mesh identities are signed using, let's say, Walt. So a hashCorp Walt as a way to sort of sign that. So you will see the installing cert manager obviously is installed on my node or on my cluster, rather. So I do have a brand new cluster that I've just provisioned about a little less than a couple of hours ago. This is running on GKE, as you can see. I will also install Walt. I'll also install Istio CSR, which is an Istio agent that we've developed when I set cert manager and add-ons. That's essentially one of the add-ons that facilitates signing mesh identities or facilitates signing mesh workloads. So that is something that I'll install as well. And as you can see, I do have my console open. I have my development window open as well, where I can see my make file. So I'm gonna be running a few commands essentially to sort of run through these various different aspects of building, configuring, securing, all of that. Taylor did mention that this is a part where we sort of try, break things. If things break, we fix. All of that will happen or everything might just work as I expected to work. So if something breaks, we'll fix it. But that also gives you the idea that it's all real and what I'm doing is actually on a real cluster with things getting set up and configured. So that's it. Keep bringing your questions. I'm happy to answer. Richard's happy to answer. Taylor's happy to answer. So we'll keep answering questions as they come in, but as a part of the demo itself, I'll keep going unless somebody sort of needs a question to be answered. If something is not clear, and I'm happy to do that as well. So like I mentioned, one of the first steps that I'm going to do is essentially to install Wart itself. And in this case, I'm going to be running Wart on my cluster. Basically, typically you would think about Wart running somewhere outside of the cluster, managed by an enterprise. In this case, just as a way to sort of demonstrate, everything is going to be configured running the cluster itself. Everything in the form of a make file, I do have a Terraform module that runs and configures the hash bar in this case. But as soon as that's ready, so we will be using that as a way to manage issuance for all of the certificates. And for those who are not familiar with CertManager, and Richard covered this a little bit, CertManager is the project that essentially is used as a way to sort of manage all of your certificates. It's essentially a certificate controller that runs in Kubernetes, helping you facilitate, manage all of your certificate requests and certificates that are needed very often as a common way to secure your ingresses. But we did one of the things that I wanted to cover also is, is the fact that it's not just used only for ingresses. We also have other add-ons that helps you facilitate or at least ensure that there are different kinds of workloads that are signed, different kinds of security mechanisms that organizations sort of drive. Security compliance that is required where some organizations require a certificate to be injected into a part so that the container that's operating within that part has access to a certificate that it can use. So we use the CSI driver. There is another project called CertManager CSI driver that essentially facilitates injecting certificates into the part itself. And that is a demon set that runs as part of the Node agent and helps you manage that aspect of things. So simply put, there's a few things that I have. Like I said, I installed Walt. So Walt itself is operating within the cluster. So that will help facilitate the requesting the identities. The second piece, what I'm gonna do really next is to run my next step. And before I run that next step, and as you can see, there are a few things that are happening here. I'm gonna set up a basically an identity or a certificate that will act or at least facilitate Istio to sign requests. So what I really mean by that is when you install Istio, the out of the box, obviously it comes with its own CA issuer that is built into Istio. What I'm really doing here is to utilize Walt as a PKI backend to set up that aspect of CA issuer. So when we install Istio eventually, as my next steps, when I install Istio, you will notice that the CA issuer that is part of Istio is gonna be turned off and we will delegate issuance of workload identities to cert manager. That is essentially what is going to happen. And in order to do that, I'm gonna prep and make sure that there is a certificate available, a certificate that is an issuing CA or an issuing certificate that will be configured and is ready for it to issue workload identities. Part of that is also where I'm going to install this Istio CSR, as you see, I'm now installing the cert manager Istio CSR, basically allowing adding that additional components to the cert manager namespace. So basically if you look at the cert manager namespace, cert manager is something that you're very, very familiar as I imagine. So something that's running already in my cert manager namespace. The CSI driver that I mentioned is also running because it's part of my bootstrapping process when I deploy my cluster that is also running. And the reason why you see so many CSI drivers as I mentioned is because it's a demon set tied to the nodes. The third piece that I just installed is essentially the Istio CSR as you see part of my step, which I just installed, it installed the Istio CSR. So it's up and running. The other piece that I wanted to really show you is this certificate resource that just got created. That was the previous step to creating the installing the Istio CSR itself. So this certificate that you see here Istio is essentially what we are going to be using for signing all of the mesh identities. This is no different than any certificate that you would typically create when using cert manager. That was the same mechanism that was used. The only difference is in this case, we use the vault issuer that is running in the Istio system namespace to issue that certificate. So so far all I've done is basically configure issuer which is very familiar to many of the folks who are operating cert manager, create a certificate which is also very familiar to a lot of folks who are using cert manager. The only difference is this specific certificate called Istio, it is also an issuing certificate because that can issue certificates on behalf of vault. It's essentially something that will be configured as part of the Istio CSR. So pretty straightforward what I did so far. So my next step essentially is to install Istio itself because what I'm now going to do is install and configure Istio using the Istio operator and I have to walk through what that configuration looks like if there are questions about it. But basically very simply put all it's trying to do is turn off the CA issuer that is part of Istio and then let cert manager take over the role of issuing workload writing please. I'm installing 114. So that's the one which I have the config for basically installing configuring and they should be pretty fast. As you can imagine, it doesn't take a whole lot of time to install Istio. You'll see Istio getting installed, you will see the Istio Ingress Gateway installed and the Egress Gateway installed as well. All of the various different components that are required for installing Istio will all be installed as part of this process. So straightforward like I said, all we did so far if I have to walk through configured a secret, configured the Istio CSR or installed it and then installed Istio itself. And then if I look at Istio system, so this is already running. So this is the Istio system that I just talked about and also since I mentioned that it's running on running on Google Cloud, we should also see a load balance or automatically getting provision should happen anytime. But basically allowing us to be able to map any of the applications that we are going to deploy via the Ingress Gateway and the Gateway resources that we will eventually be creating to allow access to the application. So nothing special here, something that you probably have done many, many times installing Istio. So something that the only difference in my Istio operator is that ability to utilize cert manager as a way to to sign the workload identities. The next step, as you see here, one of the things that we really want to do when we think about workload identities, obviously one of the things that people do or organizations really think about securing workloads is to facilitate MTLS. And within Istio, obviously there is this notion of a period of indication that you would specify what kind of mechanism or what kind of mode of MTLS you want to set up. Permissive, strict, pass through in all our different kinds of modes. The thing that we want to do here, especially as part of ensuring that everything is absolutely secure, everything is absolutely using MTLS is to set up that peer authentication with MTLS mode set to strict. And that is my next steps and all I'm doing is creating this peer authentication resource to ensure that it's MTLS peer authentication. Peer authentication. So if you see the MTLS mode is strict, that's essentially what we're trying to do here, basically ensuring that every time Istio injection is enabled for any of the namespaces, we will make sure that it is always using MTLS in a strict mode. That's essentially what I did in my step three. Step four, the next step, as I'm sort of building the scenarios here for you, what we really did just as a way to back up a little bit is set up or prep the environment to be able to sign workloads using cert manager for your mesh workloads. I did mention at some point about an Ingress Gateway as well. As you remember, the Ingress Gateway essentially is going to be something that is used as a way to access your applications typically, almost like saying my north-south way of communicating to my applications. So that is usually many a times browser-driven, app-driven, mobile app-driven, something like that. So in that scenario, I also need a identity for that specific endpoint or a DNS that is going to be configured as well. And in order to do that, I need somebody who can help me issue a public certificate. And obviously, many of you probably are using cert manager with Let's Send Crib for Acme as an Acme issuer, configuring the Acme issuer, solving your DNS or HTTP challenges and requesting certificates and managing it automatically. In this case, I am going to use Verify Cloud to get me a certificate for my Ingress Gateway. So basically, ensuring that there is another certificate resource that is going to get created, except in this case, I am going to use a completely different issuers. As you probably already know, many organ... You could have multiple issuers configured with multiple different configurations, each of them catering to different needs or different aspects of what you do in the cluster. And one of the things that I'm doing here is to create another issuer that allows me to issue publicly trusted certificates. And in order to do that, it's simple again, I'm using my own domain here, which is called sitram-ire.gcp.desk.net and the domain, the secret itself is called storefront-fast, basically allowing me to get that certificate, which is publicly trusted, as I mentioned. So we will inspect these resources as well. We'll look at what the certificate resources look like, how long are they valid, who issued that, all of that is obviously available for us to inspect and look at. But if you can finally look at the issuers, I do have two issuers, one which was the work that we talked about, the other one is the beneficiary issuer and you could have as many issuers. These are issuers, essentially, which means that these are names-based. You could also have cluster issuers that are cluster-wide, something that is very commonly used as well as a way to manage cluster-wide issuance of certificates so that people don't keep creating different issuers. That said, when you want to scope it to a certain namespace, put some R-backs around it, put some controls around it, put some policies around it about who gets to use, what domains to use. The other add-on for Cert Manager, which is also called a policy approver, which has the ability to define policies within your cluster to manage aspects of who gets to use an issuer for what domains and how they can request certificates is absolutely available. That is another add-on. I don't have it in my cluster today, but if I were to have that, I would have another resource called Certificate Request Policy that I could create and define how that would look like. And that also integrates with different kinds of policy approvers, policy framework. If you are using, for example, Kyberno or if you're using Venafype, all of those policy frameworks can be inherited or extended via those policy frameworks into Cert Manager as well. So that's essentially what it is. Basically what I really did was configure aspects of getting a public certificate, also setting up a way to request private certificates so that it's all tied into the identities that are managed within the application. So as you can imagine, I have a certificate, I have a domain that I'm going to be using as part of my Ingress Gateway. Something needs to tie in that Ingress Gateway, it's load balancer to the Gateway resource that I'm going to be eventually creating and mapping it to the DNS that I have in place. So which typically means having this specific DNS mapped to that Ingress Gateway resource, you know, all of that. And in order to do that, you know, I have, I do have another step here which is essentially going to configure that app DNS, it's a load balancer within Google Cloud, goes to Cloud DNS, sets up that domain, and then make sure that, you know, the IP address or the A record of that DNS has the value from the Ingress Gateway service load balancer IP address that was also created. And that's essentially what I'm doing in this next step. So as you see, I'm building over different things, basically configuring a wall, basically configuring is the OCSR, then, you know, setting up the peer authentication, and then also requesting public certificates. And now I'm tying in the load balancer IP address to the domain that I've created. So that way, you know, when I really hit that DNS or the IP address or the application that I want to access, it is going to get routed to the Ingress Gateway and eventually the resources that we are going to be creating. That's pretty much what it was, then what I did. My next step, obviously, is to install a sample application. And for those who are familiar with, you know, some of the demos from Google Cloud, you know, this is a slight modification of the Hipster Shop demo that is typically used as part of Google demos. So I've just used that, modified it a little bit to sort of, you know, to make it look like, you know, what we're doing more from an application that is slightly different. So you will see that when I show that. But basically, my next step is essentially deploying that variant of Hipster Shop into the cluster. There are a few other things that I'm also doing. I'm creating a gateway resource. I'm also creating, you know, the gateway resource that maps to that specific front-end service within the Hipster Shop. So that way, when we have the application that's accessible, you will see all of the identities and everything that are associated with it. So that is getting installed in the Istio CSR namespace. So that should be coming up, yeah. Still coming up, a few of them are still one of two. As you see here in this specific namespace, when we're deploying the application, these are all two of two. One because, you know, there is Istio injection enabled on this namespace essentially making sure that, you know, every single application or every single part also has the on-wipe proxy from Istio. So to make sure that, you know, there is that Istio aspects built into this. So this is the application that many of you may be familiar, you see when you see ad service, cart service, or recommendation service. So that is the general Hipster Shop application that you typically see once that's there. The next step is also again, optional, but basically I'm also installing all of the Istio tools. You know, you may, for many of the folks, you may be using things like Kiali or Prometheus or Eager, you know, all of those two observability tools that comes as part of Istio, something that you may or may not be using, but very commonly used as well. So I'm gonna install that. As you know, Kiali when it installs, it creates the CRDs first or it doesn't create the CRDs or CRDs are not available. So you do the two-time thing for installing Kiali. So now we should all install. So I'm installing Kiali as you see, I'm also going to install Eager, Prometheus, Grafana, all of the various different tools that are typically common in an enterprise, just as a way to tie all of them together as you see it. So what really happened now is things that I've installed and I've deployed an application. So you may have different mechanisms through which you're deploying, you may have all of this 100% automated and nothing stops you from automating. I mean, obviously, you can automate 100% of these things. There is nothing that I've done manually that requires any configuration changes or things like that. So I went from scratch of installing wall, configuring Istio CSR, configuring certificates, setting up, you know, MTLS strict, all of this can be packaged as part of your application development process, your pipelines that are already in place to help facilitate that aspect of, you know, deploying and managing the application. We will start to see a few more things, you know, when we originally sort of, you know, looked at certificate, we had two of the certificate resources that will continue to be the certificate resources that will be there for one to secure your knots out. The second one is the issuing CA. What you will notice is one of the other kind of resource that I did not talk about, I think now is the certificate request resources itself. And you'll see a lot of those getting created. And for those who are familiar with Cert Manager, you do know that every time a certificate resource is created, Cert Manager creates an accompanying certificate request resource that carries the CSR, that carries all of the information that needs to go to the issuing CA. And once that issuing CA issues the certificate, it brings it all back into the certificate request resource and updates the certificate resource and also updates the secret resource. Couple of things that are slightly different in the case of CSR driver and is the OCSR is that we don't necessarily represent all of those mesh identities or workload identities in the form of a certificate resource. They are only certificate request resources, mostly in the context of being highly ephemeral in nature. And you can pretty much turn this off as well to say, I don't want to preserve any of the certificate requests that are used for my mesh identities because we have hundreds, if not thousands of services that are running, there is no point in storing all of them, keep it in memory. We want to be able to track it for auditing purposes and things like that. So that is possible, but we don't necessarily have to store or store it as a resource and manage it. That's possible as part of the configuration itself. What you're really seeing here is an identity that was generated for each of these workloads, basically as a way to manage and update the workloads and everything. So that was pretty much what I wanted to sort of set up. I'm also now moving into my browser just to sort of show a little bit about what really happened from an application that we accessed. So if you look at my application, so this, like I did mention to you guys, there is this notion of a hipster shop that is very commonly used, I've slightly modified it to show slightly different things, not the products that you typically see from hipster shop, but in this case, I'm just showing you various different blocks and things like that and also how we operate. What I'm really doing is just generating some data so that we can look at all of this in Kialia as well and see how that manifests and how that shows the data. So this picture that you're seeing is essentially what I really did, basically having said manager, talk to various different issuers in this case. In the case of mesh identities, I'm using vault, in the case of the public CEA I used 205, you could potentially be using other mechanisms to sign my workloads, but then essentially having all of these MTLS facilitated as part of the mesh identity. So that was one aspect that I wanted to show you. I also said, I'm gonna show Kiali and I'm gonna bring that as well just to make sure that we can see information in a slightly different context, very relevant for the work that we just did. And what I'm gonna do, I'm gonna turn this security tab on as well. And if you see here, this says mesh-wide MTLS is enabled. My window may be two, all right, all right. Let's bring it a little bit, three sizes, excellent. All right, so you see this is showing mesh-wide MTLS is enabled and that was because of the peer authentication that we created. I'm also going to go a little bit of the last three minutes. This is no different view than what many of you maybe are familiar if you're using Kiali. What is slightly different here is this catalog that you see between the communication between services, you know, you say front end, there is cart service, you know, there is the ad service. And when I click on this, you know, you see that, you know, there is two different principles that have generated automatically. One, the source principle and the other one is the destination principle. These are S-weights that we generated as part of the identity for this specific service, essentially saying that this front end will be identified by the trust domain, namespace, the service account that it is used and the service itself. And that's the source principle which is being used to originate the request from front end to cart service which has its own S-weights and own identity. And all of these certificates are actually issued by, in our case, WART. And we'll look at, we'll inspect the certificate resource and see how that looks like. But basically all your, what you're seeing is my application that I deployed has been deployed. There was an automated way of issuing, managing certificates for all of these mesh workloads. And each of these mesh workloads have its own sort of identifier, a SPIFI S-WIT that is used as a way to generate the certificate via the STS-SR mapping it back to the organization's CA infrastructure even though I used WART it could be your organization CA infrastructure that is facilitating that. The Ingress Gateway itself, as you saw, is something that is issued, that is a certificate that is issued by Digisert because that's what I used as part of the configuration. And also in my case, I'm using really short-term certificates even for my Ingress certificate you see it's actually valid for only three days. And you might be wondering what could be the validity for all of these mesh identities. And if you look at that, but basically all of these identities that are generated for the S-Wits that were generated for all of the mesh workloads are actually valid only for one hour. So it is the responsibility of cert manager to continuously make sure that these certs are up to date. They are renewed automatically. They utilize the mechanisms of the automated mechanisms that you're already familiar with to renew the certificates for the Ingress but also for the mesh identities itself. So that was what I wanted to show and I'm gonna just quickly run through some of the commands as well. I did see one question come in that might be a good time to ask is cert manager, thank you Krishna for bringing this up. Cert manager, rotate certificates. Do you know how Istio deals with the root cert being rotated? Yeah, so in this, if you're managing, if you're managing Istio on your own and basically utilizing its self signed capabilities then it's your responsibility for rotating the root CA of the CA issuer, but if you plug in cert manager then you're essentially delegating that aspect of rotation and management of all of the certificates that are used for MTLS by cert manager. So there is obviously depends on what you're doing but if you are using the self signed capabilities of Istio itself then you're doing it but if you delegate it to cert manager, cert manager's responsibility to rotate that. Basically the cert that needs to be rotated is two faults. You know, one in this case, this certificate also is valid very for a short time so this will be automatically rotated but this is an intermediate but in the case of this specific scenario the certificate itself, the root CA is actually in vault so there will be mechanisms within vault where that certificate will be automatically rotated. So you're essentially outside of the Istio world where your root and the trust is managed outside and then there you have the ability to sort of manage it as part of your, any other certificate, root certificates that you manage on root multiple roots or all of your intermediates. So we essentially sort of separate that aspect of where your root and intermediates are managed and where your workload identities are managed itself. But just by sort of plugging in Istio CSR as a way to manage all of the identities you're essentially delegating that aspect of root CA management to whoever manages your PGA infrastructure. Hopefully that answers your question Krishna but I'm happy to elaborate as well that doesn't answer. Please keep the questions coming in if there's anything that you're looking to learn or just kind of wanna start a discussion upon a specific topic. Feel free to throw that in the chat and we'll get those questions asked at the best possible time. Yeah, thank you. Yeah, and then as I showed here basically even my intermediate is actually valid as you see here for only one hour so that's essentially how long that is valid. And if you notice what I did, I'm running CMCuttle which is the cert manager CLI too. Very helpful, very handy if you want to inspect your resources, inspect your certificates. In this specific scenario in addition to also generating an SWIT for this intermediate we also have a DNS name. But when I show you the actual workload identities you will notice that none of them have any subject information. They are all pure SPIFI SWITs, SWIT certificates that we generate. And then I can run that. Long command to run. I've made it part of my make files. So basically what you're doing is I'm just looking or inspecting, running a JSON path to pretty much describe that status of a certificate. So if you look at what I'm gonna see is a certificate that was issued for the Ingress Gateway up there. Where did I run it? Yeah, there is my command. Basically, this is the certificate that we are looking for the URI SPIFI fan that says that Istio systems, Istio Ingress Gateway service account and you'd see that this doesn't have a DNS name and anything like that. And that's typically similar with something like checkout service as well. So this is the identity of the checkout service. Again, if you look at the validity period, they're all valid for only one hour and cert manager automatically, if you don't specify any renewal before as part of the specification, will renew a certificate two thirds into the duration of a certificate. So if we hang around here for the next 30 minutes, all of these certs that are issued will be automatically renewed and there will be new sets of certificates that will be valid for the mesh identities and that is the responsibility of cert manager to automatically ensure that every cert is renewed, every cert is up to date and every cert is available for the mesh identities so that your application that's running here continues to operate exactly as it operates. And then as you sort of look at things, look at operating your clusters so you don't worry about managing the CA infrastructure as part of your Istio but essentially let Jetstack manage that for you. That's what I had showing for a lot of different things but I'll try and see. But I mean, just to summarize to one of the main reasons why this demo came together is, because a lot of people I imagine that are familiar with cert manager will be mainly familiar about the way it runs Ingress gateways and configures the Ingress capabilities in terms of workloads and where we're seeing a lot more traction and usage with the project is around being able to secure part to part and service mesh context around MTLS and that's driving, that's really driving a lot of the usage that we see in cert manager generally and what we were showing there were different interesting contexts around how MTLS is particularly being used with cert manager. Let me see another question from Satish and I'm, Jailor is okay to answer that now or did you have something to tell me? Say, I'm an on-prem managed Winify, not the cloud one. We also have Azure, Istio, Kyverno, build a solution fixed into the landscape, absolutely. So it doesn't, you could pretty much use the same mechanism that I mentioned now and very often some of the large enterprises who use a platform essentially to manage like a platform like Winify. In this case, you will have the CA issuer or a policy folder that you create in Winify catering to an Ingress. You could also have some sort of a sub-CA or a private PKI that you're configured as part of another policy folder, facilitating the signing of mesh identities, absolutely possible. You talked about Kyverno, yes, absolutely. If you're using that as a way to drive your policy framework and policies, the Certificate Request Policy Resource that I mentioned, which is part of the Cert Manager add-on as a policy approver can be integrated with Kyverno so that way you have a first-class way of managing policies that you already have and also tying it back to the Cert Manager use and how you essentially use Cert Manager. That was, yeah, so you're already using Istio, so I'm assuming you're running Istio on Azure and from the perspective of what I've shown just now, we are absolutely cloud agnostic. We are absolutely service measure agnostic as well, so Cert Manager and all of the components essentially run in any Kubernetes distribution. If you have any questions, please feel free to ask. I do have a couple here as well to kind of add to that conversation. Workload misconfiguration is often reported as being a repeated problem for many cloud-native platform operations teams leading to outages and even security risks. What's your take on the causes of workload misconfiguration in the first place? Yeah, Citroen, go ahead, yeah. Yeah, absolutely. So I think, thanks, Tyler, for the question. So tying back to some of the things that we just showed and if you sort of flip that a little bit and let the platform team sort of manage application deployment, development, all of that and also be responsible for, let's say, security as one of the things. And very often you see that people utilize configurations that are out of the box. And in the case of something like Istio, since we are in that context here, there is already a framework that is available. So people generally sort of say, all I need to do is check this box and then check the box and then I don't have to worry about it, but the challenge is somebody needs to have the ability to rotate those search. Somebody needs to ensure that it is always valid. And the other piece is also from a large organizations or enterprises which are regulated, they always have to play with some sort of a compliance. So there is regulatory compliances that needs to be adhered to. There are guidances that people follow, especially from a security perspective, whether it's the NIST guidelines or the cybersecurity guidelines that they have in place. So when the misconfiguration aspect essentially comes into play when very often we don't see that synergy between the security teams and the platform teams and the drive for things where everything needs to be driven from an organization security perspective. And that essentially translates to sometimes people saying, what's the best way for me to get a certificate? I can run open SSL, it's extremely simple. Or I can get a certificate with my email ID attached through some way and then it'll run, but nobody else knows who issued that cert, which CA was the one which was issued. So those misconfigurations and those configurations rather at the time work best over a period of time will cause a drift in some way and essentially cause an outage. And as we all know, if this application that I'm showing right now, if this were some critical business application, it doesn't take a lot of time before users start to complain about it's downtime and it's impact on the business and things like that. So those are some of the, obviously the reasons why we see certificate misconfigurations purely because the common way or the standard way of using sometimes just doesn't work. You just need to adhere to the security best practices and these are some of the things that we're driving from what we do. And we've seen what we've just heard they're born out in research. So I mean, perhaps a lot of people may have come across the red hat state of Kubernetes security report that came out last year, which asked that very question in terms of how many of all the enterprise that they surveyed, how many were experiencing a security related incident related to production Kubernetes. And 94% of them came back and said that they were of which misconfiguration and audit failure, runtime issues were all highlighted as being areas where that vulnerability had occurred. And given the complexity in scale that we're seeing across the whole ecosystem really in terms of companies that are deploying Kubernetes, a certificate is gonna play a primary role in all of those use cases. So it becomes quite foundational, we think, in terms of being able to have those, that level of best practice around certificate management in particular, to make sure that there is sight of where these misconfigurations can potentially occur. And so they don't, they don't manifest a security vulnerabilities, but that security report that you can Google on the red hat status Kubernetes security port. So it's got some pretty stark data. It's a very interesting, very interesting report to read. Awesome, thank you so much. I think one thing that I had a question on, I've really liked cert manager myself, I use that with many teams and just like that automatic ability to add certs and just kind of make things inherently more secure. In the past I've used let's encrypt and the vault methods, but I am curious are there any other clouds or is there any other support on the roadmap currently for, you know, for AWS and their cert manager or the other various clouds and everything on that front? Oh, that's a great question, Taylor. Yes, so the way we look at it, if you look at cert manager, you're absolutely right. The ACMEsure with let's encrypt is extremely common that we refer to as there are, there is a notion of out-of-the-box issuers that are available in cert manager. ACMEsure being one of them, Walt being one of them, Venafai being one of them, obviously CAsure, Self-Signed, they're all built into that. We also have community issuers. So recently there has been a development and there is an availability that people can use for what we refer to as an AWS PCA issuer. So that is being developed, that's available for people. If what I showed right now is something that is purely have to be used in the context of workloads running in AWS and people who are to use AWS PCA, there is an AWS PCA issuer similar to any other issuer that you would configure and the workloads will be signed off of that issuer. And very similarly, there is also a Google CAS issuer. So you can configure Google CAS issuer built jointly with both obviously Google and Jetstack. And similarly even the AWS PCA issuer was built with AWS and Jetstack. So there is the cloud specific or a service specific certificate issuer capabilities that are built into CertManager. So you just need to install those additional components and that will provide you the ability to interact with those certificate authority services that are native to the cloud provider. That's already available, yes. I think that's fantastic because I'm sure that anyone who's worked in the industry for a longer period of time I'm sure can remember the days where you'd pay 200, 500 or more dollars per certificate and then almost always would forget about it when it's time for renewal. So very happy we don't have to do that anymore. It's kind of wild. Cause I mean, even now with CertManager you could even set up something where like on Mondays, let's issue this way and on Fridays, let's issue that way. So really, really cool to see that that'd be wild to do of course, but really cool to see that you have a lot more options in terms of configuring or if you've other secure workloads or different kinds of concerns with deploying your applications, you know, you need this level of search or et cetera, et cetera that you have a lot more options on that front. So really cool, thank you. That's a, I've got to check that out. That'll be, need to take a look at. I did have a few more questions here too. And again, please, if you have any questions please feel free to throw those into the chat and we'll be sure to get those asked. A lot of engineers operating in their Kubernetes platforms will feel they have good observability of certificates in a cluster using a conventional dashboard. Kind of that same problem that we saw before, you know like is my cert gonna expire soon? How should people use these dashboards to monitor cert manager? Yep. Richard, do you want to take that? Do you want to answer? Yeah, I mean, I think it's a question for me in terms of the role of the person that is in control of the dashboard. I mean, is that person responsible for the end-to-end security about how the workloads are being managed amongst everything else they're doing. So dashboards could be, to my mind very, very sort of busy in terms of the data but they're providing a snapshot of what's going on within the cluster at that precise moment. And that isn't always going to give you the depth of information you'll need and able to be able to fully identify where a vulnerability might be at risk of manifesting. And for that element, you know, there are other tools I think that you can use. I mean, JetStack is very much invested in, you know in that area of product if you like. But yeah, essentially I think that it's a question to down to the capability to dashboard and the role of the person that you're relying on really, because from a DevOps perspective there's a lot of information across those dashboards and the security aspect of it could be requires a certain level of context, I think to be able to properly, proactively make sure that those misconfigurations and those vulnerabilities are being mitigated. Which is really helpful. I think that it's important to have that observability and to monitor and to kind of get a handle of what's going on within your systems. And I like what you said, you know, just about the role like what's the problem you're trying to solve? What are you trying to do? I think that that's always really important to kind of have an idea of before you start looking. You know, if I don't know if I'm looking for my car keys how do I know what I'm really looking for, right? So that's a great point. And, you know, a cert manager makes a lot of these things automated as well. So that worry about certain contexts kinds of either fades away or is less of a concern. You know, if the controller has a problem or something like that then obviously we'd want to take a look, but. Yeah, I think just to add to that, I mean, audit capability is a context that isn't necessarily the primary kind of focus of a DevOps engineer if you like, but that level of observability for audit is needed elsewhere in the security organization, of course. And of course, you know, it's a question whether or not a dashboard is the right tool to be able to deliver that kind of information to inform security policy, for example. Personally, I prefer things delivered by carrier pigeon but that's just me, that's just my personal preference. I did see a couple more questions dropping in the chat. Abhishek, I'm a student, want to make a career in CNCF. Fantastic. If there is anything that we can help out with definitely would love to be of help. I know that we support a lot of projects and workflows and clouds. So if you do want to just take a look at that l.cncf.io will kind of show the whole landscape. And I believe that some people have actually made a puzzle out of that graphic. So I don't know how many pieces. I think it's like a thousand or maybe even a hundred thousand who knows, but really interesting to check out. Uncur asked, sorry, joined a bit late. May I ask if this works with Azure Kubernetes service, AKS, that managed instance? That's one I think I can answer. Yes, one thing that's really nice about cert manager is the fact that so long as you're using Kubernetes you can install that within that context. So no matter what cloud, even if it's something that's on-prem you could integrate that with Let's Encrypt with Vault as we saw a little bit today. And you can check the session out afterwards too. We make it available on LinkedIn, Twitch and YouTube. So you can kind of take a look at that demo if you join late and miss that. But are there any questions that you all get in terms of that? Do you support this kind of workflow or do you kind of plan anything outside of Kubernetes at that as well? Have you seen people use this in non-conventional ways to issue their certs? Yeah, it's an interesting question because recently I was engaged in one of our prospects of the customer now. The way they use cert manager is pretty interesting because cert manager is their source for issuing and managing all of the certificates. But they do have different consumers, consumers that are not always using Kubernetes because there are consumers who are working in classic applications, possibly requiring certificates for some load balancers, for some patchy instances or engine X instances that are running. So there are a lot of different consumers who basically need a certificate for their applications, but in this specific scenario, so basically what this person, I would classify him as an advanced cert manager user, has built an operator that essentially pushes and makes available a certificate that is generated, managed by cert manager to any of the consumers who are operating in non-Cubernetes world. So there are people who use cert manager in very creative ways, but we do get asked about, obviously questions around how do we sort of do this, where service mesh itself is not running in Kubernetes. I don't know how many customers or how many enterprises are trying to run service mesh in a non-Cubernetes environment, but that's something that has come up a couple of times. I don't think we do anything outside of Kubernetes at this time, but that's where we are most of our focus or almost all of our focus, but there are people using cert manager as a way to cater to consumers outside of Kubernetes. So that's very interesting to see. That's always fun. It always intrigues me to see people use things in ways outside of what was previously thought of, not following that like happy path kind of development. So, and in some cases, it leads to some really cool development. So that's really interesting to hear. I'm also always fascinated to hear about service mesh used outside of Kubernetes and really just what teams are thinking on that front. So that's really appreciate those insights. That's fantastic. So Satish, I did answer that question a little earlier as well, because it's also the separation of duties. Yes, you're right. So if you are using the built-in capabilities of more of a service mesh, you have to restart the Istio parts. The fact that all of your CEA configuration and the CEA aspect of signing mesh identities and assigning your ingresses, they're all outside. So you pretty much operate in a way where there is Istio and that there is the infrastructure to support the security aspects of what is managed within Istio. So we do have enterprises, large enterprises using this in the context of ensuring that there is zero downtime for all of this. Yes, it is possible. Awesome, awesome. I did have one more question to kind of round things off, but do you want to encourage anyone to get in their last minute questions as we start to wind down the stream? What's happening at KubeCon this year for the circuit manager team? I know we have KubeCon EU coming up here soon and may definitely book your travel if you haven't yet. I saw the hotels were already starting to fill up. I got my reservation, but I'm about, I think like 47 minutes via public transport to get there. So I just got to grab breakfast a little bit earlier. So looking forward to seeing all of you there. But yeah, what things do you have planned? We have quite a sizable team going to be there this year. So we're lucky enough and thanks to the CNCF for giving the project its own booth and the project affiliate. So for people that are attending, they can come into the project booth and they can talk directly to the maintainers. We've got a pretty good representation from people that work whose full-time job it is to maintain and develop the circuit manager project. So definitely encourage people to come along and say hello there and stop by the jet stack stand as well because we'll be there to talk about everything else that we do in relation to our open source work and everything else. We've got talks going on at the main conference as well as a couple of the co-located events. So yeah, it's gonna be a very busy week, but you'll certainly see a few of us there if you're attending to go along yourself. Awesome. Is there any, and another question, I said there was just one, but I had another one, in terms of the, so it's jet stack, but I've heard rumors about you having jet packs as swag at your booth. Is there any confirmation that you're not on with that? That's a great idea. Why didn't I think of that one? Wow. That's a great idea for your return. We're talking about cloud native development, right? It's all in the sky. Yeah. I'm gonna have to look into that one, Taylor. I think it's a great idea. But, and then I'll have to like, look at my budget too, of course, but. I've heard a lot of crazy announcements around Friday, April 1st. I don't know what the relations are on that front too. Awesome. Yeah, the guy that does the gravity pack, you ever see him? I don't know if you know, but. Oh, yes. His company's not too far away from my lips. So I'll nip down the road and ask him if I can borrow it for a week about that. So. That'd be so funny. Awesome. Awesome. Well, I don't see any more questions today, but I did want to thank everybody for tuning in to our latest episode of cloud native live. It was great to learn from Richard and Citram. We really enjoyed the interaction and questions from the audience. Thank you all for showing up. This is you, you, the community make this possible and really fun, really enjoyed your questions. Next week, we're going to be joined by Andrew McGuire to learn about how to power up your machine learning with automated anomaly detection. So that should be fun in terms of monitoring and everything else. I want to thank you all for joining us today. We hope to see you again soon and wanted to ask you if you have any, any last parting words of wisdom, either of you. Well, I mean, we're, we're a very popular open source project. We've got a great website that's just being refreshed. We'd like to encourage everyone to check out certman, cert-manager.io, a great resource. Our Slack channel is very, very busy as well. So I would encourage people to join the Slack channel and or come to Citram and I directly if you've got any other questions as well. Yeah, please do that. Absolutely. Yeah. Happy to answer any questions, not just in this form, but yeah, reach out via Slack or you know, on the Kubernetes Slack channel. So we will have to answer the questions here. Fantastic. Fantastic. Well, thank you both so much for coming on today and for talking about cert-manager. It was fun to get those questions answered and you know, get demos and get a little bit more insight into the project and fantastic. I'll definitely make sure to see you at KubeCon. And yeah, thank you. Thank you everybody for tuning on in. We'll see you again soon. Thanks very much. Thanks everybody. Bye-bye.