 Hello, KubeCon and CloudNativeCon. Hi to everyone. I'm Shweta Bhora. I'm an architect, inventor, blogger, AWS Community Builder, open source enthusiast, and contributor at time. Before I start with the session, I must say what a phenomenal week this has been. We all have gained, observed, and interacted so much. It's time after today to go back and reflect back what we have gained in this amazing KubeCon and CloudNativeCon. Before we go back today, it's time for this session and it's about, you like it or not, you need it, PKI and Certificate Management. Let's get started. The contents which I'm planning to cover in this session are vocabulary and refresher about PKI and Certificate Management, vocabulary for those who are new to this topic, and refresher for those who might be aware of, but they might learn a few things new from this. We are also going to talk about a case study, which is common yet complex production case study and a demonstration and we will be ending with the five PKI design decisions that you must know. Okay, so here we are with the vocabulary and refresher. PKI, Public Key Infrastructure. What is PKI? Let's understand that. Always when you communicate over network, you have two endpoints. This could be client and server. Now these endpoints talk to each other. Let's say you are using a internet banking or a bank site where you are doing internet banking and this happens over secure socket layer or transport layer, security is required and that protocol is used. What this client and server, how they trust each other and what do they need? There are two things primarily, any endpoint or two endpoints need from each other so that they can smoothly talk to each other. The first one is the trust establishment. Client needs to know that the server I'm talking to, let's say the bank site I'm talking to is secured enough that I can share my data and information. And once the trust is established, you need to exchange the encrypted data over network. Now this happens day in and out when we interact over network. But what happens behind the scene? How does PKI come in picture? Let's look at that. To have that trust established, your server needs some authorities, the first and foremost the certificate authority, registration authority and a verification authority. What the server does is that it requests the registration authority to issue and assign certificate to the server using which it can give trust to the client that yes, it has a digital certificate and it communicates encrypted data over network. In turn, registration authority takes that request and takes it to the certificate authority who issues that certificate to you. The certificate authority signs that certificate for you and gives you two things. First, it gives back to the server a certificate and a private key. Now this private key remains with very few entities who need to have hold of that. However, on other hand certificate authority also gives a certificate plus public key which can remain anywhere available on the network because it is public and verification authority keep it with itself and can utilize when needed. Now, what happens when client needs to a browser? Let's say for instance needs to see that that this bank site is a valid site and I need to trust on that that client can request to verify the signature from the verification authority and verification authority replies back saying that signature is okay or not okay. Now once this whole thing gets established at the back or the behind the scenes, then client uses the encrypted data or sorry key public key to encrypt the data and on the other hand server decrypts the data by using the private key. Now in between there is there is no one who can possibly there is no one who can decipher that data because public key is available to all but to decrypt the data you need that private key and that's what public key infrastructure provides you. Let's and one more thing before I move out of this slide that certificate and DB store because this whole thing remains somewhere on these servers which we have seen and this certificate DB and store gets utilized. So what does PKI provides you? It provides you cryptographic identity to the endpoints before I move on, I want to emphasize on that it need not be all the communication over public network. It can be any endpoint. For example, one cluster talking to other cluster one microservice talking to other microservice. So it can be any endpoint and you can utilize PKI for communication over network be public or private. So you have to have a cryptographic identity. They need encryption of data over network so that the data remains secured. How it provides PKI provides it by governing issuance and management of digital certificates and key components for as we have seen in the previous slide, you need digital certificate you need certificate authority registration authority verification authority. Sometimes these three authorities may be same or different depends on situation to situation and also you need certificate database or store which can be referred as needed. So PKI is a framework, not a specific technology. Please keep in mind. It provides you all these things by utilizing certain digital certificates and infrastructure to achieve that. Why moving on? Certainly cryptography because public and private key difference is something which we want to point out here and we need to understand what is this cryptography cryptography is the sign of secret writing with the intention of keeping your data secret. This cryptography or the algorithms which get used are classified into symmetric or asymmetric cryptography in symmetric cryptography. What happens? You have a private key which gets utilized for encryption as well as for the description for the description of the data. Now the same key needs to be very very secured and that's why it's called private key encryption or symmetric encryption. On other hand, you have a symmetric cryptography where you have two different keys. One is public and one is secret key or the private key which remains only with few entities and public keys publicly available. Now symmetric cryptography is simple so faster but less secure. However, on other hand, a symmetric is comparatively slower but tough to break. And there are algorithms like RSA, Diffie-Hellman, ECC algorithm and many more. Now PKI uses public key cryptography framework not the private one. So that's why public key is the name given to the PKI. Moving on, to all this is your gist is the digital certificate. Now let's understand what is a digital certificate. A digital certificate is a digital document that finds the identity to a public key infrastructure. Now you use it at many places in internet protocol, including TLS, SSL, which is the basis for HTTPS, the secure protocol you use for browsing the web. For example, if you use any browser, you see that there is either a lock comes or an open lock also at times come which shows you that it is secured or not secured. Now digital certificates you need to generate. How do you generate these? There are many options that are widely available and those options adhere to the international standard, which is X549 standard. Some tools which I'm listing here is OpenSSL, LibreSSL. You have third party RAs and VAs like Let's Encrypt, Digisert, etc. You have also cloud services these days providing, for example, AWS certificate manager, Google certificate manager, and there are many more which provide these digital certificate services. Now, what is the key activities around these digital certificates? Because it's not that you just issue it once time and you can use it forever. It is there is generation and issuance which is required, distribution, deployment of those or installation of those certificates. That's when these certificates gets in use. At times you need to revoke them when there is certificate gets hacked or due to some changes, you need to pull it back. You also need to renew and rotate these certificates time to time and last but not the least certificate scanning and monitoring is essential which comes under the digital certificate lifecycle management. Why we are moving. Let's see some sample certificates. This one is from Amazon.com. It has on the on the windows it looks like this and you can run certmanager.msc to see on the Windows box and command prompt to see this kind of certificate. I'm showing a simple certificate for Amazon.com where it is issued to Amazon.com issued by the dessert. It has period of validity. It has some hashing information associated with that. Now, this is on Windows box. However, on the next box, it looks like something like this a text file. It has a now again a issuer a subject in this case, both C and C in stands for common name. Common name is same. It means it shows that it is for a it is mostly a certificate authority or a root CA. It also has public information as we discussed that public key information is easily available and accessible. It also comes bundled at times with your certificate information and the last here which I want to show is that subject key identifier authority key identifier in this case if you observe this is same. However, if your certificate is signed by some other authority then these authority key identifier will differ from the subject key identifier. So that's how our certificates look like. Okay. What are those files? What are those encoding formats and files which certificates come under X549 certificate? There are various encoding formats. For example, PEM, P-K-C-S format, PEM and there are more but PEM being the popular one. I'm showing the files which it produce these green boxes as you see either the dot PEM or the dot cert or dot cert file which it produces are the certificates and dot key which it produces is the private file and it looks like something encrypted data between for example, if it is a certificate file, then it will be begin certificate and end certificate. On other hand, if it is a key private key, it will be written like begin private key and end private key. Now, how to how does these certificates get generated? There is always a request certificates signing request which goes from a server or an endpoint in a form of dot CSR file and this request CSR stands for certificates signing request. This file looks like begin new certificate request and end new certificate request. So by reading this file simply opening that file. If you see this understand that this is a certificate signing request. The certificate signing request goes to an authority which could be root certificate authority or intermediate certificate authority which we will understand in a moment. Once this request goes to the certificate authority, it signs it digitally signs it. You get these certificate and private key file which you need to keep or install at appropriate places to make use of it. So before we move on to the next topic, there is one more important thing for us to understand that there is a hierarchy of certificates root CA to leave certificate and that's called the site certificate hierarchy or chain of trust. It can be single layer or multi-layer CAs. Why we need it. We'll talk about it, but let's see how does it look like? I've pulled a certificate for Amazon dot in where you have root CA digital global root G2 which is providing the certificate. Then you have intermediate CA which gets signed by this root CA intermediate CA then helping you sign with this server certificate. So like you have Amazon dot in Amazon dot UK Amazon dot us or whatnot. You can have multiple such server certificates which can be installed on those servers and you can view this hierarchy and that's where the chain of trust or the certificate hierarchy is gets built up for the actual environments or the pK environment. While that was about the vocabulary and the pressure let's delve into a production common yet complex production scenario that how this how does this pKi looks like in actual? Well, in all complex scenarios, you have either a corporate data center or a cloud or a cloud to cloud where you need to talk between these entities and serve outside also. In in all these communications, you have a compute layer which could be like your VM EC2s or a serverless compute. On other hand, you can have a cluster container orchestrator cluster which can be Kubernetes and to add to the complexity we have network layer which helps you talk to all the communication happening over the network between two different entities or sometimes different different domains. So this network proxy layer typically contains either API gateway or ELB or increased gateway. And what else? You have a service mesh these days. We have microservices which are talking to each other. So they can be various microservices which which reside in data plane. They talk through control plane to your external world and all this gets called at various points by internal users by application and interfaces by external users and is all the time communication happening between these systems. Now to make this communication secure what you need? You need PKI and digital certificates and where all you need you need all these yellow certificates which you see these are some simple places which I am showing. For example, you need to secure your API gateway. You need to secure your load balances. If you are using you need to have microservices and having your the those identity which is secured. Then you your easy tools or VMs need these certificates. Your cluster needs certificates and whatnot. Even your public websites need these certificates. So what we need to do? How do we make this secure? Now let's come to the what infrastructure or what PKI framework you need to establish around it so that this becomes a secure network with the digital certificates and communication happening with in the encrypted form. So first and foremost, you need for the external. So here I'm calling it external and public see it because for having certificate issues and maintain. So all your CA RA VA functionality at times is given by all these various tools or the service providers like old eddy let's and grab digits and whatnot. They provide these root certificate authority which issues certificates to your websites and other entities which need to interact over Internet. Then typically inside our secured network. We don't rely on these external certificates. So you need another root CA within your secured network could be a corporate data center or or it could be the cloud which is your private cloud and that's where you keep a root certificate authority which issues the certificates and it gives the certificates to the prime intermediate CA's or subordinate CA's also we call so for example. I see it could be your AWS certificate manager where you sign your this particular subordinate CA using this root CA and then AWS certificate manager issues those certificates on your behalf and distributes it to ELB or ingress gateway or whatnot. Similarly, a Kubernetes cluster might need a different IC all together so that once it gets its identity signed by root CA then it distributes these leave certificates to different applications as per need. Same is for your service mesh. There can be many microservices which keep on coming and going and there's a whole life cycle around those services. So you need some subordinate or intermediate CA which issues those certificates to your the microservices or the leaf entities or the endpoints. So once it is in place what you get you get all these certificates from these CA is signed installed and once this is installed your communication over the network is trusted identifiable and encrypted. So there is no chance that anybody can creep into your system and reads data and easily can take over your data. So this is what we have seen. You have a public CA which can be these tools like digital let's encrypt and on which provides and used for external facing website and interfaces. Then you can also have private and intermediate CA's reflected with these symbols in the previous diagram. You can use open SSL to issue these cert managers if he's fire and there could be many other tools. Then you also have these leave certificates which are the end entities or those digital documents which we've been talking about so far and these these are what provide the actual security to your system. Let's see a small demo which I have already put up in place but let me quickly show you that how this looks like. So I am moving to this folder or let me put it in format which is clearly visible. Okay. So let's say you have developed a root CA now this root CA is a self signed certificate and a key pair where nobody else signs root CA so root CA needs to be much more secured and you get CA dot cert and CA dot key. I have used root CA dot con to generate these files. There is some configuration. Let's say I'm using open SSL which I have used for this particular case it needs a lot of configuration. So all the time providing that configuration over command line. I have used it root CA dot con where I put all those configuration and I have generated these certificates and key now once my root CA got generated. I've got my intermediate CA signed by this root CA. So what I did I got these CSR generated certificate signing request using which my root CA signed it and provided me with a cert which is my intermediate CA certificate again because it is a certificate authority. You need a configuration a lot of configuration to be provided over the command line. So I have encoded it in this file. Now once these root CA and intermediate CA's are ready what I did I start started creating my server certificates or leave certificates. So again, I need a CSR this time my CSR got signed by my intermediate CA and I got my server certificate generated. So this is how it looks like but in no production case you will see it in at one place. This is definitely spread it across the authorities and the servers but let's quickly also look at the format of few of them. For example, let me show you open SSL give me an entering here. Let's say my second. Yes. Here I'm giving it my root CA and my CA dot cert. Yes. Okay. So this is how my certificate looks like as we have seen earlier also my issuer and my subject in this case is same. If you notice because this is a root CA and in root CA case only you should ideally see that your issuer and subject that same because it is self-signed root CA then you will also see that you have subject key identifier here and authority key identifier are same which means that your key here if you notice is exactly same again. The reason is this is root CA but if we see this for any other certificate, let's say now I'm opening it for intermediate CA and there is my cert file. Okay. So this time interestingly if you observe here that my issuer is different than my subject. My subject is ICA dot demo dot com this common name and my identifier subject key identifier is the newly generated identifier for this particular intermediate seed but authority key identifier is the one which we saw in the root CA. So that's how the certificates get generated but what happens when you have your Kubernetes cluster and Istio certificates would you be generating certificates like this for each and every microservices? Well, that's a lot of work and we cannot do it like that and that's where let's go back to our presentation where I'm providing you a tool that can be more but spiffy and spire are coming up really well. Where let's understand this piffy is secure production identity framework for everyone, which means piffy provides you a framework to get those secured identities and it's a framework now spire is a production ready implementation of these piffy API. So they usually you would see that spiffy spire word gets used together. These are both open source graduated projects. Here spiffy provides you with the facility like first of all, you need to register your workloads identify your nodes and workloads. So registration feature then Federation when you have let's say two different domains or two different clusters which is let's say one is running on Google. Another one is running on AWS or a data center to a Cloud. You need to federate those domains. You need to establish trust between those domains and that's where it provides you with the feature for the federating the clusters or the nodes and domains. Then it also gives you the feature to have your workloads identified and identifiable information given to those workloads and time to time it provides you the attestation of those nodes and workloads how spiffy provides it. It provides it by exposing the API's it to whom it provides these features to the actual workload and the node. So first thing comes as the node where your workloads run and these nodes get a spiffy ID by exchanging few things from your spiffy API's. So this spiffy identity looks like something like this spiffy where your trust domain name your namespace and your SS stands for service account which it belongs to. So that's how it differentiate that which principle it identifies to then you have your workloads. All the workloads running inside your nodes get a identity so secure verifiable identity is the name for that which spiffy calls it like SV IDs. These SV IDs are nothing but your certificates x.509 artifacts which it provides and how it does that it does it by having a spire server and a spire agent because remember spire is the implementation and spiffy is the framework of all that API is what it exposes. Now once that is in place what happens as I said first you get your spiffy IDs and some selectors along with it once that comes then your workloads get identified. So you need to have certificate signing request request you have to have to get those SV IDs from your again spiffy API's of the spire server gets generated through the same usage of these APIs and you get those nodes give CSR and get SV IDs and spiffy bundles which never leaves the node. So you have these information comes up to the node and nodes time to time distributed and gives it to the various workloads. Now doesn't matter workload go down or up and keeps coming. You have your secure verifiable identities and on top of that you need not worry about renewal let's say or the rotation of those certificates it it gets taken care by spiffy and spire through the all right. So let's talk about five PKI design decisions that you must know. First and foremost you should know how to design your certificate chain or hierarchy. It can be a single root CA authority where single root signs the your ICA's and other certificates and takes that ownership. It can also be multiple intermediate CAs to protect your root CA you can also have a subordinate CA or ICA which you are delegating to and this in turn signs the other size ICA's certificate signing requests. Other way of doing it in complex systems that you have a geographical or departmental or domain distribution where it is segregated ICAs at geography level. It can be segregated ICAs at your department or domain level which in turn take care of your leaf certificates distribution and certificate life cycle management. But to keep in mind that wise thing is to keep design your certificate hierarchy based on size of your organization criticality levels of security required and operational capacity you have because ultimately more the hierarchy more the burden you have second point where to terminate your certificates by termination. We mean that let's say if you're terminating your certificate at API Gateway then which means that beyond this point your data would be traversing it will be decrypted at this layer and data will be traversing in plain text. So it will not be end to end encryption but encryption will end here in most cases where it it can be used that terminate it at API Gateway load balancer or interest controller layer which means the network proxy layers or in other cases you might want to end it at compute computation layer which means that the VMs or EC tools or at the port containers. Now this depends on that how much of compliance how much of a data in transit do you have a requirement of end to end encryption requirements? Then yes, it makes sense to do it at compute layer but in other cases to keep it simple because it comes up with a lot of computation and performance requirements as well as the operational needs to maintain those certificates which you generate. So it's better that you ended network proxy layer because ultimately it then routes it to your internal secure network but look at your requirements and then design that what is your requirements specifically. Third thing to keep in mind the TLS version mismatches. So at times we have TLS 1.2 to 1.3 or TLS versus MTLS need to coexist. For example, in this instance, if you have to coexist with TLS and a mutual TLS where TLS is one way trust verification and on other hand MTLS is two way trust handshake. So sometimes you need to have both coexist. For example, in your in case of Istio service mesh, it automatically configures your workload containers or side cars to use mutual TLS when calling other workloads. So how would you coexist with these variations? So in such cases know the ways to exist. For example, use MTLS mode for peer authentication modes can be like strict permissive or disable where strict is that strictly you have to have mutual TLS to mutual TLS communication permissive could be that it is optional. It can be relaxed and disable can be that you can totally disable mutual TLS and keep it simple. So that's about the third point. Now fourth and important point that certificate revocation methods and design. So there are ways to revoke revoke your certificates and keep a list of it because when you do the revocation, you also need to inform various entities or verification authorities that yes, these certificates have been revoked. So how do you do that simplest way of doing is that keep a certificate revocation list whenever you are revoking a certificate and this can be automated or manual process where you keep checking whether any certificate is revoked and take action appropriately either manual or automatic. Another way of doing it in online certificate status protocol where web browsers keep a tap on that what certificates are revoked and they they change. I mean, whenever they get to know about this revocation of certificates, they maintain that list and they renew this I get the new certificates from these CAs. The the upward version of OCSP is OCSP steppling where browser browsers get this information rather than asking from CS get it steppled whenever there is a TLS handshake happening. Now why it is relevant because Sierra CRL design is often overlooked aspect of PKI design. So you should know how to design. Recommendation is for internal private CAs use CRL to keep it simple and use customer automation, but in public CAs when there is facility of such protocols, then definitely you must use of that 5th point certificate automation and monitoring again many a times unplanned area and to you need to keep a balance about what and how much to automate in operations. Few suggestions create certificate templates to keep it same within your organization on always adhere to certain policies and configuration utilize certificates rolling deployment to avoid downtime during certification renewal or expiry. You can use tools like spiffy inspire. You can also use tools for monitoring, tracing and alerting for better and easy certificate management. For example, Grafana gives you a nice little dashboard of certificate when it is about to expire or when it is already expired. Istio Kiali gives you this animation little beautiful animation where it tells you that which all communication is secured through TLS and certificates and which is not. So have your own mechanism, but tools can be any but utilize these tools for automating monitoring and enacting. Yes, we have reached to the end. We have a lot we have covered. Thanks for listening patiently. It's time for questions and please do leave your feedback by scanning this QR code. Thank you so much.