 Welcome everybody to this Cloud Native Computing Foundation webinar which will be delivered by Ori and I on just-in-time AWS access for Kubernetes workloads anywhere with Serp Manager on Thrice. A quick bit of intro, I'm a technical director at Venafire, a machine identities company and I focus on workload identities and Venafire is a company behind Serp Manager as the creators of Serp Manager and donated to the CNCF in 2020. I'm here with Ori. Ori, can you introduce yourself? Yeah, thank you. Hi everyone, I'm the CTO at Autorize. We build open source software for declaratively managing workload IEM on Kubernetes. These days I pretend to be an extroverted CTO doing webinars and such but before that I was just an introverted platform engineer. So here we go. Thank you Ori. I'm really looking forward to present this webinar together with you. It's a mouthful title kind of thing but we're going to try to break this down in easy bite sized pieces so you can definitely understand this. A bit on our agenda, we're first going to talk a little bit about AWS and Spiffy. We're then going to talk about Serp Manager and the Serp Manager CSI Spiffy driver, Autorize, you will see a real-life demo and a small recap. Before actually starting with all of the content, I want to say a big thank you to Jos van Leeuwen and Thomas Meadows, two former Fennify colleagues of mine who last year during KubeCon Europe in Amsterdam did a talk on Serp Manager Kendo Spiffy, solving multi-cloud workload identity using a defective standard tool. Really great talk. I really highly suggest to check this out, maybe even before sometimes watching this webinar if you're watching this in the recording afterwards. They will go really deep into what Spiffy is and how everything is being built up while we stay a bit more on the higher level as we want to show some other bits and pieces besides the Spiffy today and how this can all work together with Autorize. So a small bit on our honorary guest of today, it's AWS. So we are going to talk to AWS and we are going to use a service of AWS called IAM roles anywhere. It's a really great feature of AWS and unfortunately none of the other cloud providers kind of have something quite similar and what it allows this feature is to authenticate to AWS with an X519 certificate that you have minted yourself with your own tool. You mint your X519 certificate through a CA that you own, you give AWS the public keys of that CA so that there's a trust being established and then what you can do is you can exchange that X519 certificate for short-lived access credentials by using the create session feature of IAM roles anywhere and it's a very easy feature and that's what we are going to heavily leverage for this webinar. A small bit of recap as well on Spiffy. Spiffy exists out of multiple components but for the purpose of this talk we're going to focus first and foremost on Spiffy identities which exists out of three components or three parts. First of all you have the scheme of a Spiffy identity and in Spiffy that's just spiffy dot dot slash slash. The second part is the trust domain and a trust domain is kind of how you can mark your trust boundary how you can uniquely mark for example an environment so when you are an enterprise you might have different trust domains for for example production developments and these kind of mark your trust boundaries. Spiffy identities are being cryptographically signed in key materials. The two common key materials that Spiffy identity can be minted is as an X519 certificate and a Jot token. For the purpose of this talk we are only going to focus on the X519 bits but know that like the official Spiffy framework also supports Jot tokens and we're also a big part of that is also the attestation and validation of workloads. Before you can mint a Spiffy identity within X519 certificate you first want to validate that that workload is what it says it is and this can be done through multiple factors and later on this talk you will see for example how CERT Manager can help with these bits. This is where CERT Manager comes in. CERT Manager is kind of the effective standard for certificate management on Kubernetes. It's a really great tool to manage the lifecycle of X519 certificates within Kubernetes clusters. It's part of the CNCF it's currently in an incubating status as it has been donated by Vanavine 2020 but the CERT Manager team is working towards graduation hopefully later this year. It has currently more than 11,000 stars on GitHub and the main use case I think like that everybody's aware of CERT Manager is using it for securing your ingress resources or your gateway resources where CERT Manager creates a certificate that then stores your TLS certificate in a secret in a community secret that then can be used by your ingress gateway to be able to secure your web traffic. Let's Encrypt for example was one of the very first implementations of like CERT Manager and it's still like I think like the most popular used one at least I use it in my own home cluster. Always start with deploying CERT Manager and let's Encrypt every cluster. Indeed yeah like it's been used I think a 9 of the 10 clusters that I see but there are a bunch of other use cases as well of CERT Manager besides the ingress story first and foremost like it can do an integration with Vistio service mesh and some other service meshes as well to replace the built-in self-signed CA. So this is commonly done to improve the security by using external CA's and especially a lot of big enterprises they have their own CA's that they want to use and they don't want to use any self-signed CA's within their production environments. Hence CERT Manager can take care of this and it can replace that self-signed CA. It also makes it's easier to set up multi-cluster when you set up multi-cluster you need to start copying like the trust routes between different clusters this all goes away when you have like that common PKI that you can trust. Another bit that CERT Manager also has is you can build your own MTLS if you want so this either can be done by creating certificates that then store it's in a Kubernetes secret or by using the CSI drivers of CERT Manager to be able to give your workloads just-in-time certificates where you then can have your own create your own MTLS. For example common one that I have seen is lots of java Spring Boot applications that load in java trust stores. One of the other projects that CERT Manager also has is the trust manager and the trust manager is a really great project to get CA route chains public keys from many different sources and then give these to your applications so that they can trust those and this is especially great if you are needing to rely on trust stores and public NCAs kind of thing that are not publicly being distributed like for example indeed if you have your own PKI in your own company like this is a great way to distribute that towards all of your applications without needing to embed these necessarily inside your container images but you can then like easily update them without needing to update your container images and the purpose of this talk as well today is using spiffy for or issuing spiffy identities with CERT Manager and that's where the CERT Manager CSI driver spiffy comes in. It runs in so the way that this CSI driver spiffy work and kind of like how every CSI driver works first and foremost is it uses the tempfs file system on each node and with that like the private key material of the certificate that get generated never leaves that node it's like in memory and it's only there during the lifecycle of your Kubernetes spot and any Kubernetes workload so from the moment that Kubernetes spot has gone that key material is also gone because it was in memory and it's never gone. It uses the CSI token feature request CSI token request feature of the CSI driver and what this allows you is to discover the pot identity and it's going to then form the spiffy identity based on the discovery of that pot identity and embedding that in the X519 certificate signing request and the CSI token request is also to impersonate the service account so that like you cannot do privilege escalation and it does extra security boundaries and this is especially relevant for CSI drivers like the spiffy one but also like for example when you use this for the CSI secret store drivers for TCP and OIS like it uses the same feature because that's extra security on it and then the service is CSI driver spiffy comes with a second component which is the approver and this does some extra checks besides validating the spiffy identity it's going to check if the acceptable key usages are being set which is key and cipherment, digital signature, client off and server off, a request duration which matches the enforced duration so by default the spiffy identities are only going to be one hour old and it's automatically going to renew those that no sense or other identifiable attributes you can only set a single URI send and URI send needs to contain a spiffy identity of the service account which created the certificate request and you also have a spiffy ID trust domain matching the one that was configured at startup so like at startup and you will see that later in the demo we give a trust domain that's the CSI spiffy driver like it's going to match that with and then approve it and then only then the certificate will get issued by our CA and now I'm going to hand off to ORI who's going to talk a bit about authorize. Thank you so I mentioned earlier that authorize does declarative IEM for Kubernetes workloads and we're here to talk about how to do cross cloud AWS IEM access but before we dive into that I'm going to do a quick workflow of the different authorize components and roles in this so what is authorize? Authorize achieves three things with its open source components the first one is credential provisioning automation so to gain access workloads need to be identified and authenticated on AWS this happens for an AWS role on Kubernetes network policies for labels with databases for usernames and certificates and all of these different resources need life cycle management which the authorize credentials operator handles an open source project you just you just label a pod and authorize will take care of keeping all the resources and credentials necessary for authentication in sync and then you have IEM policy automation that means the creation and life cycle management of IEM policies like those on AWS, Microsoft, Azure, and Google Cloud but also the Kubernetes network policies themselves Istio policies and even access to databases this happens in an open source project from authorize called the intense operator a Kubernetes operator that you deploy on your cluster and finally we have mapping actual access so knowing that you have derived policies in place for intentional access to succeed at deploy time can be difficult when you have complicated permissioning systems such as AWS IEM or network policies where the existence of one policy can affect how another policy behaves the authorize network mapper a third open source project uses existing traffic in your cluster to auto-generate the required declarations for access to succeed allowing you to compare the declared access with the actual traffic so let's see how we actually take all of that and put that and how that works in in this specific case so in order to achieve access to AWS IEM we're going to label a pod to tell the credentials operator that we wanted to create an AWS over the credentials operator will then calculate what should be the pods spiffy identity and create an AWS IEM role and IEM roles anywhere profile and then add a CSI volume to the pod referencing the pods spiffy identity the profile and role the third manager CSI driver will then pick that up and create appropriate x509 certificate and create the temporary credentials that Matias mentioned earlier which will be able to find all of that inside the CSI volume so now we've got a pod and it has a role and it has a profile and credentials and it's able to authenticate the next step is we need to apply client intents client intents is an authorized custom resource that you apply to your cluster where each client workload declares which access it needs so once you apply to your cluster in the same name space as the pod the resource tells the intents operator what access our pod needs on AWS then the intents operator uses the client intents to create a matching AWS IEM policy and associates it with the IEM world the credentials operator created in the previous step and then once we did both those steps we have cross cloud access so I'll hand it back over to Matias for us to see a cool demo of all of this in action cool thanks so for the purpose of this demo as like we already know we're going to connect to AWS I have set it up a Kubernetes cluster in Google Cloud you can see this is a GKE cluster it runs version 129 of Kubernetes and to show I'm not cheating I actually have opened this up it's this cluster that you can see so there is nothing in AWS running so we are going from Google Cloud to AWS and the first thing we need to start with is we need to deploy search manager and this is what I'm going to do so this is a simple hell of install that we're going to do of search manager and we are going to add one extra thing besides the normal install of the CRPs and that is that we're going to disable the default certificate requests approver of the search manager install so when you do a normal install of search manager by default all certificate requests are getting approved automatically by search manager which is really great behavior the thing is that our CSI speed driver is going to do its own approver so we don't want to have like two conflicting things approving certificate requests so we are disabling search manager one search manager also has some external certificate requests approved for if you have like for example a combination of multiple issuers later down the line that you can use them together so that like for example the CSI driver only approves the CSI driver issuers and then you might have like another certificate request approver for for example like your ingress certificates which might come from another issuer the second part that we're going to do is we are going to deploy an issuer and we're going to get us a CA kind of cert so in this case we are going to use a self-signed issuer this means that it will self-sign its cert and this is especially great like if you for demo purposes if you want to have a CA and you can see like we're going to create this a CA here in the spec it said a CA is true it's quite long lift normally it will be short lift and we can also see that we have set the common name to spiffy.certmanager.io which is also going to be our trust domain and we are going to get this signed by the self-signed one and then later down the line we're going to create another cluster issuer that will contain the secret of that CA you can see this match with like the the secret name of the certificate that we're going to request and that's then what we're going to use to issue spiffy identities later down the line so i'm now going to apply this and because I have disabled the automatic approver of certificates I now for only this kind of CA I need to manually approve them luckily with the CMCTL tool which is the certmanager CTL tool I can approve this manually and I'm going to do this now and what we're going to see now is certificates all namespaces we have our certificate ready and it's all up and running so that's the first part down we have certmanager running we have a CA in our cluster now we can set up the CSI spiffy driver and for this I have prepared a small YAML file which will be our values of YAML file that we give to Helm you can see our trust domain is being set here we also refer to that issuer that we have created earlier and also for our approver bit we also need to reference the same issuer there I also need to make a small change in the images that we're using because like currently the AWS features are not part of the upstream certmanager CSI spiffy store driver I need to reference this currently to a second one I'm currently working on trying to see how we can get this into upstream as well but like what this will do is when a pod comes up with the CSI driver spiffy one it will automatically exchange that x519 certificate that it gets just in time for access tokens that can be used to authenticate the AWS so that's the main difference it has like that great session that I was talking about earlier inside the CSI spiffy driver so you don't need any extra helpers like from the moment your pod is there the credentials for AWS authentication are going to be there and they will also automatically be refreshed by the CSI spiffy store spiffy driver which is really great so I'm going to is it yes this one need to find the command so I'm going to do help upgrade and do the spiffy driver install this is pulling it and what we're going to see is that our spiffy driver is going to come up so we have the approver one so that's going to be the one that's going to approve the certificate request later down the line and then you can see like we have three pods with the CSI driver spiffy this is because my cluster exists out of three nodes and we need to have like one pod per node to be able to work as a demon set as I said like the CSI driver handles everything in temporary memory so that's why we need to have like that demon set up and running and then we're going to do the AWS bit sorry yep so we're about to deploy other eyes and before we do that we need to do some setup on AWS so that other eyes can go and do all the things that I described earlier create IAM roles IAM roles anywhere profiles and policies another thing that the terraform template will do is it's going to retrieve the public key of the self-signed CA we created before and define it as a trust anchor for IAM roles anywhere as well as configure the IAM policies and roles for the authorized credentials and intense operator themselves which will be running on our Google cloud cluster so go ahead and deploy it yeah and it's deployed and we're going to be able to see is when I do refresh here we can see this is the authorized cert manager spiffy so the trust anchor is now in IAM anywhere we also have two profiles in here set it up and this reference is the authorized intense operator role that's then like has the IAM in there and it also has a trust relationship in there maybe you can talk a bit more about that Tori yep so the trust relationship we see here is actually as you can see it has the spiffy uri for the intense operator so you can see it's the trust domain we set up earlier in the namespace authorized system which is where authorized is going to be deployed on our google cloud cluster and then the service account the intense operator controller manager and below that in the aws source arn you can see the arn for the trust anchor the whole is anywhere trust anchor yeah and this is going to enable the intense operator and the credentials operator to set up things for the rest of the pods on the cluster okay so now that we have all of that set up we're ready to go ahead and deploy authorized and tell it um what are the what are the role arns the profile arns and all of the different details that we need for this to work so you need to pass some flags here um we seem to be in the only directory but well Mattias is seeding I'll tell you a bit about the flags so we need to turn on aws we need to tell it which aws region to default to when it's creating roles and policies since we're not on aws it can't auto detect that then we need to give it the role arn for the intense operator credentials operator and tell it that we're in roles anywhere mode what the trust domain is what the trust anchor arn is and the trust profile for both the intense operator and credentials operator which we all just created with terraform so next step is we're gonna see the actual demo we've completed the setup and now we are we are gonna see a demo where we have a client and server who are communicating locally within that cluster and the client is sending requests to a server and that server is trying to submit files to s3 so before we we deploy the client and server maybe let's look at the s3 bucket and see that no files in it because it's a it's a magic trick before you do the magic trick you gotta show the audience you know there's nothing in here it's completely empty and and it's uh yeah like totally empty so that that's really great so like now we can start deploying it yep and now we're gonna deploy the client and server now the client is periodically sending a request to the server asking it to upload a file to s3 and the server right now is actually going to be failing to do that because it doesn't have access to the s3 bucket and if we look at the logs for the server and we wait a bit longer then we should start to see errors from aws we're waiting for the internet to vacuum the container out of docker hub but it's gonna arrive probably the client is still loading elevator music i noticed that it could have taken some time authorized tutorial let's see yeah it's running so the logs again it's not a good demo if it doesn't have technical problem aha there it is so so took a bit longer than usual but what you can see here in the error is that the server is failing to upload a file because it's trying to do an s3 put object operation and it's not able to retrieve credentials or refresh credentials this is basically the aws go SDK trying all the different methods it knows in order to get credentials and they're all failing because it's on google cloud how would it work so what we now need to do is we need to label our pod before that we need to do something else you're right you're right um for for this pfcsi driver to work we actually need to for our server give it the ability to create a certificate request which the pfcsi driver will do on its behalf and that has to do with the token request functionality i was talking about earlier where the csi driver is going to impersonate the service account of its subject in this case a server and it's then going to on behalf of that server of that subject it's going to do a certificate request to get like certificate issued so it needs to be able to create a certificate object and give me this of course yep if we don't do this then the pod will fail to start because the csi driver will not be able to do the certificate so we now have the ARBAC and now we're going to patch our pod and we're going to do two things first we're going to do the aws row label um for other eyes to tell the credentials operator that hi i'm this pod please create an aws row and all matching resources for me and then the environment variable aws region to tell the pod which region it should work in since it can't tell because it's running on google cloud and so to show this is all going to work quite needlessly the magic trick we were talking about earlier it will automatically create an iam role that starts with otr you can see currently this is completely empty in my AWS environment so and like when we do the patch of this one it will automatically create this one so what's happening behind the scenes we're about to see that what happened to the pod in a moment but we already can see the role that was created here and it still doesn't have any policies but maybe show the trust relationship yep and you can see that we have the principle from from before the service account name space otherwise tutorial iam service account is server and the source er n is the trust anchor it was set up automatically just because we label that pod and this is just by labeling inside kubernetes it's all like happening through authorize nothing through data forum or anything else which is really needs until now and then the last bit i guess yep so maybe before that let's look at the pod's declaration to see what the credentials operator did so can we describe the pod authorize tutorial okay i stopped being able to type when i'm live screen sharing okay so we see the sorry can you give me just one moment sorry about that of course during the webinar is when somebody decides to renovate right outside the office and make tons of noise that happens thank you to make it a bit quieter hopefully succeeded okay so we've described the pod and what we can see is we have our label and under annotations we can see that the credentials operator annotated with the role that it created and if we scroll down to the volume amounts um we can see that it created a spiffy volume with a csi driver and in the volume attributes it placed the role er n the trust anchor er n and the trust profile the role and the trust profile were created especially for this pod or this service account rather and if this pod if all pods using the service account or the service account itself is removed then the role will also be removed so the kubernetes the state of the kubernetes cluster and what's on aws is captain sink and everything is provisioned just in time so now this pod has a role it's authenticated but it can't do anything yet because it doesn't have any policies attached to its role so now we're gonna apply the client-intense cld resource um what this is saying is that the service called server wants to call this s3 bucket under this pattern and it wants the s3 put object action um once we apply this to the cluster the intense operator is gonna go and create an im policy and attach it to the role let's see it in action let's have a look over on aws to see the magic materialize and we have a policy exactly what we asked for so to sum up what did we have to do in order for all of this to happen label the pod create a client-intense now let's see if it actually works oh my god for those of you not well versed in json there's a field called status it says 200 which means we are no longer getting errors so if we check our s3 bucket now the magic trick should be complete yep we have files matias did you do that while i was away or was it uh is the the demo work the demo work the demo work and like one of the nice things as well as like authorize not only sets up like the the iam policy and uh the role for it it also does like also automatically the iam anywhere policy which is a specific one you can see here it has created this in the background which then refers to the role uh that we were showing earlier so like everything is handled by authorizing the background so you can really declare your access to AWS resources through your kubernetes cluster and there is actually no need to manually do this in AWS or need to do this through any other infrastructure as code tooling like it's all natively in kubernetes which is really nice and multicloud because this worked from quill clouds natively yep and this really simplifies handling the the lifecycle of all of these resources because you don't need to think okay if i do a helm rollback do i also need to roll back my terraform no you just do your helm rollback the resource goes to the previous state the operators take care of syncing AWS to that and you always know your source of truth is kubernetes cool perfect that was the demo time it all worked from the first try i'm so happy that the demo gods were with us today so a bit too for closing off of this webinar first of all like both authorized and cert manager are two open source projects and we're always looking for extra contributors towards both of these projects so the cert manager project can be found on its own cert manager organization and the main project is on the cert manager in there but we have a bunch of other projects as i was talking about for a bunch of different functionality like coupling with Vistio having CSI drivers the trust manager is in there so like yeah we have a quite great ecosystem already with like lots of contributors from different companies but we always want to expand this and yeah like get feedback as well on cert manager you can also find us on the kubernetes slack we have a cert manager channel in there where you can ping us maybe already you can give a bit of information on authorized and how they can contribute there so authorized isn't yet as popular as cert manager we're not yet deployed on 90 percent of clusters but we'll be there so you're more than welcome to check out any of these repositories they do most of the time it makes most sense to deploy them all in one go so they come in one helm chart that wanders them together but they can also be used standalone and we have a community slack that you're welcome to pop on to and ask questions ask like what's the best issue to get started with if you're interested in contributing and yeah come be an extroverted introverted platform engineer with me yeah and automate your whole environment through cert manager and authorized for access cool uh to recap and to uh like shortly recap what we have seen today first of all cert manager can do spiffy identities and it can mean to spiffy identity in x519 certificates uh thanks to the great coupling of the csi driver spiffy which currently is not yet in upstream but like it's open source already uh you can use spiffy identities to work with idOS iam roles anywhere and then the authorized bit sorry so once you have that authorized creates a matching aws iam roles profiles and policies and put it all together and workloads can get access to aws no matter where they run on another cloud provider your grandmother's laptop um or bare metal as long as it runs kubernetes we are covered and it works and and that's a really great thing like we showed this today from a cool cloud cluster but this could have been an azure cluster or like in the bare metal cluster that you have set it up yourself in kubernetes for example thank you for attending this webinar it was a pleasure to present together with you ori you can you're always looking for new use cases and and new things to do also if you have questions feel please feel free to reach out to us you can email me directly on the email address you see on the presentation but you can also reach out to me on uh twitter or what we now call x with my full name uh or on linkedin as well or any other source you can probably find me on uh yeah uh or it may be some closing thoughts for sure closing arguments uh it was a pleasure to to be here with human ts it's actually my first webinar ever so it was awesome the demo worked a little bit of renovation on the side but we handled it um and yeah feel free to email me reach out to me on slack just ask for a favor i do those too um and yeah try it out let us know what you think um we're really curious to see how people use this yeah definitely thanks everyone for attending and yeah we're also going to open source uh we're going to open up the instructions to fully replicate this demo by the way this will be on the authorized uh uh kit up uh organization where you can go check them out and and run it yourself because all of this what we have showed today is fully open source it uses start manager and authorizing the open source component so like you can fully replicate this in your own environment and even start using it yep and it's gonna you you can even pretend you're running the webinar and just go for the steps if you want the full experience i'm just here to inject you more perfect thanks everyone uh have a nice day for everybody listening to this webinar and hope you learned something thank you very much thanks everyone bye bye