 Okay, hello everyone. So today's topic will be the road to south of Florida playing six store ecosystem in a corporate environment And I'm Alex Ilgayev and in this talk we'll look on six or on a bit of a Different angle we look at it from the corporate perspective And in the bus in the past month and me and my team in Cycord. We're looking extensively at six store Researching and in order to understand how he would take it and and apply it in our Internal a CI environments and we wanted to achieve two goals. We wanted To sign our artifacts for the CI keylessly and we wanted also to to use get signed to get signed as it should and to easily Sign all our commits And Originally also Gabrielle ahead of DevOps in Cycord work on this topic. Unfortunately couldn't come to the to the conference So, okay, let's start with agenda as it is. This is a six store conference. So we will start Explaining what it's six or I don't think it's much necessary But we will explain what it's six or and even more important why corporate should adopt it We explore the six or tooling and then we will go into understand why what the issue could be have into into a Working with the public infrastructure of six store in terms of privacy and how to create maybe self-hosted six or infrastructure and also why Then we will explore some how to use six or to sign artifacts in several real-world the CI environments and finally We'll talk about some of the insides of this of this research And originally when I applied for this talk I also wanted to talk about the verification process and Maybe more thoroughly talking about get signed and the issue of you currently adopting get signed the corporates Unfortunately, I don't have what I won't have enough time to talk about it in 20 minutes. So Maybe in the next talks Okay, so I'm a little guy, but I'm a senior security researcher at the psychod Previously, I led the malware research team at the checkpoint where I will Doing a lot of reverse engineering for complex pieces of malware's Investigating attacks both for generating cyber crime and the APT's and now this I Investigated research vulnerabilities and mitigations for software supply chain part of psychod And psychod is a Cyber security company that provides a complete software supply chain solution for organizations And Okay, so what is six store? I don't think it's a there's a lot of talk about with all the great talks we had there so far But basically a new standard for signing very fine protecting aspects of software developer development. It was a a Originally it was built specific specifically for open source projects, but we will look at it from also a bit of a different perspective Okay, so what would be the motivation for for corporate for example our corporate when we look at it to into adopting Adopting a six store tooling So let's say let's take in an example We have two main cases for such corporates the first one is a Using Internal artifacts as a built in the CI environment of the corporate the way which are deployed into The internal production environment of the corporate So we can it can be used to validate artifacts before deploying by the corporate itself and I'm focusing mainly on the Keyless signature for example, we could verify if we have like several Personals that are authorized to To sign artifacts we could verify that these were the only one that were Signing on this artifact before deployed to production Also, it can be also used to validate the artifacts before being used by the external clients. They could be a Clients that's using this the components of of the packages or they could be used also as a Also our clients like open source clients so all this Packages could be a Can you must be signed through the CI environment of the corporate and be verified? that this was signed specifically by the Corporate for example email address or someone inside the Inside the Development process Some of the additional benefits you could have it's also could satisfy some of the requirements for salsa similarly to the To the salsa provenance builder that they're using reasonable workflows To satisfy non-fossil fiber requirements and achieve the salsa free So you could also use the cost sign for example to sign additional pieces of information In addition, we could also adopt a git sign to sign commit and tags and to allow tracing the origins of the code insertion This could all this could help to remove the burden of handling the GP G keys and Maintaining them across large enterprises and also it allows you a higher integrity level for each and every commit inserted to the code repository and even Satisfy the some salsa requirements for example the verified history So let's go over quickly some of the six or two linked understand how they work and then we'll see how can we use them? In some complex a CI environment So the first one is record. There are a lot of been talking talking about a record here. So basically it's a It's a immutable temporary system ledger of metadata built on top of a trillion tool by Google basically it's a it's a Public infrastructure six or six or holds a public in some infrastructure of record that every one can add in at the metadata to it and and query and verify that Query and verify it So in six store, it's mainly used to store signatures for artifacts and also the certificates that are the short-lived certificates that are been used to a to sign That artifact another additional important tool is full seal, which is the root CA that that's in charge of signing and And issuing the short-lived certificates The full seal basically it's it's a You you give it some OIDC tokens of open the token that proves your identity And who's the need to to check whether This token was is valid through some configuration. It has it could check it through a several identity providers It would be a github Microsoft Google github actions could use also use a dex identity provider Which is an open source popular identity provider and if full seal found that The token is valid. So it will issue it will issue a short-lived certificate according to the identity given Identity from the the token So one of the important Factors about full seal that's it's its strength is as it depends on the root CA. So if the root CA is a Is a compromise or isn't secured enough. So we can freely trust a Full seal. So it's a very important factor into whether you want or not to erase your own self-hosted version of a full seal cosine is basically the signing tool And for a for six store it allows you to sign the artifacts and maybe additional attestation Data and it stores the data on the OCI registry And find the git sign. I think we can skip that pretty explained way better than I do So if we look up from the corporate perspective the using public instance of rick or could be quite problematic For example, I used cosine to sign some simple And image of mine. It's called simple go server in this. It was stored under public instance of rick or we can see the index number Everyone can just go and query this This entry in a rick or Extract the certificate up from it from this entry and get my my email number. It could be personal to be corporate And This one could be quite an issue for a corporate specifically when also Looking on git sign which implies on the way and a lot more if entries such that and If artifacts are private or the project as privates, it could be an issue to expose a Lot of maybe email numbers. It could be also a maybe identity of the machines or get a passion workflows or whatever Additional issues that could have with the also would force your even though it's not Publicly accessible to everyone. It's still the demands that trusting the Full server to store your emails and your certificates. You need to trust the root certificate or your root certificate and and the that That gives you the Short-lived certificate So even though I don't recommend it to do it for any any organization. It's still I wanted to Experiment what it means to erase yourself your self-hosted infrastructure for a for rick or fools you and the rest of the six tour and The best source was a six or the hard way created by Luke and press several additional contributors And it It wasn't so hard actually Yeah, but it's so hard, but still I will mention it in the end it The most problematic was maybe raising the full so instance, but it demands like maintaining the The root CA and private keys which can be Very demanding for a maybe smaller company, so I don't recommend it to for every every organization But if you want it's possible So let's see some hello words for a simple signature using the self-hosted infrastructure first We'll need to define several Environment variables. We need to get the so-called new root CA. There's also a public key for Certificate transparent the part of the Foolsium So whenever we won't use it in just running cosine for our image that went to sign we give it and like some additional parameters like the the new so-called a recurrent full scene of Endpoint This leads us to some login screen similar to To the public a six or infrastructure username we're choosing our for example a Google account or identity to want to use it to sign the artifact and finally when signature Succeeds we have like entry in the record that we we can query whenever we want and would you get our The identity that the certificate which contains the identity which you that used That were used to sign the artifact It also includes the issuer Which in our case, it's off dot cycle a dev that issue this certificate Similar procedure also we can apply for a git sign We need to also to define several environment variables That defines the in the new self-hosted infrastructure when in every commit we just log in for a Google account, which is a Google account and we We have this entry that saved on record that We can verify and see that it contains the certificate Only the name of the one who made the commit Okay So not now that we made some introduction of the six or tooling and maybe how to create a Self-hosted version for a for corporates. Let's look about let's let's look over and several Common use cases of see environments in in corporate and how can we integrate? Six or tooling including signing artifacts Yeah, using that environments totally keylessly and the And the point went to mention is that we don't want the developers to know about it to be transparent as we can for the developers themselves So we will explore it to Two scenarios and the first one is using a github code management with github actions. See I Will use github hosted machines for the CI and finally we will push the artifact to docker up And and for the second case which has to be quite more complex We will manage our code in github and use github runner CI, but We will use a self-manage or manage. It doesn't really matter a private Kubernetes cluster That will run the CI builds in By a pulling a github and this builds should build the image and push it to docker So how the architecture pre-stick store will look like I'm a user I will be pushing code into github github will provision me a new VM on each on each new build and This is why job will push will build the image and push it to docker up If I will be pushing the code to a github I have like I managed in our case a gk cluster It could be a ks or eks. It doesn't matter There will be a managed cluster and in a pod inside a cluster that will be pulling github for a new job whenever the new job will receive it will provision a new pod and And that pod will do the same job including pushing the image to docker hub Okay, so first let's look at the github action case A simple solution to this issue into integrating six store could be through usable reusable workflows Very similar to the salsa proven as builder Imagine a github action Workflow when the first with the heads to job the first job is like building the The image and pushing it to docker hub and the second job will be Adding a signature to the image it could be it would be calling a Reusable workflow and external workflow that could be used in also in several other builds an important factor That it should be sent a permission to with ID token to that reusable workflow This ID token is Is giving the reasonable workflow a job token as an environment variable that could be used to give give identity to the signature process We also sending a parameter of the the image damage should be signing So how this is a work for looks like First it should be defining several environment variables We said we want to try it out with a self-hosted infrastructure. So we're giving it the right recurrent full seal we should also be giving it the The root CA for the full seal was just fetching it from some from a gcp bucket and and the inside the The workflow itself We it's very plain simple just signing it giving it the right parameters and giving it IDC provider of Gita Baction Fortunately a cost sign is supporting Gita Baction as many other identity providers. So he knows how to a fetch the the job tokens from environment variable and use it to authenticate against the Gita Bactions identity provider And so what will be the result? Imagine we're running this CI build This you're usable workflow. We are being some are sample image So when we finish running it will be succeed succeed in in Pushing and building and pushing the image and we receive our entry in recall When we query this entry We will see that it contains this certificate Full seal is aware of Gita Baction. It knows how to put the right Properties in the certificate the contents handful of information First the issuer of the Gita Bactions the workflow trigger the committia the workflow name and more So we were able to integrate integrate the six stores part of our SCI It was it wasn't so hard, but let's look at both more complex scenario For the second scenario said we are using github and Kubernetes executor for github. This means that they every Every new job is provisioned as a new pod in in in the manager or in the self-hosted cluster so in order to sign keylessly the image in the In that in the self-hosted or managed Kubernetes we need some identity In order to send to full seal and receive the self-hosted a self short-lived a certificate So where can we fetch that identity from we have several options? The first will be to use a environment valuable called CI job JWT which supply to every github Execution Github provides every Every job this variable that contains it's signed by the by the github service itself and it gives information about the The actual job that's running it can be used to many many use cases many use cases are like using it to authenticate against hushikov volt and additional cases Specifically, I don't recommend it using here because it you can't you can degenerate Token for specific audience like you can in other identity providers Additional a solution will be to use the gcp metadata service. I said that we are using GKE This means that the nodes in the cluster are a Running as a compute virtual machines on the gcp So every VM is a can access to some metadata service and get its identity from a gcp That's it is validly signed by gcp We can maybe use this identity in order to sign the artifacts over who have a Strong authentication that the artifact was signed in our by our specific nodes in gcp This can be a Decent solution, but it won't be enough because because we won't be utilizing that we have a Kubernetes cluster and we have specific Pods that were doing this build. So if this solution isn't good enough The third solution would be to using a Kubernetes identity service as I said, we have like a manage a gk cluster So we can we can ask Kubernetes to sign the token and use that token It's also a decent solution, but it demands exposing the cluster Jack JW case endpoint. So it should be a verified somehow that That the stocking is actually valid so at the final a Solution that maybe the most complex one is that use the spiffy spire It could give us the best of both worlds of offer using the Kubernetes and a server and the gcp meta server and we can authenticate nodes using gcp and authenticate wall loads you workloads using the Kubernetes workload attest So this how the architecture looks like it's quite complex, but we will go over it step by step First we need to create some spiffy spire infrastructure and There's the excellent tutorials and quick starts a for a for that I personally, I suggest to install a spire on on some In some different or a different server than on the on the cluster you're using to build your images on And you also install the spire agent through some daemon set inside your Kubernetes. So every Every node that will be will have this spire API And the build that will be running on this node. We have access to that API Also, the spire agent will be authenticated with the gcp meta data service In an attest as the agent do the being part of our gcp project For example, so when I install the spire agents and I went to the spire server to see all the agent list I could see I could see that I identified the three nodes. I have in my cluster and each of them has a specific spiffy ID This means that the spire agent was successfully authenticated against the gcp service Next we'll have to define some several selectors for the workload attestation up to this point We attested the nose now. I know need not attest the the workloads that will be a provisioned on every new build So for that we're creating to a new entries in the spire server. The first we're giving a spiffy ID of build node to every Node that is specifically in our a psych of love gcp project. We already attested the node. So that's how he chooses the nodes and And giving them the specific spiffy ID and we're doing I think additional entry we have for a workloads that every workloads that was That has parent ID of the specific build node, which is that the previous we defined and also in a namespace that called git lab Runner will receive this spiffy ID of builder which is perfectly given to workloads a Potentially we could add the additional selectors here the spiffy a spire supports a lot of selectors The more we added the more secure the process will be Next we need to allow all the new pods With the provisioned in the build to have access to the spire API Because we are using a built-in a git lab image for the for the pads a Which don't have the volume of the spire mounted GitLab doesn't have a support out of the box for a spire. So we use the a cuverno policy to mount The spire API this means the spire units unix socket For every every new pod that is specifically in the git lab runner namespace Which is the namespace in our cluster that all the new builds will be a created in So this a caverno policy just saw this issue for us and finally we need to Configure the github ci workflow To get the svd svd. It's like a identity token that we received from a spire That identifies the specific workload. So we're getting this svd. We're sending it to fulcio to receive This short-lived certificate on our on behalf on our identity and We using this surface certificate to sign the artifact Finally this entire workflow. It's like quite complex to build but it has zero key management and Building it you don't need to be aware of the identities on the underlying build system to build it so this is how the Finally ci file in git lab looks like We are defining a batch of environment variables for the for the infrastructure for we define the spiffy endpoint socket and several six story variables all these variables could also be defined through some caverno policy to be default Finally we use the kaniko Image to build the docker to build the image and we're using a cosine with our What our variables with all this you provide a spiffy and for our server so cosine Knows out of the box to work with a spiffy spiffy API and gets the svd and use it to sign the Sign the image So when we run it in the github We'll manage to sign our artifact to push it signature to To record and when we query record we would see the next certificate Certificate was issued to this spiffy idea of of the builder node, which we defined in the inspire And it was also issued by IDC dot psycho dot dev, which is this spiffy authority So this is what the the ci use cases. Let's some of the insights we had there through the process First regarding self-hosted fool. So as I said, it's may become complicated due to management of the root CA it could be quite tedious to do it and propagating the public key fruit CA and the Certificate transparency log public key to the signers and verifies could demand some maintenance work Additional usages for a cost sign could be also do some artifact attestations. So we could use the the Architecture we build right now. You can use it also to sign whether it will be provenance as bomber and ability report or any Animated that we want it will be attached to the specific artifact And also some regards regarding the manage the Kubernetes cluster or any other environment. So It can take quite some time to build the right solution and take can take some time and effort But this effort can be also beneficial for Not only signing the artifact but also additional Additional attestations that could you bring you to a higher salsa level? So some take away from this talk We were experimenting six stores a lot and we're trying to bring this this knowledge to the conference and the bottom line six stores great And So even though it was designed with open source in mind We wanted to deliver that it also can fit some Organization to secure their artifact whether they are internally used or externally by by other clients Creating only for start all the infrastructure from scratch is doable, but it's not recommended for every organization Also, you can apply some hybrid approach for example creating all in the record instance and use the full seal and Finally the support of identity solutions in six or cosine specifically is Enhancing the capabilities to support some complex CI scenarios like we seen in the in the spiffy spire and Kubernetes environment Okay, that's it. Thank you