 Thank you. I don't know your name. Thank you, Eric. Hi, everyone. Actually, I want to start with a short story-to-story. Last week I drove back from vacation with my wife. While we were driving down the car with the car back home, I asked her about the talk, this talk that I'm about to give you. And she said, why don't you practice? And I said, okay, and she's not from the industry. She's a social worker. And I start explaining to her about the talk. And after five minutes, she fell asleep. So if no one will fall asleep by the end of this session, I'll consider this as a success. With that said, my name is Dov Hershkovitz. I'm a senior product manager at GitLab. And this talk was mainly inspired by many conversations that I had with our customers and users. And I'm hoping that by the end of this presentation, you'll know better how to secure your ICD using the JSON WebToken or the JOT today without waiting for anyone to build any integration for you with the tool set that you have. So in product practice, we really love to start talking with the problem. We're talking about problem statement. Then we lean into our engineering team to find the right solution. But before I can talk about the problem, I want to set the level straight and give some background with some definitions. Now, I know it's security con, and you probably heard a lot about what is a secret. But let's do that quickly. A secret is an object that enables the process of accessing restricting data or resource. Meaning that the data itself that you want to access is not a secret. But the way to access the data, the key itself, this is the secret that we need to protect. Now, in addition, all of us need to agree that no matter who you are in the organization, we all need secrets. It doesn't matter if you are an operator, developer, DevOps, APM. Everyone needs secrets. Everybody is using secrets. We all need to access specific environment, information to look for something or to deploy an application or manage an infrastructure. For that, we need secrets, and we all use it. We can separate the secret use case into two common use cases. One is the identity authentication. In the case of service to service or microservice to microservice, where the secret is actually an API key or a database credential. In case of human to service, where the secret is the password or an SSH key. The second case is the data protection, when you encrypt and decrypt data without storing it. So encryption at transit or encryption at rest, in that case, the secret is an encryption key or certificate. In terms of authenticity, the secret is signing keys or credentials. I want to talk a little bit about the secret management problem and why it has become such a large problem, a big problem in our industry. I've narrowed it down into three major reasons. I'm pretty sure that there are more, but this is my take on why secret management became such a large problem. To begin with, every large software company needs to deal with legacy software. Software that wasn't built to work with secrets. Maybe it was more built to work with secrets in the case of human to machine and not machine to machine. And nowadays, where everything is automated, we need those legacy systems to know how to handle secrets in the case of machine to machine. In addition, the move to containers and the rise of microservice architecture bought us a lot of benefit. Easy scaling, better manageability and more. But with that, we are facing with another problem on how we can better secure our containers, which needs to be secured differently than VMs, and the rise of the microservice architecture. The more microservices we have that needs to communicate between the staff, we want to make sure that this communication happens in a secure manner. Meaning, the more microservices we have, the more keys we need to manage. And lastly, secret spores. Secret spores happens when an organization starts storing secrets in different places and reach to a point where they don't know where their secrets are. Now, two things will happen with secret spores, and that's a real problem. One, I've read last week in an article by Gartner that a devotee's engineer spends approximately 20 to 25 minutes looking for a secret, just spending time there. But this is the minor problem. The bigger problem is that when we don't know where our secrets are, it is more likely for them to get clicked. Once clicked, the path to a vulnerability is short. And there is also an early report, the state of secret spores, when you hear all about what you hear in the news, they give you different statistics and interesting story of how a leakage became because of a secret spore. Now, we all know that software is all about speed. Every software organization wants to develop and deliver features to our customers as fast as we can. We want to act quickly, we want to hit the market first, we want to close all the bugs and the vulnerability threats as fast as we can. This is why DevOps, as a practice, was adopted so much by our industry because it basically automates everything, gives you speedy pipelines, it will reduce your cycle time, it will get you to the market as fast as you can. But we know that our software and our pipelines need to be secure. And sometimes it feels that security is reducing the velocity of our cycle time because security is more gate-driven, it's heavy on process and let's be honest with ourselves, there is a lot of lack of knowledge on security because we don't know where the next threat will come from, we don't know what we need to close first, which bridge do we need to close first. This is why DevSecOps as a concept at the merge, we hear a lot about security shifting left, meaning let's think about security even before we start developing our application. So DevSecOps means let's put security in the heart of DevOps. Let's not compromise, let's optimize speed and security which will eventually lead us to speedy and secure pipeline, which is what we want. Now let's look at where we normally see secrets. Now this is a really bad example because the secrets are in plain text, but I have reason why I'm doing that. But we normally see secrets in source code when we want to develop our application and access specific resource in configuration files when we want to manage our infrastructure and in DevOps scripts in our CI-CD pipelines. Usually they are being declared as a variable or a variable type of secrets. Now there is a reason why I put it in plain text, like plain right now. Since this is a DevOps oriented talk, I want to talk about secrets as an environment variable. First of all, this is one of the most common ways to store secrets in a DevOps platform. And there are several flows as I've listed here. Not all the platforms have hold all the flows, but this is the collection of flows that I discovered while I was looking at different DevOps platforms. First of all, as you can see, yes, even secrets variables appear in plain text. And the reason for that is that first of all when we come to develop a pipeline, most of the time we won't put a secret there. We'll put something as hard-coded. We'll see that the pipeline is running and then we'll say, okay, let's change it to be a secret. Sometimes engineers forget. Sometimes it's simply impossible. So there are a lot of cases where it's in plain text. It is not encrypted. Maybe it's encrypted in the database, but when it is being pulled and being used, the pipeline is not encrypted. It's hard-coded. There were a lot of times that someone pushed a change, pipeline starts running, suddenly the CI job fail. Why someone changed the credential? Okay, so you need to go, you need to update the credential, not to mention key rotation that is not supported. We are also missing the fine-grained access control, meaning if any engineer has an access to your pipeline and can run your job, they can use that pipeline. There is no way to assign a role on a specific variable. Also, putting all the eggs in the same basket is a bad practice. Secrets need to live in a dedicated secret solution. And DevOps platform may need to run pipelines. And also, lastly, it is not 100% secure. In most of the CI platform, although variables are in type of secrets, there are ways to extrapolate the data. It is not rocket science. I'm not going to show you how. You can talk to me afterwards and I'll explain you how. But this is a common knowledge that you can definitely get the value out of a secret variable, in most cases. Now I think we are ready to talk about the problem. So, as we mentioned, secrets need to live in a dedicated secret management solution. This is why all the cloud providers, all the major cloud providers have a secret management capability, a secret manager, a key management system to store and manage secrets. There are also many, many dedicated secret management solutions that are out there, like Hashiko Vault and Conju and Akiles and more. But as we mentioned, we want to integrate between our DevOps platform, our CI-CD pipeline, to those secret management solutions to grab those secrets, because we don't want to store them as an environment variable. We need to integrate those secrets with our CI pipeline so we can automate actions such as managing our application or managing our infrastructure or deploying our application. Now, the problem is, which tool should we choose for? Should we simply choose a tool based on the available integration? Don't we want to use Best of Brit? What if my organization is using a secret management solution that is not supported by my DevOps platform, my CI-CD pipeline cannot communicate with that secret management solution? Or what if I have multi-cloud strategy and I want to integrate with all the cloud providers? So if there is a native solution and if there is an integration, then we don't have this problem. Maybe you have the vendor lock problem, but you don't have this real problem. But as much as I would love to integrate with all the secret management solutions that are out there, I simply can. So what you can do in case there is not an integration and you cannot retrieve those secrets, there is a solution and in order to understand that we need to familiarize ourselves with two concepts. One is the OpenID Connect and the JWT. The OpenID Connect is an open standard for the centralized authentication. The JWT is an ID token. This is one of the methods to implement OpenID Connect. And the good news is that all the cloud providers, the secret management solution, and many of the DevOps platform support OpenID Connect today. So let's talk a little bit about the JWT token. So it is a short lived ID token. It consists out of three parts, the header, the body, and the signature. Now the body has different claims. Those are key value pairs. And there are custom claims and standard claims. And I'm going to show it to you in a second. The standard claims are the claims that always exist in a JWT token. And the custom claims are the ones that are being injected by the solution that generates the token for me. And when a user, a service or a CI job comes to authenticate against a specific server using a JWT token, what happens is that that server is looking for specific claims that exist and then looking for the value of the claims. And only if those match, then it will authorize and authenticate that action. And this is something that you can configure as a user. You can configure which claims and which value will get authenticated and which will not. Now I'm going to show you how the token is going to look like. So this is my JWT token. So first of all, I'm in my website, JWT. On that side, you can see the encoded version, this long large string. This is the token. This is my token that I've generated. As you can see, it has three parts. It has the header, it has the body, and it has the signature. On the right side, there is the decoded version. And you can see it easily separated by this dot. There is a dot that is separated by the different parts. So the header, the signature, making sure that nobody is tampered with my token. But what's really important here is to look at the body. And those are the claims that I mentioned, all of them. So there are some standard claims like ISS, issuer. Who generated this template for me? You can see it was generated by GitLab. Expiration time. When this token will be useful. Sorry, it will be useless. Because even if it will get leaked, then you can use it only for five minutes, and that's it. And besides the standard claims, there are those custom claims that I mentioned. Now those custom claims, they provide you with a lot of data. First of all, about the user. You can see I have here the user login, the user email, my username, my email. And also specific information about the pipeline that this token was generated by. Like the project ID, the namespace path, the namespace ID, the type of branch and more. So as a user, what you can do, you can authenticate and say, OK, only if this user with this pipeline on that branch is being authenticated, then I can give the CI job or the service an access. And if not, I won't grant an access. So two users can do the same job. One will get authenticated and one will not get authenticated. So it gives us this fine grained access control that we are looking for. Let me go back to my slides. So what you can do, assuming we have this tool set, we have a variety of platform of solution that support the OpenID Connect. As a user, I can integrate my OpenID with JWT in my DevOps workflow and then pull secrets from the secret provider and then enable my pipeline to run securely. So the steps that you need to do in order to configure it is the following. The steps are more or less the same. I gave this example on GitLab, but it is more or less the same no matter which DevOps platform you are using. So the first thing is that you need to configure trust relationship with OpenID Connect on your providers. So you need to go to the providers and tell them, I want to authenticate using an OpenID Connect with the JWT token. This is the part where I am setting the custom claims and the value only this user, only this claim and only this value will get authenticated. And I've linked a few examples on how you can configure OpenID Connect with the JWT, AWS, GCP, but as I mentioned, Hashiko, Opaquiles, Conjure, all the secret management providers support that as well. Secondly, any DevOps platform, it doesn't matter if it's GitLab, GitHub, Circle, Jenkins, they are capable of generating a JWT token. That JWT token is sent to the cloud provider where it is being authenticated. This is where the matching happens on the claims. And if it is authenticated, it will give us back an access token. And with that access token, we can go ahead and retrieve secrets. I will show to you how it looks in the configuration. But this is more or less the same. This is what you need to do with any platform that you have today. Now, I have linked a couple of examples here on how you can configure OpenID Connect on all the cloud providers. Some working examples on how you can configure pipelines using Vault, AWS, GCP, Conjure. But you can go and grab that example or you can simply go to the documentation of your provider and just look for OpenID Connect. You immediately see it with the example and everything. It doesn't matter which tool you are using. Now, let's look at an example, a real example on how to integrate between GitLab and HashiCorp Vault. The reason I'm showing you the integration with HashiCorp Vault is because in GitLab we did build the native integration between the two tools. So I'm going to show you both methods, how you can do it with an integration and how you can do it with the JWT token. So I'm here at the Pipeline Editor, which is one of the ways to configure pipelines in GitLab. And when I define my pipeline, I need to set up two stages and two jobs. This is job number one with Vault integrated client and job number two without the integrated client. And I can also show you how it looks in the visualization. This is stage number one, job number one, stage number two, job number two. Now let me break it to you. So if we have a native integration, the configuration is pretty straightforward. You can follow the documentation, but basically what we are doing is, as I mentioned, the name of the job, then we assign a stage. We need to use an image to run the scripts on top. So our CI job could run on that image. And then I'm using the secret keyword. The secret keyword is a predefined keyword. The test GitLab use the Vault integration. And then I need to declare an environment variable. And then all I need is just to tell GitLab where to fetch the password for. So I'm looking for the password field underneath the Vault demo path. This is it. And then there is one parameter, and that's it. The secret will be underneath the secret variable. And then I can use it in the script here. I'm just echoing the script. Sorry, echoing the secret. But then you can use it as a part of your pipeline. Now, the second example, if GitLab did not, or any other provider did not build an integration for us, we can still get the secret's form vault, using, as I mentioned, the JWT. So following the same steps, declaring the job name, a stage, an image. And then those four lines that you'll see in here is just like updating, downloading the HashiCorp line so I can use the CLI. And then I need to declare where HashiCorp client located. And this line over here, this is where I'm sending the token, the JWT token, you can see here. This is the JWT token that GitLab can generate for us. I'm sending it to Vault. And I'm asking Vault to get me that access token. And this is where this all-matching happened. This is basically the number three here. This is actually number two and number three. So this is where I'm sending, and I'm getting back the access token. And once I got the access token back, if all of it was successful, then I can ask Vault to get the secret from the same path, the same path, the Vault demo. I want the password field from Vault demo. So I'm doing the same thing, but without the building authentication. And then when I retrieve the secret, I'm echoing it, but I can use it as a part of my pipelines to do basically any action that I want. Back to my slides. Yes, it's not because if it is being rotated, if the secret is changing all the time, why do I care? This is something that Vault can handle. Okay, so as I mentioned, link some documentation. Now, like any solution that we have in our industry, there are benefits, there are pros and cons, and let's go them quickly. If you use the JWT in a DevOps workflow, the benefits is, first of all, it's way more secure, because you're using an ID token. Even if this ID token gets leaked, it has an expiration time of five minutes. Second, you have no vendor lock. You don't need to wait for anyone to build integration. You can use the tools that you have today right now. And lastly, secrets are stored in a dedicated system, and you can get all the features surrounding secrets. The downside of using JWT in your DevOps workflow are, first of all, you need to know how to configure OpenID Connect on your providers. The setup, as you saw, it's a bit complex. Although you need to do it one time, someone needs to do it, someone needs to maintain it. Even in GitLab, we have a handful of engineers that know how to do it. The rest of the engineers are using what one of the engineers built for them. And lastly, there is an associated cost. When you store secrets in a secret management provider, whereas if you're storing it as an environment variable, it doesn't cost you a dime. If there is one slide I want you to remember from this talk is this one, basically recapping this entire presentation. The challenge, the way I'm saying it, is that secrets are typically stored in a dedicated management solution, okay? But we need secrets. Our engineers need secrets to integrate them with our DevOps platform to run CI-CD pipelines. The solution, if you don't have a built-in integration, or you don't want the vendor lock, then you need to configure OpenID Connect on the target client, and then set up your pipeline with the JWT to retrieve those secrets. The secrets are securely stored and integrated with your existing DevOps tools. With that, I want to thank you, and I hope you find this presentation useful. All right. Thank you very much, Dole. Do we have any questions? We have a little bit of time for questions. One right there. Can you repeat the question after you? Sorry. Thanks. One thing I wanted to add about the environment variables of one of the risks that with putting secrets in environment variables is that sometimes when there's an error in the system dump, it'll just dump all the environment variables, and they'll put in insecure logs. So that's just another reason why environment variables are generally not a good idea. And thanks again for the mention. We also have a guide on Conjured that will forget lab specifically how to use this. Thanks a lot. We are looking for the documentation part to be added. Let's talk. Do you have a question over here? So I wonder... So basically when you use JWT to access secret in wild, you use JWT to exchange the wild access token so that you can use the wild token to access to the secret. And during that process, basically, is that developer need to know which wild row to assume so that... My follow-up question will be in each of these steps in the pipeline, how do you control which step in the pipeline can access to what secret? So when we have this command that I just showed you, we are specifying which secrets we want to retrieve from which path. Right, but that means there is a wild policy already exists. Sorry? So you must define some wild policy to basically allow which access token to access to this particular secret. I need to configure that on HashiCorp. This is the part that I haven't shown. How do I configure that in HashiCorp? So it doesn't matter if it's HashiCorp or any other, or Konjo or whatever, but there is a part that is missing here. I was just showing to you how to set up from the pipeline, but in the provider side, there are different ways where you can configure that and outside every secret provider have a different way on how you can specify which secret you want to retrieve. So this goes to the... Yeah, my ultimate question would be like who is going to be responsible or who will you suggest to manage such authorization policies to access secrets, I mean access secret from pipeline? Who is going to be responsible? Which team? Usually I think it really depends on which organization you're working on, but normally when I'm speaking with teams, it's normally DevOps engineers, there are series, the ones that are maintaining the pipelines. And yes, there is there is some complexity, I would say, when we ask them to configure it in the secret management solution. Sometimes they don't I would say like doing that, but as I mentioned, it's either that or use it as an environment variable or wait until a vendor will build a native integration for you. So it's up to you to decide. Any other questions quickly? Great, well thanks, Dov. That's great.