 Hello. Thank you for joining me today for a talk on secure policy distribution with the Open Policy Agent. My name is Ash Narkar. I am a software engineer at Styra and I'm one of the maintainers of the Open Policy Agent. I care about developing secure systems that can be easily deployed, scaled, and managed. In today's talk, we will look at a quick overview of OPA and some of its features. Then we'll talk about how to do policy distribution. Finally, we'll see a demo about secure policy distribution with OPA. Let's get started. What is OPA? OPA, which is the Open Policy Agent, is an open-source general-purpose policy engine. When you use OPA, you are decoupling the policy enforcement from the policy decision-making so your services can now offload policy decisions to OPA by executing a query. Let's understand this a bit more using this figure. Imagine you have a service and it can be any service at all. It can be a Kubernetes API server, it can be your own custom server, Kafka, Terraform, any service at all. Whenever your service gets a request, it's going to ask OPA for a policy decision by executing a query. OPA is going to evaluate this query based on the policies and the data it has access to, and send a decision back to your service where it gets enforced. You can see that we have decoupled the policy enforcement from the policy decision-making. The policy query itself can be any JSON value. For example, if you're doing AP authorization, this policy query could be the HTTP method, the path, and so on. If you're doing Kubernetes Admission Control, your policy query could include the pod manifest, for example. As long as you give OPA some kind of structured data and you write policies that make sense for that data, OPA sends a decision back to your service. That's why OPA is called a general-purpose policy engine because it's not tied to any particular data format. Even the policy decision OPA makes can be any JSON value. Not just through false or allowed deny, it can be any other JSON value as well. Let's quickly look at some of OPA's features. At the core of OPA is a high-level declarative language called as Rego. With Rego, you can write policy decisions which are Boolean, sets, objects, arrays, and so on. OPA is written in Go, and it's designed to be as lightweight as possible. All the policies and all the data you need for evaluation are stored in memory. You can deploy OPA as a sidecar, a host-level daemon, or you can embed it inside your code if you're writing a Go. OPA does not have any runtime dependencies, which means it does not have to pull down any external services or does not have to talk to any external services for making a policy decision. You can however extend OPA to do that, but that's completely optional. OPA does provide you with some management APIs that allow OPA to pull policy and data from a remote service, or upload its decision logs to an external service, or upload its status to an external service. Finally, along with the core policy engine, OPA provides you with a rich set of tooling that you can use to unit test your policies, that you can use to benchmark or profile your policies, and also there are integrations with IDEs like VS Code, IntelliJ, and there is also an online Rego Playground, which allows you to experiment with Rego policies and share your policies with the world. These were some of OPA's features, a high-level declarative language, multiple deployment models, management APIs for control and visibility, and a rich tooling set. Now, let's talk about policy distribution. Today, a lot of organizations are using microservices to build out their infrastructure, and what they do is they run OPA alongside these microservices to authorize access to these services. OPA's architectural flexibility allows them to do that because you can simply run OPA as a sidecar, and you can authorize access to these services. Now imagine if you scale to tens, hundreds, or even thousands of microservices, and you have OPA running next to each of these microservices, the question arises, how do you distribute policies and data to each of these OPA's? How do you ensure that each of these OPA's have the most up-to-date versions of policy and data? Finally, how do you load these different versions of policy and data without having to restart OPA every single time? So to help with all these issues, OPA supports bundles. So what are these bundles? Bundles are basically GZIP turbos that contain policy and data. So you can have your bundles stored on an external service, and OPA will periodically pull down these bundles from your service, load them, and start enforcing the policies inside of those bundles immediately. So for example, you could have your bundle stored on S3, and you could point OPA to that bundle on S3, OPA will pull it down, and start using the policies in that bundle immediately. So this is really convenient if you want to distribute policies to all these different OPA's which are running alongside your microservices. So bundles are great, they're awesome, but we do have a problem. How does OPA know that these bundles are coming from a trusted source? OPA will assume that the policies and the data in those bundles are authentic and not corrupt, but is that always the case? And what if an attacker corrupts the policies in the bundle and now OPA downloads this corrupt bundle and start enforcing those corrupt policies thereby compromising your system? How will you be able to trust the decisions that OPA is making? Introducing the signed bundle. The signed bundle looks and feels exactly like a regular bundle, but it has an extra signatures.json file. And the signatures file, it dictates essentially what files should be included in the bundle, what the SHA hashes are for those files, and is cryptographically secured. So this is how a signatures file looks. It's essentially an array of JSON web tokens which encapsulate the signature of a bundle. And if you want to decode a JOT, what you would see is that, for example, in this case, the bundle includes a couple of files like the manifest file, and it includes a data.json file, and you can also see the hash of the content of each file. So this is simply how the signatures.json file is structured. So now what you'll see, so this was a signed bundle and how it looks, but now let's look into how OPA actually verifies that signed bundle. So whenever OPA receives a signed bundle, it's going to grab that signatures.json file, open it up, and the first thing it will do, it's going to verify the JOT signature with the appropriate public key or secret. So you need to provide OPA with this public key or secret out of band, and you will see how that is done during our demo. Next, OPA will try to verify that all the files which have been specified in that JOT payload actually match the files in the target bundle. So it's gonna make sure that there are no extra files or there are no lesser files in the target bundle to what's mentioned in the JOT payload. And finally, it's going to take every file in the target payload, in the bundle, generate a hash for that file, and then compare it with the hash recorded in the JOT payload and make sure they are equal. So if any of these steps fail, OPA is not going to activate that bundle. And so only if all these steps are successful, OPA is going to activate this bundle and then it's gonna start using whatever policies and data are included in that bundle and start enforcing those policies immediately. So this is how OPA verifies a signed bundle. So we've also provided some command line tools, some tooling that can help with signed bundles. So we have a brand new OPA signed command that you can use to generate signatures for the contents of a given directory or an existing bundle. And all you have to do is provide it with the signing algorithm that you want to use. By default, that's RS256, but there are others like HS256 and much more that you can use. And also we need to provide the signing key. So if you're using an asymmetric algorithm, you provided the private key. If you're using a symmetric algorithm, you provided a secret. And the sign command will generate the signature for all the contents of the given directory or the existing bundle. We are all familiar with the OPA build command, which is used to generate regular bundles. And now you can use the OPA build command to also generate signed bundles. And it just takes the same parameters as the sign command. You give it the algorithm and you give it the key and it's going to generate a signed bundle for you. And finally, if you have to verify a signed bundle, you have the OPA run command, which takes in the signing algorithm and the verification key and it's going to verify that signed bundle. So if you're using an asymmetrical algorithm like RS256, you provided the public key. If you're using something like HS256, which is a symmetrical algorithm, you provided the secret. So these are some of the command line tools and the tooling support that OPA provides for signed bundles. Now, let's see signed bundles in action. So to set up our demo, what we have is a healthcare application by Acme Healthcare, which allows its members to see their invoices. And we are going to enforce a very simple policy which says that the claims in the invoices should be hidden from members. We will see the simple policy later, but all it does, it says that we have to hide this claims inside the invoice from the members. The policy will be distributed to OPA using bundles. And let's see the scenarios that we are going to address in this demo. So what happens when a malicious user updates a policy? We're gonna see that in the demo. How does that compromise the healthcare app? And finally, can signed bundles provide a compensating mechanism? So we are going to try and answer all these questions as part of the demo. So let's jump into it. So what you see here is essentially a dashboard which allows a member Alice of Acme Healthcare to see her invoices. There are two buttons here. The first one, it simulates a scenario without OPA in the picture. And second one simulates a scenario with OPA actually enforcing certain policies. So if you were to click the first button, you can see that there are a couple of invoices and pay careful attention to the claims. You can see the claims that are part of this invoice. And now, if you were to click the second button, you would see that the claims are hidden, which means that the OPA policy is getting enforced. So with this base in mind, let's see what happens if a malicious user now tries to corrupt the policy. So say you have a malicious user who simply changes the policy, okay? And this malicious user now creates a bundle with this corrupt policy. So this bundle is now going to be pushed to your bundle server. OPA is going to pull it down and it's going to enforce the policies which are in that bundle, which are now corrupt. So if you go back to our sample application and if you click the same button again, look at that. You can now see the claims in the invoices, which means that the corrupt policies which were pushed by the attacker are now being enforced by OPA. So how can you prevent this disastrous scenario from happening because now you can see your system is compromised? So how can you prevent this scenario from happening? And again, how can you trust the decisions that OPA is making? Because OPA simply trusted the policies which were in that bundle, those policies turned out to be corrupt and now OPA is enforcing decisions which are compromising your system. So how can you prevent all these things from happening? Will signed bundles help in this case? Let's find out. So let's revert back to our original policy. And what we do now is instead of creating a regular bundle, we'll create a signed bundle. And you can see that we are providing the algorithm. We are providing a private key and we're gonna create a signed bundle now. So if you look, the only difference is that you have a signatures.json file which is included in the signed bundle. So now this signed bundle has been pushed to the bundle server. OPA is gonna download this signed bundle and now verify the signed bundle with the given public key that we have to provide. So let's see how that actually happens next. Seeing now is the runtime configuration that is provided to OPA. And you'll notice that we have provided the public key inside of this runtime configuration. And since we're using RS256, we are providing the public key which is going to be used by OPA to now verify that signed bundle. So let's see if that worked out. We'll check out OPA's logs and see that. So you can see that the bundle has been activated. And so now we can go back to our demo. And so again, if you simulate a scenario with OPA in the picture, it's hiding the claims again. And so the reason this happened is that we generated the signed bundle. OPA verified that signed bundle and enforced the policies inside that bundle. And the policy said, do not show the claims inside the invoice object, which is great, which is what we wanted. So now again, the malicious user comes in and updates the policy. So let's do that again. So now we have the malicious user coming in just like before, updates the policy and builds the bundle. And now this bundle has again been pushed to the bundle server and has been downloaded by OPA. So the question now is, like before, will OPA just activate that bundle and start enforcing those current policies or using signed bundles helped? So let's check it out again by looking at the OPA logs. So if you look at the OPA logs now, you'll see that there are a bunch of errors in that log. And these errors in the log are happening because OPA could not verify that bundle which is just downloaded. And when OPA cannot verify the bundle which is downloaded, it's going to keep on using the last downloaded signed bundle and enforce the policies from that bundle. So now if we press this button again, you can see that the claims are still hidden, which means that those corrupt policies which were fed by the attacker were not being enforced by OPA. And so in this way, we have prevented an attacker from feeding corrupt policies into OPA by using signed bundles. We used signed bundles in this demo from a security perspective to prevent an attacker from feeding corrupt policies into OPA. You can also use signed bundles for other use cases, like if you wanted to rotate the keys inside a non-discovery bundle, and there are other applications of signed bundles as well. So this was a talk on secure policy distribution with the Open Policy Agent. I hope you all saw how signed bundles can help in reducing OPA's attack surface. If you'll want to learn more about signed bundles, please check out the OPA website on openpolicyagent.org and check out the documentation about signed bundles. Join us on Slack if you have any questions and also check out the project on GitHub. And if you like what you see, please do star the project. Again, I thank you all for joining me today and I'm happy to answer any questions.