 Hey, everyone. Hey, Brandon. Hey, Barak. All right, let's give a few moments for everyone to join. I'm going to paste in the meeting link. Just put the name in down for attendance. And if anyone can help scribe today, they'll be great. Today's agenda is we have a presentation on check-off. It's just going to be done by Barak. All right, we're going to wait a couple more minutes so that people can slowly trickle in, since Zoom has some joining issues sometimes. While we're waiting, is there anything that anyone would like to bring up? We already have one thing on the agenda next week, which is to talk about the DevSecOps Pipeline reference by Vinay. Do we have anyone else that would like to put items on next week's agenda? We're still waiting for one or two scribes to help up today. So if you can help scribe a little bit, they'll be great. All right, let's start with just kind of going around with updates. I see most don't have updates on just in comment. Do you have something that you wanted to talk about? No. OK. And Cameron? Sorry, no updates. OK. All right, cool. I noticed that we have several new names at least that I haven't seen before. If you're new, would you like to introduce yourself? Sure, yeah, I'll go first. I just started joining these meetings. I'm Trishan Kubsami. I work with some CMTF projects such as stuff in Toto and CNAP and I'm a security engineer at Datadop. Pleasure to join all of you. Real fun, Trishan. All right, thanks, Ray and Ash. Sorry, go ahead. Yeah, so I just had a quick update on the OPPA project. So OPPA is looking to graduate and will be formally applying for graduation in the next few weeks. So if you all have any questions or concerns, please let me know. And I think that's pretty much my update. Thanks. Just security questions or can we ask other things about OPPA? Oh, anything is fine. Like if you have any OPPA questions or want to know about OPPA, you can connect offline or anything else, specifically graduation. No, we're using it. We're planning on using it in a pipeline at the large financial institution where I am. And I think the question that's been coming up for us is management of the policies and like at scale. And is there any discussion? I didn't investigate too much on the website, but is there any discussion about how you generate context, i.e. with the integration of a graphing database back end? Does that make sense when I'm talking about? So yeah, I think there are two separate questions, one around policy management itself at scale, the other around integrations with the graphing database. Is that fair? Yeah, the graphing database, in other words, to provide context, I don't know if those questions have come up before both of those. So yeah, the first one definitely has come up like many times around scale and bundle distribution, around policy distribution. And we have few mechanisms for that. A feature called bundles. I think we can chat after this call on Slack. I can probably more details on that. And regarding the graphing database, is something like Grafana or something? Is that what you had in mind or? Neo4j, I mean anything, but something that would provide, like as you're doing the validation, I think OPA gets closer, but what concerns me is the absence of context to some degree, and maybe I'm missing that, but I don't see that right now. Maybe it's documented and I'm just not graphing it. Why don't I reach out to you on Slack after the call? Are you on the six security Slack channel many times? Yeah. Awesome, so I'll reach out to you. I'll thank you after the call. Oh, okay, thanks. All right, cool. And also OPA has done the security self-assessment as well. So if you haven't already seen that, you can get that in the six security. You can have, I'll place a link in the leaving notes. Thanks, Rindan. All right. All right, if not, I think let's go ahead with today's presentation. So we're gonna have Barack and Guy, who are gonna talk about a check off, which is a static analysis tool for infrastructure as code. So are you guys all set? Yeah. Thank you, Rindan. And thank you for setting this up. Let me just share my screen. I'll be showing those stuff alone tonight. So my name is Barack Schoster. I am the CTO and co-founder of a cloud security startup named Breach Room. And we have various open source tools. Among them is the check off open source that we would like to discuss about today in presenting. So a little bit of context. So check off was originally developed at Breach Room. It was released at last Christmas under the Apache 2 license. It is currently maintained only by three employees and it covers configuration frameworks like cloud formation, Kubernetes, serverless, and ARM with more than 360 security policies written. So check off is a static analysis tool for those infrastructure code frameworks or configuration manifests. And we would be happy to add check off into the CNCF foundation. So what's check off? Originally we built it because we looked for an internal tool to enforce policies over our Terraform code within Breach Room. And we saw like a lot of open source tools at the time. But not all of them had the content meaning the rules. For some reason it was like great engines but the content, the rules themselves that are usually common in a lot of organizations either based on the CIS benchmarks, SOC2 PCI best practices, the AWS foundation or the GCP and Azure. And what we decided to do is just to write a bunch of, we wrote at the beginning 50 checks that validates different configurations. And we open sourced it and some people contributed more and some people joined and contributed even more. And the basic principles of it is that we want to just open source off the common context of the common rules to the community. The engine obviously is also open source. Everything in check off is open source. And we were more familiar with our security background. We were more familiar with Payton as programming language. And we found it very easy to extend, to use inheritance to do API calls in some of the cases. So this is the language that we chose to build in a check. So how does a check look like? Over here there is an example for a Terraform policy to enforce encryption over RDS resources. So it's like, I don't know, something like 10 lines of code for a check. You mentioned the resource that you'd like to monitor, the AWS DB, you give the check a name and then ID and a category. And basically what you will do is to take the parameter in the configuration of that resource and you will enforce whether it is, you will check whether it is configured correctly, meaning with storage encrypted equals true. And then the check is passing. If it does not have storage encryption or is configured with storage encrypted false, the check will fail. And what check off will do is basically run just like a test suite in your CI CD pipeline on your period commit hooks or in other use cases that we'll see in the following slide. And we'll validate all of your infrastructure code against all of those 300 and growing checks. So some interesting statistics of stuff that we've done with check off is we decided to scan the Terraform registry, like the public registry registry.terraform.io, which is almost, it's something like 3,000 models and providers at the time we made the scan, it was 2,500. And what we found that by those 360 checks, 44% of those modules published at the Terraform registry were misconfigured. Meaning had either encryption issues, logging issues, IAM issues. And that's a lot because we also took some metrics about the downloads of those and they had several millions of downloads, meaning several millions of potentially provisioned resources that were the manifest were downloaded from the Terraform registry and had some kind of a security issue. And those are the percentage like logging is a common missing piece in the Terraform registry modules, backup and recovery, encryption, Kubernetes issues because it's hard to configure Kubernetes correctly, networking, some minor IAM issues. And it's not because people have written the coding Terraform, it's actually helping to identify and solve some of the issues at scale. It's because they lack the knowledge of how to configure it correctly or lack just the tool with all of the content. So you can have an engine to write rules in, but without the content and the rules, it's really lacking. So we really wanted to give it as high as possible coverage to all of those different categories. Common examples would be the open S3 bucket issues that are highly famous, but also encryption issues, but also did you remember to turn on access logs or VPC flow logs, et cetera? All right, so how do you install checkout? It's really simple. checkout is a PIP Python package. So you can just PIP install checkout on Python 3.7 and up. If you're using Mac, you can use brew. If you'd like, you can use Docker. So you can basically install it on every modern environment. As for running it, it's also pretty simple. You just need to choose a directory where your infrastructure code is in, infrastructure code is in, where you're Terraform, Kubernetes, CloudFormation, Serverless, or ARM templates are. And you will just get a full colorful report in CLI as a beginning with each check, its ID, whether it's past or not, the lines that are problematic and if it's problematic resource, just a print of those lines that you should probably fix or choose to skip or ignore. It also has a JUnit XML reach format, JSON format. You'd have marked down where you can put the table of the issues and using JUnit XML, if you integrate that into build pipelines like code build or Jenkins, you can get the full colorful report of the JUnit XML like you can get from a unit test or a test suite on your CI pipeline. All right, so if you'd like to take a look on those 300 checks of different resources, you can just go into the website, into check of IO scans, resource scans, and it gives you the full list divided by the different frameworks and different resources. Some might look similar, for example, the check for encryption in CloudFormation and the check of an S3 bucket and then the check of encryption in CloudForm. A S3 bucket will have the same name, but they have different implementations because you're writing your resource in a different manner in this configuration file. So like every test suite where you can, for in Java, for example, you can choose to JUnit Ignore a specific test. You can use annotations of comments and write check of skip and the check ID and there isn't just for documentation, it's mandatory. And you can choose to skip a check over a specific bucket. A sample use case would be if you have an S3 bucket that has public ACL configure and it's publicly accessible from the entire world. In some cases, it's a legitimate act. For example, if the S3 bucket is serving as a public static content host, even though there are other ways to do that, it is a possibility. So the bucket is public and it's legitimately public and you can tag it by one by just writing this comment inside your Terraform code and you can choose to skip this resource from this public S3 bucket check. You can also skip a full category of checks. If you'd like, for example, if you don't want to enforce encryption for some reason or if you don't want to enforce putting description on every security group, you can just choose a check and skip the whole category without editing each and every resource just as another parameter of your CLI. So checkup has four basic execution modes. One is the local meaning running on CLI on your terminal, on your laptop. Another one is a pre-commit hook. Another one is CI-CD pipeline and the fourth is inside the Kubernetes cluster. And we're going to drill down each one of those to see the use case and how that can be, how that can help actually to reduce the amount of misconfig in the cloud. All right, so let's start with a pre-commit hook. As an engineer, when I'm writing code, I'd like to get feedback as fast as possible on how good my code is. So we generally use linters like TFLint or CFNLint or linters for Kubernetes or even for our node code. So checkup can interact just as another linter where you can configure it in your pre-commit hook and it will block you from committing either bad turn from code or malicious or if you, by mistake, have for that a specific security configuration on your code, you can just run checkup as a pre-commit. It's important to say that you can also do other use cases with checkup other than security. Backup and recovery can always be tested using checkup. You can even write custom policies over pre-meeting only specific types of EC2 instances to reduce cost, but the main policies that you'll find there are around security. There are some around cost like, don't forget to put retention over your logs. All right, so using the pre-commit hook, every engineer in the organization would not be able to commit a new infrastructure code without approval or rejection of checkup. After an approval, meaning checkup has scanned this local code, you can push the code into your Git repository, GitHub, GitLab or Gitpack. So that's one scenario. Another scenario is just putting it inside your CI CD pipeline. So let's say that I don't have a pre-commit hook, but I have opened the full request in GitHub or GitLab and now I want to run my tests. So your CI system, whether it is GitHub action, Jenkins drone or anything else, can run checkup on every full request. And it's just running the infrastructure security test. If it rejects, it fails the build, you cannot do the change to your infrastructure code. If it approves, you can start your deployment trigger and just apply your infrastructure code into your production environment. Well, I know the drawing is focused on AWS, but it is the same for every cloud. Since Terraform is multi-cloud and checkup is scanning configurations across the, or three main cloud providers. So by implementing this kind of pipeline, you prevent any change to your production environment that is not validated at least in some aspects of security configuration issues. The third mode is running checkup inside a Kubernetes cluster. So since checkup can analyze Kubernetes manifest, you can scan those on build time while you're writing the code. But in some cases you will use templating frameworks for Kubernetes clusters like Helm, Customize, Rancher or GitLab provided Kubernetes automatic DevOps. And you'd like to know if your runtime environment in your provision Kubernetes cluster has the right configuration. So you can just deploy checkup as another container in your cluster with a specific configuration YAML and checkup will actually do API calls to Kubernetes to the cluster that is deployed within. And we'll check if the Kubernetes and we'll download locally the Kubernetes YAMLs and we'll check if they are valid or not valid from security perspective. So the main advantages of using checkup which is a policy as code engine is using a version control policy engine where you can add custom policies if you'd like. You can run the best practices that the community provides. It can be peer reviewed. Since it's written in Python, it's very easy to use inheritance or do custom stuff like API calls. One of the stuff that we're doing internally with it with a custom check is every time there is a deployment to production we're running a check whether there is one that is currently happening or not. And this is just another check implemented as part of the CICI CD pipeline. And it's very easily integrated into the different CI tools. Another advantage of using such tool compared to into runtime analysis, for example, a tool that samples a configuration in AWS or others is that it's decentralized. It's not centralized in the security team where you need to go over a bunch of alerts and correlate to the user that made the change. Since it's running on each and every commit and each and every pull request, it's distributed by design because it's part of the CI CD pipeline. So if a user done commit and the commit failed, you will see that on GitHub. Any questions before going to the project's roadmap? All right. So the roadmap contains those current items that we're open to add more is the first is policy sharing. Currently, Chekhov has 360 built-in policies. It has the ability to load additional policies from an additional directory. Just as a parameter, but a lot of community members ask for the ability to load policies from a GitHub repository meaning a directory that is version control and the end that is not located near the infrastructure code repository itself. So this will enable enterprises just to have a centralized Git repository where custom policies are. Another one is we do support scanning ARM templates, but we don't have a lot of content there. A lot of policies written. So we are working on adding more policies there. We want to support natively home. We want to add a relationship engine. For example, in Terraform, managing VPCs and VPC flow logs, those are two different resources. We would like to be able to write a policy such as does each VPC that has a public-facing resource has a VPC flow log. So this is our relationship engine or a graph engine that we would like to add. We would like to add COOP CTL plugin and Kubernetes admission controller just to have another way to interact and with the Kubernetes cluster configuration and be part of the pipeline of managing it. As for the project statistics, so it was released six months ago. It has a family on downloads, 976 GitHub stars, 32 contributors, 250 pull requests and the test coverage are about almost 90, 86%. You can, you're more than invited to join the Slack or visit the GitHub project and try it out or the homepage of the documentation website. All right, questions? All right, thanks so much. I think that's a couple of questions in the chat and if you want to start looking through them. Yeah, sure. Hi, this is Michelle. You answered the question about the admission controller in one of the slides. If you don't mind me just jumping in, but the challenge that I keep having and I see this everywhere is once you get to the issue of scale, you have problems with context and ontology and flat rules don't, at some point, it's gonna hit a wall because it doesn't have that context. What are you thinking about for that? Yeah, so what we've seen from running that on large organizations, so the organizations that are using it are members, teams inside AWS, teams inside GCP, Salesforce, I know that they've executed some of those. So what we heard from those is that a lot, like 80% of the common issues can be caught using flat rules. So let's say that you have a lot of engineers in your organization and all of them or a bunch of them are writing Terraform code and you don't want to review every pull request. You get like the 80% of the issues which are forgetting to put encryption or forgetting a public resource as part of this open source tool. And we would like to add a graph ability or the relationship engine that can give you the ability to write policies with more context. So for example, activate a specific policy only on public-facing resources which means the correlation between security group rule and the security group rule, security group and the EC2 that is attached to it. Yeah, so essentially you're saying create more of an engine. Right now it's not really that you can't really, I mean, I don't want to be offensive but you can't really call it so much of an engine but so you're saying create an engine that you can create, it can have an idea of context. Is that what you're saying through a relationship engine is what you're calling it, right? Yeah, so currently we have an engine for flat rules and the content over those infrastructures code and we would like to have an engine over relationship rules, that's right. Yeah, sure. A quick question for our great presentation, by the way. So my two questions, one is if I need to update the rules, I need to encode them in Python, is that accurate if I want to add more rules or more policies? So there are two options, one is to always run pip install minus you and get updates to the community rules automatically just by updating or pulling the latest Docker that's also an option. And if you'd like to add another rule that would be a rule in Python, yeah. Okay, so the way to add custom rules is by writing them in Python. Yes. And have you compared this with the Conf test project by any chance? Is there any comparison or do they complement each other? So I was not aware of Conf test when I started check on, I was aware of OPA and I know that Conf test is somewhere related to OPA. So I, and I see that there isn't in the chat a question about OPA. All right, so OPA, as far as I know and feel free to correct me, to use OPA you'll need to have a deep plan generated of your tariff for example, which means that OPA will be able to run test over the rendered variables inside or the evaluated variables inside the manifest. And check-off has some logic to evaluate all of the variables by the default values without applying the plan. Why we wanted to develop it that way? Because in a lot of times when developing on your endpoint you don't want to put secrets of production environment which are usually variables or somewhat injected into the plan on those endpoints. So both Conf test and OPA as far as I know requires those variables or plan files. And since we wanted to execute check-off everywhere we couldn't rely on those. So I think that those two tools are completing each other where you have OPA for, you have check-off for plain manifests and the relationships between those. The engine does a variable evaluation and not only flat rules. So there is a relationship graph between variable dependencies and resources. There is just not one yet for between resources. Yeah, so they are completing where OPA does the more runtime like analysis or almost dynamic because it relies on the production values and check-off is more like a static analyzer of manifests. Yeah, so the comparison not between OPA and check-off but between specifically Conf test and check-off because I think Conf test is for managing configuration files if I'm not mistaken. I'm not certain about the terraform use case in particular but I think the purpose of Conf test is to validate configuration files like you shift left basically. That's the idea with Conf test. So I was just curious if it's specifically versus Conf test and check-off, not OPA. OPA is completely different one. And thanks for your answer. Yeah, sure. So Conf test, as far as I know, does not have... So I know that there are repositories with bundles. We tried to create one main repository with a huge bundle of checks of best practices. Okay. But I think that basically it has very similar capabilities. Good question on the... In Europe, what's your sort of subject to at least in terms of terraform? Subject to all the vagaries of terraforms operating and formatting and dynamicism with regards to... With regards to like, you know, doing four loops and the count modules, is that something that is currently handled by check-off? So I'm sorry, I haven't heard you well. So I'm going to repeat the question just to verify if I heard the right one. The question is whether check-off can run on internal modules, like modules that terraform reports? Sorry, sorry, I just switched my speaker mic. Hopefully that's better. But the terraform language, HCL, has a lot of dynamicism, doing four loops and conditionals and a lot of that to actually write a policy you need to actually evaluate all of that to be able to check and attribute. And even like the Python HCL2 parser doesn't even get like operator precedence correct. So it's a little unclear to me how robust that is outside of using the Go library for terraform. So check-off has some Python classes called evaluation that handles some of those use cases. For example, importing a variable between inside a resource or any tests on traversing logic over modules. So some use cases are already covered within the engine but not all of those. So it's basically, we're using the Python HCL2 to parse the file and then we have additional logic to traverse over files and correlation between resources and variables. There is still work to be done to make it perfect. Does that answer? Hope it is. It did. I think like in actually using the HCL library it was clear that it's not correct in all cases. And so that's why I was just wondering about have you done validation against sort of more common community modules of terraform just to validate that check-off is correct? So we have compared that to TFSEC which is reasoning Go. And we got very similar results when running on common vulnerable by design projects like Cloud Goat, TerraGoat and Kubernetes Goat. So Kubernetes Goat is not versus TFSEC but we have validated using those projects. Thanks. All right, cool. Any more questions? I guess that I would love to know how can the process of submitting check-off into the CNCF can probably look like from here? Yeah, so you mentioned to me that you guys are looking at incubation, right? So from our side I think what we can do is to do a security assessment as part of that process. So I would create a, if you go to issue you can create a security assessment there and there's a couple of steps here to follow but at least from CNCF security side our recommendation is for incubation and graduation is based on kind of doing the security assessment and the results from that. So I have a question here because you all have the application in for the sandbox process, correct? So sorry, we can submit either to sandbox or to incubation. You have already submitted for sandbox. Yeah, I haven't gotten a reply yet. Right, so the next time that we are planning on being able to review is going to be July 14th that will be next week. Right. So I was curious about like the, if you were actually looking for a sandbox or if you were looking to be able to come in as incubation. Got it, what would be the main difference between the two? The incubation process is the one that kind of the, I think of as Brandon was talking about as far as being able to go through the assessment there's a security review that would be involved here as well as being able to find a TOC sponsor to do due diligence. And Justin Cormack can correct me if I am speaking out of turn for any of that. The sandbox process is the process that we've just changed requires TOC to review and it's just a simple vote. Yeah. That's right. Cool. I see Kapil's comment over the integration with Bridge Crew on the chat. So it's a non-mandatory parameter. Yeah, but just in terms of going to CNCF and having a hard coded implementation to your corporate product that seems a little odd. Got it. So it's an integration whether you choose to integrate that with the commercial offering or not. It's a choice of the user. Most of the users are not. So come back to the question. So are you looking to just submit for sandbox or incubation or is there something that you guys have to just take a look at and decide first? I'll have to take a look and decide first. Okay, all right. Then whatever fits. I think that the natural progression is usually sandbox and incubation. And then for sandbox, the security assessment isn't required, but nothing's stopping you from doing it. And that's something that we recommend anyway, because when you try and go towards the incubation, that is required anyway. But yeah. Right. Cool. Cool. I think sandbox is gonna be the right place to start here. You know, the sandbox incubation and then graduating. And by starting out at sandbox, you're gonna get more orientation. By participating, getting a security assessment, you're gonna have the opportunity to really expand your exposure to folks in the CNCF. You're basically going to be going around flipping bits and getting buy-in from folks in the community. So the more you can be visible, the more that folks throughout the ecosystem recognize that your contribution, the easier subsequent stages are going to be. So I think sandbox is the right place to start. Right. Cool. Thank you. Okay. Thanks so much for the presentation. It was good. Is there any other topics that we'd like to bring about today? If not, we can call this 15 minutes. I thought we should have Michelle talk about that ontology. She's building. Well, it sounds like a topic that can... For real. It's really open-ended though. I feel like we could spend like an hour on this. Seriously, do people want me to really ask about that more? I love a good ontology. Yeah. For me, it's a struggle, right? I, it's a, I mean, in a way it's kind of a graphing problem. And I, because when you don't have context, I mean the flat roles in themselves out of context don't always mean something. It's just a problem that I had personally had. It's a problem at scale, right? So that you can eliminate, so you can make sure that you're not generating a lot of noise, false positives. The same as you have with SAST. I mean, you could consider it like a similar problem through contextualized or temporal risk scoring. Yeah. Unfortunately, the ontology forum occurs at the same time as this meeting. So we probably will never solve this. Yeah. That sounds like a easier problem to solve than the actual problem itself. Put it under the heart. All right. If not, next week we're gonna, Vinay is gonna be going through kind of a DevSecOps reference. And this is kind of tying into the work that we talked about with the DoD recommendations for CNCF. And that's kind of like a vendor perspective on it. So that's something that Vinay is gonna present next week. And of course, we can reserve the rest of the time to talk about ontology. If not, I think that's it for today. Thank you, everyone. And thank you, Brock. Thank you. Thanks, Brandon. Thanks, Brandon. Thanks, Brock.