 Howdy, everyone! Welcome to Cloud Native Live, where we dive into the code behind Cloud Native. I'm Taylor Dolasal, a senior developer advocate at HashiCorp, where I focus on all things infrastructure, application delivery, and developer experience. Every week, we bring a new set of presenters to showcase how to work with cloud-native technologies. They will build things, they will break things, and they will answer your questions. Join us Wednesdays at 11am Eastern Time. This week, we have Amir here to tell us about Cubescape, an open-source tool to test Kubernetes deployment security. This is an official live stream of the CNCF, and as such is subject to the CNCF code of conduct. Please do not add anything to the chat or ask any questions that would be in violation of that code of conduct, and basically please be respectful of all your fellow participants and presenters. And with that, I'd love to hand it over to Amir to kick off today's presentation. Amir, take it away. Thank you very, very much, Tyler. So I'll start by sharing my screen. Can you see my screen? Yes, that looks good. Great. So thank you very much, and welcome, everybody. I'm going to talk about Cubescape. If you're not familiar with Cubescape, Cubescape is an open-source project that Armo released, and we maintain it together with you guys and ladies, of course. And I'm going to share a very, very, very short presentation, kind of explaining what Cubescape is and why we developed it in the first place. And then I'm going to share a little bit on what's coming next, and then we're going to deep dive, and I will show you real use cases of how to use it and what you can get out of it. So Cubescape is the first open-source tool that scans Kubernetes according to multiple frameworks. The idea is that you can use Cubescape to scan, I'm sorry for that. You can use Cubescape to scan your cluster. You can use Cubescape to scan YAML files, handle charts, and it gives you a really, really nice report on what is misconfigured and how you need to remediate it. It is really, really skyrocketing in the number of Git stars that we got and the love that we got from the community, and I really hope that it will continue. We have over five, 15,000 developers who are using it to monitor their clusters, and you can see the name of the company and the persons who are using it, and I think that it's quite obvious why. The reason is because misconfiguration is one of the causes for data breaches. This is according to Gardner, the number one cause for data breaches, and as you can see from a survey that Red Hat conducted not a while ago, you can see that people are worried about misconfiguration in their clusters. That's why we created Cubescape in the first place. Now, what Cubescape gives you today? If you didn't try it, I urge you to go and try it. It's free open source, and it will stay free and open source forever. It gives you the ability to scan cluster, YAML files and Helen charts. It has multiple frameworks. Right now, we support MITRE for Kubernetes and the NSA and CISA frameworks. We give you an ability to create your own framework, and I will be talking about it later. It can be inserted to your CICD pipeline, and as I will show you, you can identify drifts from one scan to the other. When you're scanning for misconfigurations, most of the tests that are being done are very, very generic. This means that in some cases, you will find unreasonable stuff in their results. For example, it will tell you that a Q proxy is privileged. Okay, that's how it's supposed to be. In this case, we created exceptions that will let you personalize the results, and it will make sure that you're focusing on the right things that needs to be remediated or fixed. We give you a very easy risk score that you can follow and see the trends over time, and we show you the history of all the tests. We understand that we are part of an ecosystem, so that's why it is integrated into CICD pipelines. I will show you how easy it is to deploy it and use the UI. The UI is super friendly and super easy to understand and work with, and the cool part is that you don't need to install anything on the cluster. It's just copy, paste, run, and it works. The next things that we're going to develop are, of course, more frameworks. This is where opening a request to you guys. If you have anything, any framework that is of interest for you, please shoot us an email, and we will definitely try to create it for you. We are going to extend the capabilities to support more tests or controls that are based on the worker node and API server settings. We are going to add, actually, we got help from the community and the community, and so on in the community developed a plugin for one of the managed Kubernetes settings. Again, if you want, we would love you to contribute more in the project. We are going to do Slack integration so you can send your scan results to your Slack channels. We're going to add recommendations, meaning that when you look at the exceptions, sometimes it's hard to understand if something should be in an exception or not. What we are going to do is we are going to tell you, hey, this is a normal behavior. Kube proxy is supposed to be privileged. You can put it in an exception, and you're good to go. It's not a real misconfiguration. We are going to give you the ability to run a Chrome job, a CubeScape as a Chrome job, and then scan your cluster on a regular basis. We are going to give you the ability to set the controls based on your organizational need. For example, one of the controls that we're going to introduce soon is the ability to choose which container image repositories you're willing to accept in your organization. Of course, it differs between one organization to the other, and this is where we're going to let you set it according to your specific organization. We understand that risk posture misconfiguration is changing over time because the cluster is changing over time, and we want to show you the major changes in your cluster. This is the inventory information. We are going to show you in the graph, which I will show you in a second what kind of big stones are happening inside your cluster, like a new namespace or a new deployment is being introduced. The last thing that we are going to do before Christmas is the RBGraph. We understand, and this is a feedback that we got from a lot of customers, that managing RBGraph understanding, following RBGraph in Kubernetes is quite challenging, and it involves describing a lot of Kubernetes objects like role, roll binding, etc., and then building the puzzle in order to understand what the role is doing, what the subject is doing, etc. What we did is we take this data out of your cluster and we build a graph out of it, and we give you the ability to walk through the graph and understand what are the capabilities. We give you built-in queries and that you can simply query who has high privileges, who has over excessive privileges, etc. This is the near-term roadmap that we hopefully we are going to deliver by the end of this year. Then we have the long-term roadmap where we are going to again add more frameworks, because this is the gist of the tool to give a lot of frameworks for misconfiguration. We're going to add container image scanning, again same feedback that we got from you. We are going to integrate the Kubernetes audit log in Iran detection policies against it. We're going to add policies that will be, instead of just identifying issues, it can really prevent these issues from occurring. The last thing we are going to add auto remediation, which means that you will be able to say, if I have this issue, please remediate it for me. That's a presentation and I finish the presentation. From now, we are going to go through the demo. I'm going to switch over to my browser. I'm going to share my browser in a second. Tyler, can you see my screen? Yeah, it looks good to me. Okay, great. What I did here is I have a Jenkins CRCD pipeline. What we did here, we created an environment where we scanned the YAML files before deployment. Here you can see I run my pipeline. I set the threshold to be 70%. This is the threshold that if my re-score is above 70%, sorry, if my score is above 70%, I cannot continue. If it's lower, I can continue. I'm running my pipeline and you can see it is failing. The reason that it's failing, and you can see that this is the command that I actually ran in my pipeline. You can see that my score was 15, and it was lower than 70. This is why it failed. If I go to the tasks, you can see the different controls that were tested. You can see that, for example, the checkout service is mapping the host PID, which is not a really, really good practice. We failed the pipeline from continuing. If I go back and I run it, and now I will set the threshold, let's say to be 50%, which is the score that I know I'm going to get. Now it will work. I'm setting it to 50, and you will see in a second that the pipeline succeeded because we got 50 and the threshold was 50, but you can see that the same tests have failed. Now let's go to my cluster and see how my cluster looks. This is my cluster. It's running with GKE. You can see I have a namespace called shop. If I look on the pods, you can see that they are running. Basically, what we did here we deployed the hipster shop application by Google. You can see I have an external IP. Please don't try to hack it, but you can enjoy, of course, connecting to it. Now I'm running it, and this is my front end, and you can see that I can buy things for Christmas. The application is working, and let's say that I want to scan it. I'm very familiar with a good tool to check it. You can go to Cubescape, and this is the Git page for Cubescape, and you can see we have basically all the information you need here, and some of it in the documentation that I will show in a minute. Deploying Cubescape is just taking this command and pasting it into my cluster, or the place where I'm running a CubeCTL, or I have a CubeCTL installed on. In this case, it will be here. Now it asks me to run this command, and so I copy and paste it, and it's scanning. That's it. I get the results. You see that the result here is 72. It's a little bit different from running it in the CICD, because some of the tests require real objects like, for example, network policies. We cannot test it when we have a YAML file, but we can test it when it is deployed. You can see here that for each one of the controls, we give you a status if it failed or succeeded. We give you a description of the control. We give you the namespace, and then the deployments that failed, and then we give you how many assets we scanned, how many failed, excluded. I will talk about exclusion in a second, and how many have passed. At the end of the report, I get a nice link. The idea is that I can send the reports to the UI, to our UI, or I can keep it in the CLI. By the way, we have a lot of options here. Customers, users who are savvy on privacy can use things like keep local. They can download the frameworks and run it without any object being or result being uploaded to us. You can read everything here in the documentation, which is quite extensive. Coming back here, if I click here, basically, I jump to the UI. In my case, I jump to the UI because I'm a registered user, but if you're not a registered user, basically you get something like this. It's quite easy to register. You can register with a username and password, or you can register with Google, a GitLab, or a Microsoft. Coming back to my case, you can see the UI. The UI is quite easy to understand. I ran so many scans today because of this demo. You can see them here, grouped, but usually people do not run as many scans as I did in one day. You can see the graph. You can see, for example, that in my case, the trend, my risk is going up, which is not good. I would prefer that my risk would go down. You can see here that since I ran a lot of tests, it went up and down and up and down. You can see that you have a score for each one of the frameworks. We're telling you how many controls failed out of how many tests that you have. In the next release here, you will have the ability to add your own framework and choose controls out of the existing controls to create frameworks that are suitable for your specific organization. If you look at each one of the controls, you can see if it passed, if it failed. You can see the control ID, and when you click on it, you get a detailed documentation telling you which framework it is running, the description, the related resources that we are scanning in order to see if this control passing or failing, what we test exactly, the remediation, and soon we're going to have examples here that will show you what is a good setting versus a bad setting. We have a description, again, of the control and the remediation of the control. Now, since we're running tests, scans, you can see all the different scans that you did, and you can jump and see these scans and review them. We always compare it to the previous scan. You can see, for example, I ran too many scans today, so probably we will not see any difference, but here, for example, you can see that I had 41 bad resources, and it went down to 18, so it's good. Someone worked and remediated stuff for me, and I'm very, very happy for that. So, I can always compare it. Now, let's say I want to see which resources failed, so I can click here, and I can see the namespace. I can see the deployment that failed and the deployment name. I can set exceptions on a specific resource or on the entire namespace. The difference is that when I do it on a namespace, new deployments that will come on the namespace will be in the exception list, and if I do it on a resource, it will be based on a specific resource. I'm going to show exceptions in a second. Let's say that this deployment, the checkout service should have these settings in our case. I can set it as an exception, and once I set it as an exception and I run scan, I'm going to scan it again, and then it will take me a few seconds. So, if I come back and I need to refresh my screen in order to see the new scan, so this is my new scan from now. You can see now that it will come in a second. It takes a few seconds for this to load, something that we are working on. Really quick, Amir. There were a couple of questions in the chat. Would you prefer I ask them now? Okay, cool. One thing that came in was Kubernetes audit logs. It would be amazing to have them out of GKE, and then a follow-up question was, is there any kind of specific configuration to scan GKE clusters? So, we scan right now what we do is we are connecting to the to Kube CTL and we can get all the objects that the API server exposes. In the coming releases, we are going to have specific integrations with GKE and AKS and EKS. And again, if someone from the audience would like to contribute this code, we will be more than welcome. Did this answer the question, Taylor? Yeah, I think so. If there is any more follow-up, please feel free to ask in Twitch, and I can relay that. I had a few more quick things, and then definitely let you get back to your demo. One person remarked that, I get an error when I try to buy something. Can we debug it so I can order a hairdryer today? I thought that was a good one. And then K9's rules, I definitely agree to that. And then a final comment was, I can never remember how to navigate inside. I always get stuck instead of a namespace looking at jobs instead of pods or something in reference to K9's, but looks like that is all the commentary right now. But if you do have any more questions, please feel free to throw those into chat, and I'll ask them to Amir. But with that, Amir, I'd love to throw back to you. Thank you very much. So this is the control view. We also have the resource view, and basically the idea is that you can choose a resource, let's say deployment, and then you can see which this deployment, the currency service, you can see which controls it is failing on. So it's kind of the opposite view of a failed control is a failed resources. Now jumping back to the command line, we also support setting exceptions in command line. So when you scan, you can decide that you want to scan, but you don't want to scan specific namespaces. So if we go to our git, you can see that we have an example for that as well. So you can see that it has the exclude namespaces, but we also have the ability to do the same exclusion as I did in the UI. We can do it also in the CLI. So we have the exception JSON. And in this case, what I'm telling my QtScape is don't scan these specific namespaces when I'm running the NSA framework. And basically, right now it's a part of the git, and you can see it there, but as soon we will have it in our documentation as well. So I'm running it now with my exception JSON. You will see it's running the same way, but probably the risk score will change a bit. And so when I look at it on the UI, let me see. And I can decide again if I want to see it in the UI or I want to see in the CLI. I think that all in all, we're basically, we covered most of it. If you have any specific question, now would be a good time. All right, please feel free to ask any questions in the chat, and I can get those asked. It looks like right now we don't have any, but I definitely have a few on that front. So QtScape's a really interesting tool. I heard about it just before the session and got to check it out. I thought that was quite interesting. When I delved a little bit deeper, I was curious as to where the idea for this tool came up, and if you have any insight around the formation of this tool. So we've been doing posture for a while. We've been developing it for our own use. And when we saw the NSA, and we were kind of thinking about how we need to release it as part of our product. And then, and then we saw that the NSA released the hardening guide for a Kubernetes. And we thought that it would be a good idea. And, you know, we saw the buzz around it. And we thought it would be a good idea to contribute it to the community, like other projects, like a project that scans the CIS benchmark or other frameworks. And at the end, it led us to a conclusion when we saw that the industry is really using it and you know, they gain value out of it. And we decided that we don't want to stop here. And our goal is to be the company that gives different frameworks that different customers can use and gain value out of it. What we saw is that, you know, we are working in a tech industry, in a tech world. People are super occupied with a lot of stuff. And they are really, really appreciative when you create these controls for them and they can just easily run it and enjoy the value out of it. And now that's our goal. Our goal is to create more frameworks and give it to users. So they would use it to solve real problems in their day to day job. Interesting. Thank you so much. Yeah, it's one of my personal curiosities is always to hear about where the tool came from and kind of what that motivation was to create that or donate that or kind of, you know, work with that. So thank you. Thank you. When it comes to some of the happy pads for scanning and things that might not be as optimal with scanning, are there any systems or instantiations of Kubernetes that make this easier to work with or more difficult to work with? You know, something like AWS Fargate or something like that, perhaps. Yeah, so right now we look at Kubernetes. So probably I'm not sure if it will work on Fargate, probably not. So it will work on any distribution of Kubernetes as long as you have the permissions to run the controls. So for example, if we have a control that needs service accounts or to check your service account configuration, if you don't have the RBAC privileges for service account, this specific control will fail. But other than that, if you're running the cluster and you have a QCTL to the cluster, you can run everything. By the way, we are developing an in cluster component and then customers, users will probably install a name space with CubeScape and other capabilities inside their cluster via hell or any other capability. It's there down the road. It will be out there and it will give more capabilities and features to users who are going to adopt that. I think that one of the early feedbacks that we got from the community was that for them it's super cool to run CubeScape because it doesn't require any installation. So they can run it as I did a few minutes ago. It's easy. It's easy to understand. You can register and enjoy the UI. The UI is super simple. You can make it and adjust it to your organization and we are very responsive on Git. So people that open Git issues with us, we answer maximum 12 hours and that's because most of our development team is in Israel and many of the users are in the US or the far east. So that's the reason. Gotcha. Thank you so much. We had another question come in that asked, is it working in an air-gapped environment yet? Yes. So I started to mention that and please follow the documentation that we have. Basically what you can do is you can use CubeScape and everything is written here in the documentation. I don't want to repeat it. You can use CubeScape. You can download the definition of the frameworks and then once you do that, you can use it to scan via CLI and you can use flags like keep local that doesn't send any data to our SAS UI and you get the same results in the CLI and you can plug it to your CICD. As long as you keep updating the frameworks, you're good to go. You don't need to be connected to the internal. Nice. Good to know. I need to try that out with some of my Raspberry Pi clusters that I have here at my house that I haven't connected to the Wi-Fi yet. So be a good test for an air-gapped setup. Cool. Let me know how it works, Tyler. Absolutely. Cool. One of the things that you went through were some of the recommended ways to run CubeScape. You had talked about a Helm chart. You have that ability to run that from the shell. Are there any thoughts around what the recommended way for most contexts might be for running this tool? Is it via the shell and an operator typing it out or is it a Helm chart or a scheduled run or anything like that? Everyone, different people, different organizations have different use cases and requirements. I cannot tell you how I advise people to run it. We run it as part of our CI CD. We are 100% Kubernetes company. All the things that we do, all our backend is running on Kubernetes. So we use CubeScape in our CI CD pipeline to identify misconfigurations that might happen to us as well. So eating our own dog food. And then we scan the cluster, the cluster is using a Chrome job. And our DevOps engineer is watching it on a daily basis. Soon he's going to have it in Slack. So you will be very appreciative of that. So that's my two cents for you guys. Cool. Cool. Yeah. Thank you so much. I know that the more and more that I learn about technology and the more that I stay in the industry, I keep finding myself saying hearing somebody's problem and then answering back with, well, that depends. And I find that that's a far more common answer than I've ever expected. So it's difficult to get the context correct. And it's really important to realize that everything is trade offs. Nothing is 100% good or 100% bad. There are just different considerations to think through. And that's can be difficult. It's really complex. Speaking of that, do you have any interesting stories around vulnerabilities that you found either personally, like things that were surprising, or any that you've heard from people using this tool that they didn't expect or were just like, oh my gosh, I had no idea this was a vulnerability. So actually, one of the things that we decided as a company, we decided that we want to be always to try to be the first ones to add to CubeScape information on new vulnerabilities, mainly vulnerabilities in Kubernetes. So just to give you an example, last Thursday, this vulnerability, you can see it here on the screen, was published. Oh, I think I muted your screen. And so I can't see that unless you share. So last Thursday, the engine X vulnerability was disclosed. I don't know if everyone knows the engine X vulnerability, but basically engine X has a very, very strong service account tied to it. And in some conditions, you can expose this service account and get the access to secrets, et cetera, et cetera. So it was disclosed on Thursday evening Israel time, probably morning US time or West US time. And on Friday, lunchtime, we released CubeScape with a version that has the ability to test your cluster against this vulnerability. It's super important for us to give the users the ability to understand if they're running something which is vulnerable. And yesterday, I tend to have, I'm the VP producer myself. I'm the VP product of ARMO. And I talk with users from time to time. I'm the person who's nagging the user. No, I'm kidding. I'm the person who's contacting the users, asking them to give us feedback, to give us the direction. How can we help them? And I had a talk with a customer who told me, I ran CubeScape. I found two vulnerabilities in my live cluster. And it helped me to identify it and remediate it. He didn't want to tell me exactly which vulnerabilities, but he told me that he ran it and he found vulnerabilities. It always makes you happy that you created something that gives value. So I hope it answered your question. Absolutely. Absolutely. And I think that that really does make a difference, being able to see that. And nothing feels better too, right? Than being able to see that there are 40 issues and you're bringing that down to 18 or two or zero. That's always a good feeling on that front. And we got your screen sharing again, so you should be good on that front. So if there's anything that you want to go through. Awesome. With that, if anyone else has any questions that they'd like to go through, please feel free to share on that front. And then if there's anything that you're excited about with the project, I know in your slides, you had kind of talked about some of the things that were next up for the project, Amir. If you wouldn't mind bringing that back up, I'm kind of curious to take a look at that again, as well as kind of hear about some of the things that you're most excited about when it comes to what you're working on, what's up next for the project. Can you see my screen now? Yes. I see your browser and the restream tab. Oh, so let me, let me, probably I didn't share the entire screen, so. No worries. I feel like that's still, despite all that we've been through around the world, streaming is still one thing that we're always trying to work out the UX on. So I hope you see my screen now. I don't see it now, but no bother if that doesn't come up. So, you know, personally, I like the roadmap. I created it together with users I've been talking to, but I think that the part that is really, really interesting for me is the fact that we are going to add more controls and put them in different frameworks and you will be able to run any framework that you want. So you don't need to go and, you know, many of the users that they talk to, they collect open source tools and they kind of stitch them together in order to create, you know, a solution that will protect their clusters. It can be that they take one tool to do the image scanning and one tool to do the audit log analysis and one tool to do the RBAC management and one tool for CIS and one tool for best practices and one tool and they end up being with a lot of tools. And we want to be the right tool that gives you everything you need in order to protect your cluster. And that's why I'm very excited when we release a new framework and see customers who are running it and using it and soon they will be able to create their own framework and adjust it to their own needs. So, you know, it's like asking, when you ask me what is my favorite feature in the roadmap, it's like asking the father which son he likes the most. And I like all my kids. It's so true. I know that I'm the eldest of three and I know my siblings and I would always ask our parents, you know, who's the favorite and they would never give an answer, which is the best answer. Exactly. That's funny. That also reminds me of one of my favorite comic strips that I'll go ahead and put that in chat for those of you that have seen it. You'll remember that the standards XKCD comic where it starts off saying there are 14 competing standards and then somebody is just like, we'll come up with one unite them all. There are 15 competing standards and it's just kind of illustrating how difficult it can be to kind of get everyone's buy in and kind of make things happen on that front. So again, you know, the whole it depends trade offs makes it difficult to kind of come to agreements on that front. And thank you for working on such a cool tool to help bring all these things together because like you said, it's might be easier to go kind of like with the CIS standards or, you know, whatever it might be. But when it comes to actually getting an understanding of what your entire security perspective looks like, it's helpful to have all of these to take a look at and to benchmark your clusters against. So, thank you so much. Exactly. And again, we are very open to feedback. And if you have anything, just, you know, drop it on, on, on GIT or on our Discord channel. We have a Discord channel. We're constantly watching it. We have a developer channel there, announcement channel there. So, you know, you can register to our Discord and be part of the community. And we, we are very focused on a tube scape and making it better for your use. So, you know, we created for you and we need you to be with us on that. Absolutely. And I'm honestly curious to hear any of your use cases or stories or kind of things that developed from that, too. So please feel free to share those. I'd be very curious to hear. Awesome. I have one more question, but we do have a few more minutes. So if there are any questions that you want to get answered, please feel free to throw those into the chat. Otherwise, we can get things kind of closed out and give you some time back. So drawing to the end, one of the questions that I always like to ask is, what's the best way to get involved with the project and contribute in meaningful ways, you know, whether it be documentation or if there's a certain, if there's a certain perspective or set of features that you're trying to add, what are some of the ways that people can most beneficially get involved for the project? So we, if you look at our GIT, there are some issues that we mark as we're asking people to help us and contribute codes to. So, you know, we constantly watch our GIT and we are very, very active on that. So I think this is the best way for someone to start the discussion with us. Of course, if needed, we can move to Slack, Discord, emails, old telephone calls. Do you support carrier pigeons or anything like that? That's usually what I prefer. I don't think that pigeons can fly that far, but did pigeons fly over the Atlantic Ocean? I don't know. But anyway, so we can definitely jump on any medium in the Soviet Union. We do appreciate help. So, yeah. But let's start with GIT. I think that this is the channel that we observe the most. And of course, the Discord channel is a very, very good medium as well. Fantastic. Excellent. Well, thank you so much. I did see one more question come in. Will there be support for CIS? Yes. CIS support will come toward the beginning of 2022. I don't know if it will be January or March, but it will be somewhere in between, and it will be supported as well. Awesome. We had another comment that said support for RFC 2549 is important, but I'll definitely have to read up on that one. I'm not sure that went off hand. I'm writing it to myself. Absolutely. I'll go ahead and put that in our chat. Oh, IP over Avian Carrier. Yes, absolutely. Sorry, it didn't make the correlation. That's funny. Yeah, with talking about data transfer and everything on that front, there's a, I worked for a previous company where we did talk a lot about data transfer and the rate at which it would be faster to load it up either on a USB device or a big hard drive and actually drive it, ship it, fly it somewhere faster than being able to transmit it over the wire. So carrier pigeons are important. Thank you so much on that. Awesome. Well, I think those are all the questions that we've had. Thank you so much, Amir. Thank you everyone for joining the latest episode of Cloud Native Live. It was great to hear from Amir about Cubescape and different ways that you can test your Kubernetes deployments. We also really loved the interaction and questions that we got from everyone in the audience. And don't forget that we bring you the latest cloud native code every Wednesday at 11 a.m. Eastern time. Next week, we'll have Karen Chu and Kent Rancourt presenting. So what's new in Brigade 2? Thank you so much for joining us today. We'll see you next week. And with that, Amir, any parting words or things that you'd like to let the fans know? Thank you very much and please use Cubescape. Awesome. Well, with that, thank you so much, everyone. We hope you have a wonderful rest of your day. And we'll see you later. Have a good one.