 Hello everybody and welcome to another tech and talk Really pleased to have a good friend Liz rice with me who's on the aqua team these days And she's been doing a lot of work around Kubernetes and she's got this Exciting new and maybe a couple of another exciting news. There may be two things combined in this talk Coupe bench that I'm going to get her to do a deep dive and explain what that is and the reasons for developing it What we try and do with these tech talks is have conversations, so we're going to let Liz do her presentation And we'll have live Q&A at the end of it if you have questions during her talking or demoing Ask them in the chat But the whole goal here is really to just look at some new innovations around Kubernetes cloud native stuff and Things that I'm interested in and you are too hopefully so without any further ado Liz take it away Thank you very much Diane and Hi everyone out there in the world Yeah, so when Diane asked me to join the the tech and talk today, we had just announced cube bench But I'm going to take a couple of minutes at the end to talk about a new and exciting thing We just announced today called manifesto as well So, yeah, loads of exciting open source things going on at Aqua security at the moment So, let me talk a little bit about cube bench So I thought it's start by just setting the scene a little bit with If I can work my slides anyway There we go, right and just a little bit of background really about how Security fits into the container life cycle Aqua we you know our product is About securing containerized deployments and what we've done with cube bench is a nice kind of open source compliment to the You know the enterprise product that we have So container security really can apply at all sorts of different stages through the container life cycle really quickly, you know When you've built the image you want to Know that that image came from Well, maybe it was built on a base image that doesn't contain vulnerability So you want to keep scanning your images for for vulnerabilities You want to know that the base image came from a reputable source. You don't want somebody being able to kind of inject unwanted exploit code into container images that you then build from and store in your register So that's kind of a first step of image assurance then we get on to something we call environment hardening and That is really where cube bench fits in So this is around saying is the host that we're running on secure Is the container engine that we're running on secure? There's these benchmarks that I'll come to you in a moment that help us test for those things We also have Parts in our product that will help container users ensure that their secrets are managed securely I can talk, you know for a whole other half an hour or something about Managing secrets and containers But suffice to say that, you know, that that's an important Aspect to managing your containers securely and making sure that you're not passing, you know keys to your database or other important elements Around to your containers without some kind of care and retention and Then we get into the things that I think really quite interesting about security in the containerized world which is looking at Restricting your the privileges of your containers to the least privileges that they need to run So that can be making sure that a container doesn't run as route It could be Making sure that you're using a security profile like a set comp profile or an app on the profile Appropriate to your container images and then finally there's network controls making sure that traffic only can flow between Between pairs of containers or between containers in the outside world as expected so We can see through those kind of five stages that security plays a part all the way through the container lifecycle right from Building the image all the way through to it's running and it's communicating with with other containers and with with the outside world so there's a whole load of things that and Could go wrong a whole lot of opportunities for bad actors to kind of try and exploit and I think that brings us on to the the center for internet security So that's what CIS stands for the Center for Internet Security and they have Lots of publications around best practices for all sorts of kind of parts of your IT infrastructure and A couple of months ago they published a Kubernetes benchmark There's also a Docker benchmark and we actually have an implementation of that within our Pay-for-product, but we decided that for Kubernetes we build it and open source and put it out in the open and contribute it to to the community so the Benchmark test itself. I'll just Show that document so this is actually the 1.1 version. It's the the second version and it's been updated to include Security best practices for Kubernetes is 1.7 and If I kind of scroll to the contents page If I go here and Hopefully you can see there's 252 pages in this document. It's not a small document and It's got all sorts of sensible best practices around How your host should be configured and how your Kubernetes settings should be set up And a lot of these tests are easily automated that they're written in such a way that it's easy to automate them So I'll just bring up an example Hopefully that's big enough for people to read great So The the benchmark consists of a number of descriptions of the problem. What the reasoning is behind this potential problem and a way of checking whether or not you've Got that particular problem on your on your system in this example, it's really saying Run PS to to find the QBAT API server executable and check whether or not it's got the allow privilege allow privileged argument set and tests like that are you know really very easy to Automate and that's what we've done with the with the benchmark and you also get some remediation information You know what to do if you're If your benchmark fence essentially And it'll discuss the impact and obviously every Every deployment is unique and there may be reasons why you choose to Ignore some of these recommendations, but as a base point, it's a really good place to start You know some some expert people have gone to the trouble of thinking through lots of potential security flaws So It's a good place to start and by automating the tests. We're hoping that this makes it really easy for people to to run them and keep their deployments secure and There are different tests for different types of node. So There's a section for master notes that we can see here. There's also a section for regular nodes and for federated nodes and Perhaps the next thing I should do is actually show show it kind of inaction So well before I run it, I'll just take a look at the the config files So for example, if I look at the master yeah, but the tests are all configured in yaml format And we can see, you know There's a pretty clear correlation between this yaml and that text description that we were just looking at And the reason why we've chosen to do it in yaml is that it's much easier to add new tests keep them up to date as the benchmark evolves and what we've been talking just before the recall about Documentation changing quickly in Kubernetes, you know, the whole world of Kubernetes is evolving really quickly So there is no doubt in my mind that these benchmark tests will need to evolve to to keep up with that Yeah, but that you know, it came out pretty quick after 1.7 was released, too So whoever is at CIS is actually doing a pretty quick turnaround on getting these updates. That was impressive. Actually, wasn't it? Yeah, I I've joined him with that conversation I you know, my contribution is small compared to some of the other folks in there who've been authoring these These tests, but yeah, they had that out really in a matter of a few weeks after the 1.7 release So it's really impressive and and and good work. And I think there's more, you know You know more help would always be welcome there. I think This actually came up on a Sigorth call, I think it was Where you know, there's some Pretty intricate security settings in Kubernetes, which I am by no means the expert in and Seeing how those things interact with each other is kind of expertise that I'm sure that the CIS would like to kind of, you know, leverage that expertise in the benchmarks going forward But yeah, they're certainly very key. You know, I think they want these benchmarks to be Appropriate living documents that people, you know find authoritative So, yeah, really really good folks working on that. So That's kind of Okay, and I'll just show it's just a YAML file for each of the Different types of tests of the master the know the federated and there's also An overall conflict file. Now, this is Something that we've recently Added, you know, Q benches also, you know, moving to keep up to date and There are different installation Tools, you know, cops, Kube, Adam Things that might be pronounced in ways that I don't know how you're supposed to pronounce And Openshift as well, I'll mention open shift shortly But these tend to have different names of the executable, you know, there's hyper cube They may put the conflict files in different locations and have different names of those conflict files so We've created some what we call installations which give us some default Settings for these different installation tools so for example, here's one for cops which, you know We hope is correct, but corrections very much welcome And then you'd run the appropriate what you'd select the appropriate installation And if you needed to you could modify these to make sure it matches your Your deployment It's actually one of the things I think is and You know quite tricky for the people writing the benchmark to kind of keep track of What the best practices should be given that there are so many different tools that actually result in slightly different ways of configuring communities So I finally I just wanted to get this one run, so we'll run the master tests and Takes very little time. So let's just scroll back to the beginning of the output and You can basically you can see that there's a number of warnings and then for each Test it either fails or passes or and there may be some of the warnings. Yes, there's some with warnings This is typical for CIS benchmarks in general It adds up to at the end we can see a number of passes fails and warnings and You might also want to automate this to kind of keep track of Whether things have changed So we support a JSON output for the same test results And you get the remediation so you can you know follow the instructions to try and improve the security on on You know on your own deployment. There's quite a few things as you can see on this particular machine that aren't set up perfectly according to the To the best practices So what sort of privileges do you need to actually run this command? And to run the command itself, you don't really need any special privileges at all. You need to be able to see the The output of PS and you need to be able to And Some of the files will check the permissions on Conflict files and check the ownership. So if you don't have permission to even see into those Directries, you wouldn't you wouldn't get a good result, but you don't need, you know root or anything like that A regular standard user should be able to run these tests So Let me see what else if I had in my slides there might be yes So the github it's all on on github both the code which is a go application and And the conflicts for the tests and We would love to hear from people, you know, if they have any issues if they Particularly with the configuration for different installation tools, you know, we're trying to try our best to kind of cover the different bases, but You know, we'd love to hear from people who are actually using it in the field for having issues Okay, I Was just gonna quickly mention this whole thing of Automating compliance. I actually stole this slide from from one of our marketing decks And it's kind of showing how In this case, it's actually the docker benchmark, but you use that so you can see it's quite small but so it adds up the the number of warnings and passes and info Totals for that benchmark so For kind of regulatory for compliance purposes you can keep track of well, you know How was this host set up what were the you know, was there some sudden change in our host configuration that meant that You know some machine or some set of machines was suddenly not As secure as it was before So the intention is to automate all these things And make it just really easy for people to to you know, to know what state their machines are in And I thought I would mention OpenShift Really briefly because it's obviously, you know I think in a lot of cases OpenShift have kind of gone above and beyond what Kubernetes does for a Lot of security aspects and there's a whole bunch of things here that I'm you know Not super familiar with so don't ask me any difficult questions about OpenShift specific security Suffice to say that you know, I know that it's a you know, a real Focus for for Red Hat and further for the OpenShift team So I'm sure there's a lot of really, you know, good work has gone in there So there might be an argument that says well, how much of these Kubernetes tests are really applicable So we actually just opened an issue in the last day or two So kind of raise that question because I think on the one hand, you know, a lot of things are in common and Some of the things like config file permissions and ownership And the defaults from OpenShift are going to be great, but there's no reason not to automate Just testing to make sure And I also noticed that some of the Well, a lot of the benchmarks tests refer to kind of parameters that you pass in on The you know when you're executing the binary thing in Kubernetes and I think those can be passed through Sort of transparently in OpenShift. So just checking for those is not a bad idea But you know whether there's What it'd be really good to hear from the OpenShift community about their thoughts for Yeah, but what would be great is in extension to your test specific Right, right. Yeah, if there's demand for that then that can be a really good approach. I think yeah, yeah And I think you're going to see over time more enterprise distributions and of Kubernetes that are, you know, like like OpenShift value ads Container platforms that have Kubernetes under the hood that are doing a whole lot of other things You know, I'm not just adding security but doing multi-tenancy and all of those kinds of things and OpenShift will eventually not be the only one Benchmarking will change a little bit and I think that's an interesting aspect to this whole thing too and Going beyond vanilla Kubernetes benchmarking Yeah Yeah, and it'd be interesting. I think based from the tools perspective, but also from the CIS perspective, you know, whether Whether that benchmark should be extended to cover OpenShift or have a separate one for OpenShift You know, I'm like, I really don't know what the right approach is there, but I'm sure that someone at Red Hat who has an opinion of that So we'll look forward to hearing from whoever that is out there at Red Hat Land and who tells me, you know what I should be saying and what I shouldn't be saying about this I think automation is the key to everything, not just security, but everything, CI, CD, All, you know, the whole upgrade, everything. So the more we can do that, the better and I come up I actually come out of Early on in my career, an IT audit background So for me, this is near and dear to my heart and being able to automate this stuff is great because it makes it much easier for The compliance officer in any enterprise to accept this new world of containers and microservices If they can see a report that says you pass the CIS, you know, CIS benchmarking, blah, blah, blah They want that piece of paper and the documentation and it to be a repeatable process that they can show at the end of every quarter or every upgrade or wherever So this kind of stuff is pretty damn important in my humble opinion Right, right. I couldn't agree more. I mean, I think that, you know, every time we can automate anything in security or anything else that's just kind of repeatable Automations means we're more likely to catch problems and that's got to be a good thing Right, so I think I have one more slide which has the link to where is it the link to that issue about OpenShift config in QtBench. So that's kind of a starting point for a discussion, maybe I think that was everything I was going to say about QtBench itself, but is it okay if I briefly mentioned my new list? If there are no questions yet, I think I've been asking them as we go Tell us about this new project Thank you. Yeah, yeah, so if You know, you might remember that before I joined aqua. I was working on a project called micro badger which looked at container metadata in particular looking at labels and the metadata that you could associate with a container at build time and it had always kind of You know, I'd always been aware that you know, well that only solves part of the problem There's lots of metadata that you want to associate with container images that changes after the image has been built So examples of this could be A really great example is the vulnerability scanning report. So, you know, if you scan that image, let's say every day Over time that report will change because even if the image itself doesn't change new vulnerabilities can be found So it's really nice to be able to associate You know that report the latest version of the report with that image And you know know exactly where they're put together or another example might be Keeping track of whether that image has been through Whatever approvals it needs before it needs to be before it can be deployed You know test status Going through some kind of security audit maybe you know And you can't You don't want to rebuild the image to keep track of this information because Well, it wouldn't be the same image anymore And so what we've done with manifesto is it's a prototype tool at the moment where you can Basically add metadata to Kind of alongside an existing image in the same registry. So we store it as data in the side of the registry And we just have a simple tool that lets you get put and list Well, you any arbitrary metadata and the joy of Story in the registry is well, there are two things one is it's existing infrastructure So, you know people have got their own on-premise registry great Can keep the metadata in the same place and not have to worry about An additional set of security concerns or access permissions and what have you The other thing that's really cool is And we haven't implemented this yet, but this is the the direction of travel Is to use notary to sign the metadata So you know that the metadata in the same way that you can know the images The correct image that you know Came from the place that you expected it to come from The same can be true for the metadata So that's really what it is and we you know literally released it this morning. So I hope you can see some feedback so far. There's been some some some good Some good comments and and I think I'm going to be talking about about it at the Moby six scanning meeting coming up shortly So, yeah, that's that's manifesto So you mentioned something notary in there. Can you explain what that is? Yeah, so notary is a component I'm pretty sure it's part of docker rather than part of the Moby project But I might be wrong No, no, I'm not sure about that actually But anyway, it came from docker and it's an implementation of something called the update framework to tuf or tough And it lets you So you have a piece of data whether that's an image or maybe it's some other software package I think that the tough spec came out of software packaging in general Or in our case in in the case of manifesto the data could be some metadata and you Basically take a hash of the the data and you sign it and you store it in notary So you can verify If you've got a piece of data in your hand You can check with notary that the hash matches and the signature is what you expected So it you know came from the person that you expected to to sign it and Yeah, so it's a really good way of proving provenance and it also has Kind of Making sure that it's the latest version So I think particularly in this context of software packages and updates If you request, you know, you you say to notary I want the The Well the notary information about this particular Package it will always give you back the latest version, not some old version So Yeah, really nice piece of technology and I think docker have done a great job in using that for image provenance Okay, well I'm gonna have to look up that I hadn't heard that one before that's what I was asking So so manifesto is like fresh off the pressure press And you're off and running with it. So if notary isn't an open source part of docker Is there a replacement for it or you know, it naturally is open source. I was just Pausing whether or not I couldn't remember whether it's moved into maybe or not Okay But it's open source either way So if you want people if people want to get a hold of you or to work to give feedback on cube bench or to To test it or or make a pull request against it or manifesto. What's the best there you go the best there you go Yeah, so those are the both projects are on github under aqua security Um, and I you know, I'm on twitter. That's a very good way of getting hold of me. I think um Yeah, I'm also liz at aqua sec.com. So More than happy to hear from anyone with Feedback or comments or questions or contributions or ideas All welcome So and you also mentioned I think a couple of sigs Is kube bench coming? Um, or have you done any presentations into any of the kubernetes sigs on the ship or is there Here's sig where you know a cube security sig that you're sitting in or something like that Yeah, there's there's a a a sig called sig orce Yes, which covers kind of all kinds of security aspects really And uh, yeah, I presented it presented kube bench Into that sig and also into the kubernetes community meeting Um And actually there was a question there about whether we'd like to see kube bench become part of kubernetes And the answer is absolutely yes, we would There's like an incubation process that we're just starting to look into So if anybody's listening to this and would love to be a sponsor or a champion for kube bench I would love to hear from you Well, that's that That's always that the the thing with kubernetes is that they're really trying good. They're actually kind of good about Um, not incubating too much and maybe pushing things out to the cncf to be incubated over there As side projects like prometheus and and other many many other things that cd and you know, the zillion other things are coming over there soon um So it's really interesting sort of um semantic question of what should be in kubernetes and what should be um in a cloud native Relation this this seems pretty um kubernetes specific so Yeah, but the whole concept of automating your benchmarks and using it it seems to me that this is something that the cis folks ought to be involved in too um the the You know besides writing these white papers that this kind of tooling I'm making it available for the different things that they do benchmarking on i'm wondering if we can't get some And especially whomever it was that turned out this 1.7 paper so fast um Get them involved in it as well Yeah, I think there's obviously some overlap between you know, there's a set of people who are working on the cis and you know working on kubernetes as well Yeah, there must be so we'll have to track those folks down and figure out where they're at because It almost seems like something that that person should you know or persons Should be involved in um in this as well And you know when you click to download the pdf of the benchmarking There should be another little link there to click and download kube bench and install it and run it That's a good idea, isn't it? Yeah Yeah, that kind of cross community collaboration It you know trying to do that that'll that'll get you a long way And um also probably help you with updating On each release too so and and doing that So i'll definitely reach out to um my open shift and red hat security people and And send this all away over to them and see see where they're at And if we can get some open shift origin community love in there as well Sounds perfect. Lovely. Yeah Fun stuff, but this is um these are the kinds of projects that really make kubernetes Production ready and make people happy about deploying them on their enterprise clouds or in their private clouds or On their public clouds, so I really appreciate the work you've done An aqua set continue continues to do In the security space and It's going to it's going to be an interesting each release every three months It's going to be interesting keeping up with it all so Yeah, the momentum is huge isn't it and it's really exciting So, um, are you coming to austin for the coup con that's coming up in december? Yeah, I'm not sure but I hope to be there. Yes So we'll be there I'll be there the day before doing an open shift commons gathering So we'll probably have a bunch of our open shift security people there as well Doing one of their lunch breakouts on um sig so If you This thing to this wants to come and talk about it or see it in action Or talk about other security issues on december 5th in austin texas Um open shift will be having another gathering All it's the day before Um coup con kicks off. So there'll be It'll be a good intro for all of the kubernetes and cloud native stuff that come Really well in Berlin so It was it was lots of fun for me It was like the best prep class you could take for going into coup con because it was like, oh, okay So if you don't know all the vocabulary come to the open shift thing Learn the vocabulary figure out who the thought leaders are so you could stop them through the next two days Your questions answered it was great. So I'm really pleased to do that. That's kind of how I coerced liz into doing this talk Was running into her after all that gathering in berlin. So Yeah I'm pretty good at this uh coercion thing. So The one question I always ask everybody on these tech and talks is You know, we all have different um favorite things out there and um cloud land and technology land Who would you like to hear from on an upcoming speech or someone you think that the community at large Would be not just specifically to open shift even the red hat Yeah, so it's sort of a couple of people. Um for that one is my Former colleague and very very good friend. I'm curry Say she has been doing some uh some research into kind of real life cloud native Usage, you know talking to a bunch of enterprises. So, uh, um, I'm sure she'd have some interesting Things to say about that research Yeah And the other person I was thinking of was um garris rushgrove from puppet Yes, yeah, because he's um always doing really interesting things around particularly this whole metadata thing which I kind of find interesting and Yeah, and and he's always got some really good experience and good opinions to to share Yeah, and and is a wonderful speaker. It always has, you know, great sort of thought leadership And it's very inspiring and garris always tries to coerce me into doing something It's like, yeah, okay or going to some puppet comp event and presenting on something Um, she's always fun too. So you have two good suggestions. So thank you for that thing. Um, Thank you for your time today. This should be up on the open shift blog And on youtube in a day or so depending on how fast the internet gods work And uh the editing webmaster person who does all of my editing from the lj bank. She was just awesome Um, so I really want to thank her for that work and thank you for coming today And hopefully we'll get to see you sooner than in december somewhere out there on one of the upcoming multitude of conferences and events that we have to go to but Thanks. This has been my pleasure. It's been great. Really. Thank you very much for having me All right. Take care