 Well, hello, everyone. What are some examples of two things that go together well? I'll start, peanut butter and jelly. Could I get a little audience participation here? Examples of two things that go together well? Cookies and milk? Salt and chocolate. Salt and chocolate? One more, what's one more example? Bacon and eggs, excellent. These are all excellent examples. How about Kubernetes and security? My name is Al Aduberry. I know you're all so surprised. My, yes. My name is Al Aduberry. I have the pleasure of talking about SIG Security, what we've been up to with my colleagues, Pushkar, Savitha and Ray today. So a little bit about what we do in SIG Security as a whole. So SIG Security is a horizontal non-coding SIG that drives security initiatives for the entire Kubernetes project. So some examples are checking in with other parts of the project, the other SIGs making sure that the core components of Kubernetes are secure, doing some vulnerability management in addition to the four sub projects that you'll be hearing about today. So what really makes us stand out as a sort of security organization is really our approach in that it is very open and community driven. I know I've definitely experienced that some of the organizations I've worked at, the security organization being sort of a policy enforcer and being a little bit regimented and with our open community based approach, what really comes across is that what gets done and what we focus on is really based on who comes to the meetings, what are they excited about, what interests them and what do they think is important about driving security forward in the project. And that really speaks to our four focus points which are four sub projects, which are self assessments, which is the sub project that I lead, docs, which is Savitha, tools, which is Pushkar and, let's see, third party audit, which we'll be talking about today, which is driven by Ray. Now if you wanna come and join the good trouble that we make, hop into our Slack channel, SIG Security. Also you can join our Google group, which will add you to all of the meetings including the bi-weekly SIG Security meeting and all of the sub project meetings as well. So first a little bit about self assessments, the sub project that I lead, there should be a picture here. There we go. So this is the lightweight threat modeling sub project in Kubernetes. So our output is lightweight threat models, totally shocking I know, but the outcome that we're really trying to drive through those outputs is not only making Kubernetes itself more secure from a coding perspective, but also to build the security community and the security muscle in the Kubernetes project. So we do that by inviting really as many people who are interested as possible in these lightweight threat modeling exercises so that by the end of the exercise, we've now left several people armed with the knowledge of how to conduct a lightweight threat model which is a pretty awesome thing. So what have we been up to? Well currently we are focusing on the threat model for the vSphere CSI driver. I do happen to work at VMware, that's a coincidence. So today we have completed all of our data flow diagramming. So we've got all of our sequence diagrams and component diagrams in order. And now we're actually poised to begin the stride model or the threat modeling part of the process. So that is where we are at today. Now, just a really quick high level overview of the process that we use. So really it's about starting with the people. Once we have figured out a piece of the project that we want to threat model, getting the team together. So like I said, this doesn't mean that it's just experts who are involved. We wanna take beginners with us on the journey so that they can then leave the exercise with the knowledge and go forth and threat model other parts of Kubernetes. Then it's about mapping the attack surface. So putting in the time to draw the component and the data sequence diagram so that we can see how everything works. Then we conduct the threat model. As we do that, we of course write up our findings in a pull request. That pull request goes through a review process to make sure we didn't miss anything. And then we merge it and then it's done. But wait, it's not because a threat model is a living artifact that should be continuously reviewed as that piece of the project evolves. So how to get involved? You can hop into either one or both of the Slack channels either for the self-assessment sub-project itself or for the vSphere CSI driver. You can also welcome to shoot me a DM in Slack as well. And you can also, this is actually a call for help. If you have any threat modeling experience and you want to help with the vSphere CSI driver threat model, we would love for you to participate. So please jump into one of the Slack channels, shoot me a DM, we would love to have you. And yeah, we are always looking for targets for our next threat model. I love a good roadmap. I am a product manager. So if you have ideas for other parts of the project that could use this, we would love to hear from you. And now I am gonna hand it to Pushkar to talk about tooling. All right, can everyone hear me? Cool. So thank you, Ala. We are a sub-project called Tooling where we make Kubernetes more secure by writing code. There are two tracks that we currently work on. One is scanning for vulnerabilities in Kubernetes. So today we have been scanning Kubernetes, Kubernetes for Go model vulnerabilities and all the release images for almost two plus years. The scan jobs run every six hours and we continue to triage anything that seems important to fix with the rest of the community. We can't do anything alone like any security people can. So we work a lot with a lot of other six and they are the co-maintainers for all the code that we have written. So for vulnerability scanning, we have CIG architecture, we have CIG release, we have SRC, which is a security response committee which we work really closely with. And then we have the second track which is something more visible for end users which is the auto-refreshing official CVE feed. So anyone has heard of that, used it as a CVE feed in Kubernetes official documents? All right, cool. So now you know what it is. It is essentially a webpage with a JSON feed and an RSS feed for all the CVEs that Kubernetes SRC will publish and typically they are all the fixed CVEs. You can essentially use a Slack bot to subscribe to that feed and anytime when your CVE comes up, you'll get a message for it. It was a alpha feature back in 125 and now it became a beta feature from 127. Currently there is a lot of discussions going on in the background. We've got a lot of feedback in terms of how we can improve it. We will soon be publishing a lot more concrete information about what we want to do and at that time you all can come and help all of us in SIG Security Tooling Channel. We are always looking for people who can help us out. The last thing I would say is we also do learning sessions every other meeting and we meet every other week and anyone of us who is somewhat working on anything related to Kubernetes security want to present something that they've been working on whether it's a project which is open source and does something with Kubernetes security or something else. You're free to join us. There is a GitHub issue that you can use to add yourself for one of the upcoming meetings. With that, I'll pass it on to Savita. Thank you Pushkar. My name is Savita Raghunathan and I am the lead for the SIG Security Documentation Subproject. And in this subproject, what we do is, oh, see, sorry about that. In this subproject, what we do is we create security awareness through documentation. What that means is that we are constantly updating all the concepts related to security, creating new tutorials, updating all the examples so that the folks who are consuming Kubernetes can consume it securely. And we do that along with embodying the principles of SIG security that I mean that we like to share and learn and we also are very welcoming SIG. So if you're interested, please stop by one of our meetings. And I want to quickly highlight the current project. One of the long-term projects that we have been working on is the Kubernetes Hardening Guide and I want to give a shout out to Rory McQueen and Kailin Edwards for reviving this specific issue because it's been open for about one, one and a half years. They revived it, they divided the topic into, divided the issue into multiple topics so that the contributors can pick it up and not feel overwhelmed. And we already have our first topic that is merged and that is authentication mechanisms and we have another contributor who is working on RBAC and there is a PR app. It's up for reviews so if you're interested you can check it out. And we are also looking for contributors on this particular issue and the issue's hyperlink. This slide's gonna be available, it's already available on the sketch so if you click on it and if you like any topic please feel free to just express interest in. I want to mention that if you express interest that doesn't mean that you have to work on it alone by yourself. That means that we can find more collaborators and contributors and come together as a group and we can work on it together. And moving on, so we do have some upcoming plans and I know that makes Aula happy because I have future issues that's been listed forever. And one of the things is that RBAC guide and I really want to have a good tutorial for communities RBAC and add more concepts on that particular topic. And we already have a security checklist available for the administrators and we would like to create one more that is focused towards application development. And if any of this topic is very interesting to you and you want to participate, please stop by our meeting, it's monthly on Thursdays at 2 p.m. Eastern and we are also a project that collaborates synchronously, so stop by this Slack channel, start a thread if you're interested in any topic and if you really want to improve anything on community security on the website, feel free to create an issue, give a shout out in the channel and someone will actually partner with you and help you champion that effort. That's it for me, so next I present to you Ray Lahano, who's gonna present about the audit. Hello folks, my name is Ray Lahano, professionally I work with SUSE, but I like to contribute to Kubernetes. So I lead the external audit sub-project, a little bit of history, it actually started as a working group which actually published the first audit, we'll talk about that in a little bit. Like the other sub-projects and security in general, our goal is to make Kubernetes more secure. But with the external sub-projects, we do that by coordinating regular security audits. And some of the tasks we do, we help to find a scope, we like the community to help as well, we want community input to help define the scope, we release the RFP, publishing the RFP is also community-driven as well, so we're open for reviews. Then we review vendor submissions and we coordinate with this NCF, the Linux Foundation and the Kubernetes Security Release Committee as well. So that's us in general. A little bit of history, the 2019 security audit, sorry, was the first security audit and that was published in 2019, and that was done by a joint effort by Trail of Bits and the Trades. That was based on Kubernetes 1.13.4 and we're on 1.28, so Kubernetes moves fast. Last year with efforts, it would help with a number of six security contributors, we actually released a blog post that was a bit of a retrospective, like what happened since that 2019 security audit? Where were at? What have you done? And what do we need to do? It turns out we actually need to do several, some stuff still, out of that 2019 security audit, there was 37 findings that were found, issues were created, 21 are fixed and five are deemed that needs a Kubernetes enhancement proposal. That means that it's more than just a bug fix or a patch that there's actually a change to the features of Kubernetes. So this also identifies that we also need help. If you are interested in contributing to Kubernetes and to security, we have lots of issues that needs help solving. So out of the 2019 security audit, we have 21 that still needs to be fixed. As of April 19th of this year, we published the latest security audit based on 1.24 and that was conducted by NCC Group. Out of that security audit, it was based on 1.24. The scope was the main components of Kubernetes, the API server, the scheduler, various controllers, the use of XCD, like Cubelets, also a secret store CSI driver. There are many components of Kubernetes. So we are, there is a gap and I'll address this later on as well that we want to audit more components of Kubernetes as well. Sorry, I think my images are not showing up for something. Okay. So we have an umbrella issue open for those 19 findings for that was released in this past April. Link is on the deck as well. So there's 19 findings. I'll go over some of the more key ones in a little bit here. So just a high level breakdown of the audits. I do want to focus that there's no critical and there's no high findings that were found. So we're going to focus on the medium findings for the first parts of this talk. Depending on time we'll go into more of the findings here. Based on the category, this access control, authentication, identifying users, configuration, security configuration, cryptography, data validation. And also the findings that were discovered out of the components that were part of the scope, only findings were found in Cube API server, which is also a big component of Kubernetes and two in Cubelets and four from the architecture review. So like I said before, that we're going to focus on the six issues first out of those fours from the Cube API server, one's with Cubelet and one's from the architecture review. So the first one is additive access control. This is the one we all know. It's actually stated in the Kubernetes documentation that authorization in Kubernetes is mostly handled by RBAC. I guess there's some folks that use ABAC and permissions are purely additive and there are no deny rolls. We state this in documentation, there's no deny roll. So this was cited as well. And the recommendation is to enhance RBAC to support explicit deny rolls. Next finding is that under very specific conditions, and I have the conditions listed here, an authenticated user can have their privileges escalated to cluster admin. And so those conditions are when the client CA file equals to the request header client CA file and request header allowed names is not used is to specify the certificate common name that's permitted for that request header off. So the recommendation is to reject any conditions when that specific condition is actually valid. So this title, common certificate authority possible for client CA and request header CA. So this is with the CUBE API server. The third medium finding is actually FISC fixed and this is about path traversal namespace specifier. So a user when they do request, they could actually use directory traversal sequence dot dots as a namespace in their request to the Kubernetes API. So you could find the full path of objects that they should not have access to. And so the recommendation which was taken was to add a check on the namespace identifier and API request. Next one is the redirection of API server traffic to Cubelets. So there's a feature in the Kubernetes API servers that has a proxy feature and we're gonna have two findings about this proxy feature. So the first one is that when a user can actually get cluster admin permissions if they have specific permissions or already if they have get on node proxy or they can either patch or on node status and also create on node permissions. So they can manipulate the API server to authenticate itself as the API server. So the Kubernetes API server proxy is a feature in case the user does not have any network access to the workload or to the node. So you could actually proxy your request to the workload or to the node. Because of that, there are some findings that came about with this, the Kubernetes API server proxy feature. So the recommendation here is to implement additional defense measures to prevent the proxy requests being sent to the API server swap port. Next one is API server proxy disables to TLS cert validation. This again is with the Kubernetes API with the proxy feature as well. And the proxy server proxy or the Kubernetes API server sends TLS connection or has TLS connections to the pod services and nodes and someone who can intercept those and can read or modify. So this recommendation here is to implement and implement TLS cert validation for the pod service and node proxies. I think this might be the last one here. And this came about and was also published by AquaSec and also was referred to as also from NCC as well. This is privilege escalation via node proxy permission. So I have the link for the blog from AquaSec as well. So a user with certain permissions this time is get and create permissions on node proxy and the cluster as it has full permission on the Kubernetes API. So when this proxy request is made to the Kubelet or a node, the API server's own client cert is actually used to authenticate, not the user. So you do get full access. I do wanna make a note. Those are the six findings. There's more and I'll reference those a little bit but I do wanna make a note as well that we do have an audit roadmap since we know Kubernetes is very big and we have the main components but there are more than the main components. This is just a screenshot, small bits of what was proposed as the audit roadmap. So things like even worker nodes on running on windows, cluster API, gateway API. There's so many more components of Kubernetes that should be audited. With that, the subproject's goal is to conduct regular audits as well. So these regular audits since the core features are very big, we might highlight more the smaller components like the cluster autoscaler and several components together and do the main components at a specific cadence. This roadmap is on GitHub and is open for any pull requests as well. We want this to be community driven, not just the SIG but everyone needs to think about security. So please, everyone who has an idea of any components of Kubernetes that should be audited, please make a pull request. And we have an issue already opened, just opened this for the next audit. So this is just a tracking issue. So this is what we've done in the last audit and a little bit more as well. So this is just a track and an umbrella issue, just track the activities that's required for the next to query audit to take place. I do wanna make a note, as you might be thinking that the first audit was 2019 and the last one was published in 2023. So we did have a goal at 2020, the pandemic happened and we released the RFP in 2021 and the audit was published in 2023. So one of the goals, I hope to have a faster release or have that process done a little bit faster. So that's a lot of coordination efforts. It's a lot of, if you don't know anything about security, if you don't know, if you don't code, that's totally fine. We're open to anyone that could help with just coordination efforts. So with that, the issue is open and the link is on GitHub as well. So just going over, we helped to find the audit scope. Like I said, that's community driven. We want community to help out. Creating the RFP, we could help create a base of the RFP, but we want the community also to have their input in. After that, we get proposals in and vendor assessment. Then we go through the actual audit and finding subject matter experts. Like I said, like I said, you don't have to know Kubernetes very intimately. We all we had need to find is the subject matter expert behind those certain components that could talk to the vendor for the audit. After a few more steps here, we do coordinate with the security release committee and also the Linux foundation and the CNCF as well. Then we publish the findings and we start over again. So this is an ongoing cycle. Even if you don't have time for now and you can always hop out in the feature and also help jump in whenever you want to. So a little bit more as well. I'm going to go over to some of the more findings that we didn't highlight in the medium findings, some of the low ones. There's also on the umbrella issue as well, there's status of a fix or it won't fix. So the other one that we another went to that is a finding is the multiple concerns with network policy. This was deemed as won't fix because this was as a result of communication with contributors. This is because you need to implement a CNI plugin to implement the Kubernetes network model. And some CNI plugins don't enforce network policies. So this is not exactly. So the print recommendation is if you want to use network policies, of course, use a CNI. And that uses network policies. Next was lack of cohesion between core access control mechanisms. That's because we, in Kubernetes, we have RBAC to authorize and authenticate users. And we have admission control for resources. So there's needs, there's not exactly, they don't really, yes they do and they don't, but there's lack of cohesion between them. Next one is the weaknesses and pod security standards in restrictor profile. And the restrictor profile of pod security standards does not enforce group ID to the zero. All right, so the next one in terms of the QBAPI server, just going over the low ones. As you can see, there are several, that are actually next slide, we'll see that there are several low risk issues that were fixed as well as just going over some of these, the authentication source, not showing in the audit logs. So it kind of manipulates some things that it might be a security concern, so it's really as low. Do you like the empty-deer volumes that I support mount options? So this is when, this side is low, and this is when empty-deer, you could run, you could execute programs from empty-deer, and so there's no option to do it, no exact on empty-deer. So that was a recommendation for the empty-deer, while I just do not support mount options. Last one is the loopback token, usable externally. So this is the, I think this is QBATM and NITs, the loopback token, actually I'm gonna comment that, sorry about that. The QB, this is the API server, creates an ephemeral loopback token, add a NIT, and this could be used to attain system master's privileges. A few more low ones for a QBAPI, that the inaccurate, exported URI header, actually let me go to log it, we incorrect bootstrap tokens, so I'll try to recall the first one. So bootstrap tokens, and this was fixed, if you, someone used incorrect bootstrap token, you could, and then it would log the actual value of that in the logs. Do a time check here. A few more that were fixed recently as a non-constant time comparison of service accounts, so the comparison of service accounts was not an in-constant time, so theoretically it could kinda, it's always guessed the value of the service account. So I do want to leave some time for questions, so I'm actually going to go into the last one here. The privilege escalation via nodes proxy permission that was already reviewed as a media fix and we do need help to fix that, so if you are interested in contributing to security in a very impactful way, that is one way to help out. So I want to save some room here, about six minutes for any questions. I've got two questions. One is that you mentioned the Slack channels, but what is the workspace? It's a Kubernetes Slack workspace. And then there's the hashtag, SIG Security, SIG-Security, and the sub-projects have their own Slack channels, SIG-Security, DASH External Audits, SIG-Security, DASH Cluster, we like tooling, you know, security docs as well. The main work space is Kubernetes, right? Was that a question? The main workspace, like where? Yeah, yeah, the main Kubernetes workspace. All right, cool. Second question is, when you say the audit, what is your framework actually? What are you actually using to audit against? And to be honest with you, I'm coming from a security background, so my thinking is that what you're presenting is more of a, I guess, a code scan than the full audit, because there's more components to the audit than just the code that you need to have, including multiple controls, including how you, what the access is, how you maintain the code, where, you know, who has access to it, who can commit to it, and so forth. So that is kind of like, if you can address that. Yeah, I do agree that most of the, I do agree that this audit was a lot of code scanning that they did, and we also, the SCCF law did fuzzing as well from another vendor that is difficult to do an audit that's very comprehensive with a large size of a project, and this is something that we were trying to address as well, so I do agree that we need to expand the framework of the audit, so. Any questions? Hi, I'm short. If people wanted to get involved in SIG Security, is there any kind of like skill requirement or experience requirement to get involved? Like who can get involved in SIG Security and how? That's an excellent question. Anyone, anyone who is enthusiastic about Kubernetes and security is welcome to join SIG Security. There is no specific skill set required. There's no specific knowledge required. All that is required to participate in SIG Security is that you are somehow involved in Kubernetes or you want to be, and that you're interested in security, and we will embrace you and help you connect to what interests you to work on to make an impact in Kubernetes in respect of security, so yeah, come on in, join us. The water is warm. Any more questions? Hey, how's it going? So if we are contributing, what is the level of effort like an average, like how many hours are we, or is it just like based on whenever we have some time? So this is totally up to you as well. If you want to contribute five minutes, 10 minutes a week, you know, if you maybe do security docs reviews, that's totally welcome. Or if you want to drive a cap forward that fixes one of the findings, that might take more time in a week than you're welcome to do that too. So any amount of time is welcome. And do you guys have like weekly or bi-weekly meetings? Yeah, we do have bi-weekly meetings on Thursdays, and you could even just join the meetings. If you wanted to join the meetings, that's a great way to be part of SIG Security. How can we go about getting invited to that? So there is, in the GitHub, there's the Kubernetes repo, there's community slash, in the community repo, there's all the SIGs. And you just click on SIG Security, and we have the link to the Zoom when the Zoom is at, or the Zoom link, and also link to the agenda as well. Okay, thank you. We will probably edit these slides and upload it and with that information. So if you go and about like now, by the end of the day, just to give us some room, we will have all the information there. I think that just to editorialize on that a tiny bit, one thing I think that you've really been amazing at, in docs, is breaking down the tasks and like the increments of work that people can do to take on to rapidly contribute. It's like, oh, we literally do have five to 10 minute tasks that you can take on and make an impact in SIG Security. So that's definitely something that I take inspiration from as well, so Savitha and what is it, Ian and Kailin, Aurora rather, have been really great at. So that's part of what we try to do to make security welcoming and digestible for people is to say, hey, like, here's a five minute thing that you can go and do and make an impact. And you can triage issues, that's another thing, or you can just come to the meeting and take notes. We appreciate people who take notes and we really need note-takers. It's hard to run the meetings and take notes as well, I struggle a lot. I cannot type and talk. I make typos, I'm very cautious about it. So one of the ways to contribute is to show up the meeting and to take notes that help us create a slack thread, stuff like that. We have many ways to contribute. You don't have to know security and this is like, if you know, it's amazing. And we also encourage that if you are, please come to our sake with the mentality of learning and sharing. We really need that and we really encourage that. So I think we're just about literally to, okay, so we're now officially out of time, but I do just quickly wanna shout out our amazing chairs who are sitting right here, Ian and Tabitha. They have done such an amazing job in creating security community and having a sub-project leads connect to what makes us excited. So thank you. Thank you.