 My name's John. And I'm John. There we go. And this is Hacking the Kubernetes software supply chain with .zip domains. So in this talk, kind of what we want to present is a little bit of background about .zip domains, the trouble they may cause, the role of social engineering in that, a little bit of a demo, and then the defense against something like that as well. So before we get too much further, let's go into a little bit of background on what .zip domains are, how I somehow found myself with the kubernetes.zip website, and I get into a demo. So .zip domains, or top-level domains, became available in May of this year. And generally, to kind of the outcry of the internet, people generally just said, this is probably just bad for the safety of the wider internet. Having a top-level domain like johnscoolwebsite.zip, or johnscoolwebsite.zip, or something more nefarious, like myfiles.zip, or kubernetes.zip, just gives you kind of the air of, oh, is it a website? Is it a zip file with a bunch of stuff in it? Who knows at this point? .zip top-level domains are out there. So I got it in my head that, oh, who has kubernetes.zip? Somebody picked it up already, and this was months later at this point, and I was able to get it for like $12. The first thing I did was immediately redirected to kubernetes.io. I talked with Dimms a little bit, and we just made sure that, okay, we don't want this out in the wild. I can only imagine the kind of things people would do with some nefarious actor saying, hey, can you help me figure out what's wrong with my cluster? Here are my logs, kubernetes.zip. Can you help me? Yeah, so .zip top-level domains, websites, probably not a good idea for just the general safety of the internet. But let's break down how a malicious actor would use social engineering, a .zip top-level domain, and some tricks with a malicious URL to make this a much, much worse kind of attack. So a quick show of hands. Who thinks that this URL goes to google.com? Nobody, wow. Who thinks it goes to bing.com? A few people, yes, you would be correct. So let's break this down real quick. The first part, HTTPS, that's the protocol. The next part, before the at sign, is essentially a login, like a username login to this website. And then bing.com is where that would be directed. Let's do this again, but with a more malicious looking URL, again, we have HTTPS, github.com, Kubernetes, Kubernetes, archives, refs, head at kubernetes.zip. Same thing here, this actually goes to my website kubernetes.zip. We have the HTTPS protocol, and then the orange part is the username login at kubernetes.zip. The really tricky part here, you think that those paths would be escaped in the URL, but those are actually Unicode 2215 slashes, not the ASCII slashes, which gets interpreted as one username login. So that's really like the social engineering trick here, is that that whole orange bit gets interpreted as one username login and sends you to kubernetes.zip. There it is, very bad, very scary. Let's jump into a quick demo here. So what I have here is just my terminal up with some pre-baked stuff. I don't wanna have to download a bunch of this over the conference Wi-Fi, but this is that Google Bing URL. Let's click that real quick, and that's just gonna take us to bing.com. It takes us right there. Now, some browsers, and I've had a very hit or miss with this, we just saw it went straight to Bing, but sometimes a browser, maybe depending on caching, will actually say, hey, this URL may be trying to trick you. In Firefox, it'll literally say, this may be trying to trick you to take you to a different website. Sometimes with caching, sometimes not, it might take you just directly there. So that took us right to bing.com. If we look at the real URL, this is what the real URL for the master zip on top of GitHub looks like. And if I click that, if I click that, there we go. That's gonna start downloading a zip file. I'm just gonna cancel that real quick because we don't wanna have to do that over wifi. But what that zip file that I just started to download is, was if I go over here to code on the Kubernetes, Kubernetes repo, and then I come here to download zip, that's that same URL. So I could be some cloud provider. I could be somebody who needs to distribute Kubernetes and get the code, build it, send it off, run it, whatever, and maybe this is the way I do it. I'm gonna download the zip file from the tip of the master branch on Kubernetes. So what I thought that the most likely attack using a Kubernetes zip file top level domain would be, is to actually have just a very simple redirect to an S3 bucket of some malicious code that looks almost identical to what you would get from the top level zip file here on Kubernetes Kubernetes. So let's look at that. If I now cat the other one, we notice here, again, that at sign, and in my terminal I can almost not tell with this font that these are Unicode slashes and not the ASCII slashes. So if I click that, ah, there it is. That is the warning that says this may be an attempt to trick you. But you know, we're being socially engineered so like there's big problems. We gotta do stuff now, so I'm just gonna continue. And it does something almost similar. It's downloading that zip again. Let's cancel that out. And that's a very similar experience to just getting the zip off of Kubernetes Kubernetes. So now let's say that we're gonna continue down this path, this kind of thought of experiments of having the code bundled. And we're gonna go to some pre-baked stuff that I have right in here. And if we pop in here, you know, this, we've seen this, this is Kubernetes Kubernetes. It's got a bunch of go code. This is the stuff, like my boss is telling me I need to download this right now and build it and ship it. Looks good enough, let's go. I might build it, you know, do like make command. Let's cancel that just so that we don't actually build a bunch of stuff. And what that's gonna drop is into our outputs, which anybody familiar with actually building Kubernetes will have seen this local bin for this architecture. We see a bunch of this stuff. This is great, we got the Kubernetes things. Let's ship it, we built it. Now let's actually run kubectl just to demonstrate. I'm just gonna do like a get. Oh, that's bad, what is that? And that's really kind of the end of this proof of concept is showing that there are some modified bits, some modified code that I was able to essentially just serve you over S3 via that malicious looking URL and a very similar flow to what you would do to get that zip file off of GitHub. Let's go in here and let's search for that. Clusters, is that how I spelled that? Why is this not working? Oh, I'm in the wrong directory. Let's go up a few directories, one more. Let's search for that again, there it is. And this is the chunk of code that I injected in here into just the top level Cobra command, but this could be anything. This could be some bits that were modified on the kubelet, on kubetm, on kubectl. And at that point, anything is possible with modified code that you're distributing or building or running, anything is possible at that point. So again, really what we're talking about here is the role of social engineering in these kind of attacks. What this sort of scenario looks like is like, oh, like something bad is happening and we need to patch this critical CVE. Here's a link to some of the code that we need to ship and it looks legit enough. If you were to click that link in this Slack message, that would take you to my Kubernetes.zip website, redirect you to the S3 bucket to get the malicious zip bundle. Somebody might click into the CVE, be like, oh, that's pretty bad, this is a real CVE and we should probably patch it, we should probably fix it. So, where I'll hand off to Sean is with this bit. Software artifacts without software validation can be dangerous, so Sean, what do we do about this? Yeah, so what do we do? For one thing, like John said, this is a social engineering hack. This is kind of the bad idea. You're busy, you're in a rush, you don't know, you're doing other things and I can't look it up, can someone give me a link? And they do, but that's not the link that you want. So really making sure that you get your sources from somewhere that you can actually trust, so that you're going somewhere that you recognize in getting those. And then not only getting that source, but then you can make sure that you're actually downloading what you're expecting by looking at the contents and that the provenance of that source is really coming from who you think it's coming from. So talking to a few people, there is the GitHub link, you can download the sources from there. I actually found out a lot of people don't realize that the Kubernetes project has a website where you can go and download actual release artifacts from it. So that's, you know, don't grab a random link, actually go to somewhere, recognize, that's maintained by the community and has everything you need there. There's also downloadkubernetes.com that has each of these things, each different artifact that you might want. And one of the very good things here is besides the download link, there are these links for checksum and signature. So not only can you download from a source that you can trust, but once you download that, you can get that checksum and then looking at it locally, you can take that checksum, look at the checksum file that has this long string and then locally you can run SHA-256 sum, Windows has a similar PowerShell command, you point that at the file that you've downloaded and you can make sure that the number that you're getting locally actually does match the number that's published that is validating that yes, this is the same file, it hasn't been modified, it hasn't been corrupted during the download. So you make sure that what was published on the website you actually get. Now the one downside about that is these are both from the same website. What if this website was compromised? So a next level besides just actually validating that checksum is, oh and have some links there to help, there's actually some very good documentation. Besides that checksum link, there are the signature and certificate links and that's where you check some, make sure that nothing was corrupted and what you downloaded. The signature is where you can really validate that this really is coming from the Kubernetes community or from vendor acts or whatever software that you're pulling down. So very similar to how we check the checksum, we can download those signature files from that site and once we get the signature, we get the certificate. These demos always go so much faster when you're just doing them by yourself and think that you're gonna talk and it's just gonna have to catch up. And then they do give a nice command on the website that shows you, run this command and that'll take this signature file and verify that the signed zip file actually does match and then that's where you get this verified, okay that yes, this certificate file which you can open and look at and see who it's coming from is what signed this software artifact. And then one more thing that they do publish out there, every release of Kubernetes also generates, every build generates a software build materials and if you haven't seen software build materials, as bombs, what that basically is, is a text file that has standardized format that lists everything that's included in a software and a deliverable of what you're getting. So it's a text file, you can open it and read it. It is a standardized format so you can figure out where things are. It's not something that you really just wanna browse through. Again, they do a great job on the website and there's a link at the bottom here. They give this sample script that you can run. There are a lot of tools now that are starting to work with Sbombs where they can check the signatures in that Sbom file. They can also scan because of this list, everything that's in the software deliverable, it can go beyond that and see are any of the included dependencies here versions that have known vulnerabilities, soft security issues. So you can really, if you tie this in to those types of tooling, you can really get a lot more confidence in what you have and what you're going to put into production. So ideally, you can run all these commands, you can look at this Sbom, but using these tools, doing these checks and validations, that's a great thing that you can just put as part of a CI CD pipeline and then as part of your own build process, you can really make sure and have that validation that yes, especially through automation, what I'm pulling in and using, I know is something that's going to be good. Now I did also want to bring up another option. I'm one of the co-chairs for this special purpose operating system working group undertake runtime, so I'm legally obligated to mention it. But another option, there are, you can go and use distros that are geared towards running Kubernetes and running the containers. So here are a few of the options that are part of this working group. They've all done the work for you and do that validation as part of their build process and making sure that what is included in this distro is what's actually coming from the upstream community. So main thing, watch where you're getting things from. Even when you're in a rush, it's so easy to just grab a link from someone, but kind of like way back in the day when people could email around viruses and nothing could stop them. Same way, don't just click on things. Validate what you have, make sure it is really what you think you have, and make sure it's coming from where you think it's coming from. And the supplies to Kubernetes applies to a lot of other projects out there. A lot of folks are publishing the checksums, publishing signed artifacts, so take advantage of that where you can. And then my own personal thing, don't just pipe a random URL into a super user shell. It is really convenient. Again, if you're getting this, if you know where you're getting this from, it is a great convenient way to install a lot of software. It's also a great convenient way for someone to get you to install a whole bunch of other things that you don't know you're installing. So just be careful. And hand it out. Yeah, this is just a quick attribution to Bobby about, I think, really discovered the Unicode and the ASCII really applied this well-known, I think, hack to URL spoofing to using top-level domains. So a huge attribution to Bobby, and thank you very much. Yeah, short and sweet, we would love to spend a little bit of extra time doing Q&A here if anybody has questions, but yeah, I hope this was enlightening to the dangers of social engineering and the unfortunate unsafety of the internet these days with certain domains and things. Yeah, go ahead. Can we get the mic for questions on? Go for it. Okay. Okay, so a short question and then a more substantive question. I noticed while you were going through this talk that the URL that GitHub generates ends in the branch name dot zip. So a lot of these repos are gonna have master dot zip and that domain has actually been registered. Is it known who that domain belongs to and are they a good actor or a bad actor? I mean, that's a great question. We good, you know. And then the more substantive question is when you're talking about verifying signatures and certificates and keys, of course, it's great to publish them on your own website, but the first thing that a malicious actor would do if they breached your website is generate their own certificates, keys, et cetera. Sign everything so that it looks legit. Is there another site because that's always the answer, right? Is you should get these certificates and keys from a third party channel. So where else can we get these keys and search and things to verify these through an independent channel that is under separate control? That's a great question and yeah, I'll reiterate that. That is an issue. If you're pulling, you know, as I showed, there's this download site. It's got these convenient links there, but as he said, if someone were to compromise that site, it's easy enough for them to replace those checksum and signature files. So you can look at that signature file and depending on how thorough they are, you know, maybe you can tell if that's actually the signature of who you think it is or not. But yeah, ideally you're getting the zip file or the software artifact from a known good location and able to verify those signatures and check some from another separate, well-known trusted location. I'm not aware of like a common registry for those types of things. That's actually an interesting idea. I think it gets to the interesting point about like the software supply chain and like even something like that could end up being a single point of failure and where do we end up distributing all this trust to? I really hope that people aren't just downloading zips off of GitHub or releases, but in theory that in itself would be a single point of failure. Yeah, I don't know if there's an answer to that. To answer this question about like who has master.zip and you know, it's registered. You could dig into the DNS records, try to find from the registrar who has it. Most of the time those registration details are masked by the registrar, so who knows? You could reverse DNS, look these up and figure out who the registrar is and stuff, but typically DNS registrars make that very hard to figure out who actually has a website. So I mean, even if I put in and dig has additional stuff, Kubernetes, and this is not a talk about dig, but even that just comes up with, I mean it looks like also Google's DNS servers. So this wouldn't surface information about me, it would just surface information about Google's DNS, unfortunately. If there's no other questions, anyone else? One more. Just good news that master.zip is registered by GitHub. Oh, okay. Very good. It's safe, everybody, it's safe, that's great. And that probably makes sense. I mean, I'm sure there's plenty of good websites that point to actual use cases for things. I could see that being a legitimate use case for GitHub with that domain, but again, it's not hard to find really good socially-engineerable URLs that, some of those Slack messages, I would not be able to tell. I would click those links right away. And that's the scary question we hope that you can ask yourself and your organizations, like people just kind of suck at those kind of validations. That brings me back to my last slide. Software artifacts without software validation are just dangerous, so. Yeah, I don't think we have too much else. Was one other thing that may be worth checking out. There are some projects here that are working on this whole thing. Tuft, if you've heard of TUF, that's a great framework for everything that builds in a lot of these security checks. So if you're looking at how do I distribute my software and make sure that I'm able to provide a really good way, that sort of addresses some of those issues of, some compromise something and changed signatures. Tuft has a lot of protections for that. There's also Sigstore and some other projects around here that can tell you a lot more about it than I can. Yeah, hopefully in a few cube cons, this is a relatively solved problem where getting some random code zip file off the internet would be pretty easy to not shift into your pipelines. But with that, we'll be around for a little bit for a few minutes. Yeah, thank you everybody very much.