 Thank you. Hi everyone. Yeah, I'm going to get going. As it was so well introduced, I'm going to talk about pollinating build-out-of-stations with Kubernetes, Tetragon and EBPF. There's going to be lots of B-related puns, so please, like, I'll say in a second, I've got terrible humor, just laugh, help me out and try and have some fun along the way. So yeah, without further ado, I'll try and make my slides work and we'll get going. So level zero, no talk would be right without an introduction. My name is Tom, yes, unfortunately that is me, on holiday in Morocco. I am an open-source engineer at TestifySec, relatively new roles, so this is the first time I've got the pleasure of saying that. Yeah, if anyone has seen the B movie, You Like Jazz? Yeah, help me out, guys. Come on, just give me an easy day here. My powers or my strengths. I come from mainly a Kubernetes background and the sort of DevOps-like. I do security now, and my main language is Golang. I'm kind of proficient in Python a bit, but I shouldn't say that in public. Weaknesses, definitely EBPF, and I'll preface this. I am not EBPF guy, so take it easy on me. Hopefully for everything that you learn from this talk, I'll get to learn from others afterwards in terms of discussion and conversation and stuff. Security. I think anyone with just security will probably say they're not good at security. Humans aren't good at security, and that includes me. So yeah, listen to what I say, but don't necessarily trust everything I say. I mean, that's probably the right thing. Humor is terrible. I've got a weakness for Tiramisu, and I've got terrible dancing. If you're coming to Kuboroki, you'll definitely see that, and you'll see more of what is in that horrible picture in my intro slide. So level one, and that is building secure software. So my journey began in trying to understand this about two or so years ago, and someone said to me, can you go away and learn about this? So I went away and I tried to understand what this whole new thing called supply chain security was. And the main thing that occurred to me is it's all about trying to understand how we can secure the build system. At least that was the main focus that I found when I was starting out. So let's sort of work through how we build software, right? And at least, and sorry, EBPF and Cilium, if I've horribly doctored these B images, but I thought they were fun, we love to put in a ton of libraries, right? We love to just use other people's code. Well, I want to feel cool like pretending that I wrote it myself, but really the real genius is writing it for me. So I'll put in a load of libraries. I'll automate the build. At least these days, I've done enough Kubernetes and YAML that I really don't want to have to spend too much time doing this. Really, this is copying and pasting a load of YAML from somewhere else that builds my go thing and puts it in some parcel that I can deploy places. And that's about it. And then maybe someone on my team will review it. They might be on the bus flaking through their phone and be like, yeah, looks good to me, bro. And then we're done. That's pretty, pretty easy, straightforward way to build software, right? Let's look back for a second. And hopefully, this is big enough. But if you look really closely, and if you were eagle-eyed before, you might see a bit of a problem. And just to preface, GitHub, please don't ban me for violating the terms of service if you do see this. But I actually did this. What I want to point out to you is my little GitHub actions YAML. And you'll notice that I pulled in GoReleaser, except it's not GoReleaser. This is the GoReleaser GitHub action, but what I actually did was I created an organization called GoReleaser, and I dropped the E in the ER. After that, I just copied the GitHub action and added a bit of hacker magic. Yep, I just got pwned. And all of a sudden, as you can imagine, people left right and center are pulling in from my GitHub action and building their stuff with my potentially malicious software. So yeah, this stuff is getting easier and easier, but really to secure it is getting, I think, harder and harder. So yeah, as I sort of alluded to, game over. So how do we, you know, after I insert the coin and start the game again, how do we proceed to level two and try and solve this problem? I want to introduce to you attestations. I understand that this might not be a security audience, so I'm going to try and do my best job of explaining it all to you. An attestation, at a simple point, in the way I like to see it, is a signed piece of metadata that gives you an idea about something. In this case, this is an attestation, the main body of it called a predicate, which basically tells you, in this case, what commands will run inside my GitHub action. Of course, I've got my GoReleaser release dash dash clean, which I expected to be there. And then I've got this evil hacker magic, which you can see I'm not really a red team person, so don't expect any crazy hacking from me today. But if I have this attestation, I can transform it, or that predicate rather, I can transform it into this bundle of JSON, it's all it is, with a base 64 encoded version of it, sign that base 64 encoded payload, and hey presto, I can verify it with my key, and then I can start, I mean, at least if I'm really clever, start inspecting it and trying to understand what went on in that build, and hopefully stop that GitHub action from ruining my life. How can I verify it? Well, one of the ways that I can do it, full disclosure, I'm a maintainer on a project called Witness, which is a project for really, I would say, easily being able to create in total attestations and verify them in your supply chains. But I can use a tool like Witness to go and verify my attestation, and I might also have some policy to sort of inspect those attestations, but I'll show you later. I also hopefully will have a key to verify the signature for that policy, because that policy wants to be signed as well. So hopefully level two complete, right? Well, unfortunately, this is where the evil villain in this story comes in. So a wild hacker has appeared. Oh, hold on. He's used an exposed Kates API server to run a process inside my build job. Oh, no. What does that mean? Well, unfortunately, for me, it's super effective. By the way, I've been playing way too much Pokemon on my phone recently. There's this was a module thing that you can install on your iPhone. It's great, but I won't spend too much time talking about that. The attestation looks the same, but I thought we were securing the supply chain, right? I thought this was solving all my problems. But in fact, it's made everything just as bad as it was before. I didn't solve anything. What did I get from my attestations? So what do we have to do for level three? We have to beat the evil hacker, right? And how do we do that? If we want to be the very best, B-spell, B-E-E. Yeah, hopefully you got that. It's terrible. Right. So yeah, how do we beat the hacker? So we want to generate our attestations as I showed you earlier. We have our GitHub pipeline or whatever is running. And we're going to generate our metadata to secure ourselves, right? Awesome. But where is that metadata coming from, right? In this case, I've got my runner running in some environment and the evil hacker exposed it because I exposed the Kubernetes API server, which we'll come onto in a moment. But if you look at my generate attestation.sure that I've mentioned above, I'm just basically even pulling that from the source code that the runner used to invoke my build or something like that. It's not telling me what actually happened when the build took place. So when evil hacker comes along and executes his evil payload with Wget or whatever, that's all out of band from what the attestation is generating. I can't see any of that. Let alone verify it. So I started sort of working through this and going down the security rabbit hole of how do I fix this? Well, I could have this like observer pod, I guess. And all I want from this observer pod is nothing but facts. And I sort of looked to myself and I was like, well, Tom, where are you going to get all these facts from? And this was like a year, a year, two ago. And I thought to myself, I have no idea until I came across, um, ebpf, which I knew nothing about now, um, technically. And I still know very little, little, little about today. Um, but I went back to the evil hacker and I said, you're going to pay for that. I'm going to write a Kubernetes controller is going to watch all my runner pods and keep them safe. Preface this. This is all on Kubernetes. If you're running on GitHub hosted runners, I can't help you, unfortunately. Um, it's going to use ebpf. So hackers like cool story, bro. It's not that deep. I was just, just having a bit of fun, but whatever. If you want to take it, take it more seriously, how do I, how do I map ebpf events to Kubernetes pods? And this was me about two years ago. And then finally, or maybe three, two years later or what felt like it, I came across genuinely, I hope Liz is in the audience, the most awesome talk ever. Um, I remember where I was. I wasn't at, uh, security con, but it was a talk about a project that would map all of these ebpf events seamlessly straight to the pod. And just basically did my job for me, right? That's what I want to do. All the magic's done inside of this project that's called tetragon. I advise if you don't know about tetragon, watch this talk. It's super cool. Um, so I made a testagon. What is a testagon? It's a Kubernetes controller. It uses GRPC to stream and cash those tetrachon events. And then it watches for annotated events or annotated pods, pardon, and then condenses them into the tetrachon events into an attestation so I can verify it later. It also has a totally AR generated mascot, um, which you should have seen in that, in that last picture. So final, final boss level four demo. Now I was up until 4am trying to get this to work. So if it doesn't, please just give me a break. I'm super tired at this point. Um, also, um, the builds are taking like crazy, crazy long. So, um, I've gone ahead and built some already, but I'm going to go and set one off anyway, just so we can look at it later. So a testagon at the moment only works with tecton pipelines. Um, so it, uh, yeah, a runner, so to speak, running in Kubernetes, but I built it with the intention that it can run with anything, whether it be, um, a runner from, uh, like a Git lab CI runner or, um, I'll go workflows or whatever it is that you're building your pods in. Um, so if I go and take my pod.yaml, um, which hopefully you can see here, which has got a digest and so on. Um, I can try and create the pod like this and hopefully the evil hacker will be stopped. Aha. Okay. So we're seeing a few things here. Um, so it's saying it failed. So this is a gate keeper policy that I've made ahead of time, which I'll hopefully show you in a minute, but it's saying a few things saying no open to please. That's interesting. That's an extra curveball that I threw for myself. Port 80. What is this? 2001. And then, um, why are you opening main.go so many times? Wild hacker me thinks. Okay. So let's try and see what's going on here. So if I get pods, I should have that task run that I was running from before. Um, yes, 69 seconds. See if this is giving me any, anything interesting. So I mentioned earlier, I'm not a hacker. So don't expect anything crazy from me, but I'm cloning this thing along with my artifact, the artifact that I'm planning on bill by the way is called might test, which we'll show you in a second. It's not very clever. Um, but I'm cloning this thing called solar sploit, which I don't recognize. And it's talking to me about active PIDs target PIDs and operations not permitted. My main.go. Oh God, this doesn't, yeah, this doesn't look good at all. Um, let's see if we run this, run this container. Let's see what it does. Let's copy that and docker run again. If this doesn't work, please don't kill me. Go on. We'll let that, we'll let that build, but it's doing some crazy stuff, right? Um, but luckily we've got a test gone, a test to go on running in the background, seamlessly creating a stat traces and reconcile errors in the background, but you should get the picture. Okay. Yeah, it's built for the wrong architecture. Basically what the solar sploit is doing, um, and I'm simulating the wild hacker, um, it's sort of injecting this into the build is using, um, a load of syscalls to pause the go build in the middle of it executing and it's injecting again, just some evil hacker, evil hacker logic. Um, so when I run my mic test, which you won't see, and maybe I can just delete everything at the end and run it for you on my AMD 64 Kubernetes cluster. Cause for some reason this is all, um, yeah, you should see some extra log lines and a load of stuff's been injected maliciously and we, we absolutely don't want that. So, um, let's see this build is just finished. So it's saying stored in archivista. So let's go check that out. Archivista is, um, it's part of the intoto organization and it's basically a really easy way to store and query your attestations. So you can upload them at the end of your builds and then you can do what I'm doing now, which is check them out. Um, so I can see this attestation was generated for my image with this digest and I can see that I've got an attest to go on attestation, which just looks pretty cool. Um, apart from, well, hold on. If I just scroll down, you can see this can be quite long. I'm going to go main.go. Um, there we go. You can see that I've translated the TecheCon events, which my controller was looking at essentially into, yeah, into a form that I can query them and start writing policy against them. I appreciate that this doesn't look very, um, hold on. Let's see if I can let's see if I can make this look a bit better. Let's see if I can try and use my laptop properly. How about that? There we go. TecheCon provenance. So I'm just trying to condense those events down into something that I can query and make a policy for or get along the line. Um, and I'm running out of time, so I'm not going to try and go into that too much detail, but what that means is I can start writing a policy. So if I go to my policy at TecheCon, as I mentioned to you, the witness project has a, a built in policy engine. So I can start writing policy files to check the attestations that are in Alcavista or wherever else. Um, yeah, to make sure that the right ones are present. And hopefully I can inspect them as well to try and verify that they contain the things that I want. You can see that I want to have a material attestation, a product attestation, an environment attestation. We won't go into the detail of those today. And then I finally want to have my attesticon provenance at the very end. And you'll see this blob of, of rubbish. What if I get that out? Cause it's a base 64 encoded version of a different file. You're going to see some rago. So this is where I was getting for what I was getting from my gatekeeper policy earlier. I can set things like if there are any TCP connections made with the destination port 2022, like SSH or something. I don't want to do that. Port 80, shame on you, you've been to please change that to 443. Um, we don't want to use that. Um, it's, it's an unsafe way of communicating with, with, with remote servers. Um, yeah, socket for 80 that same, same sort of thing. Um, and I've even gone ahead and blocked an IP related to something you've been to related just, just for the giggles. Um, also there's a, there's a, uh, cat of how, um, hello, TXT, um, or, or yeah file opened of hello, TXT. I don't want that. It's just an example, but most importantly at the very end, I've got this. So I've noticed that, um, when the solar sploit was running, um, an unnecessary amount of times was the, uh, main dot go of my project opened. So I've added this, uh, this constraint to basically go, why are you opening main dot go so many times? Um, so that's all, that's all fine and well. Um, how do I fix this? Well, if I go into my hack, I've got a fixed, fixed version because I don't want to do them live. Um, right. So I've changed a few things here. Um, I've switched to alpine get. So I'm not, I'm using Ubuntu and I also don't need to apt install, get and all of that stuff, do the apt update and apt upgrade. Um, I've also removed the evil hackers injection. I appreciate that in real life, I hope the evil hacker would be a lot more evil than this. Um, and probably wouldn't have access to the task YAML itself, but you see the point I'm trying to make. Um, so if I go ahead and I try and not break everything and I apply the fixed task, um, we should get too much time not doing Kubernetes. We should get the ability to rerun this. Um, and theoretically everything should go. Um, so yeah, that, that's basically the premise of it. Um, there's still stuff that I need to fold out. Um, but yeah, I'm happy to at this point probably take some questions unless there's anything pressing that I've missed. Um, I don't want to go over time. So, um, and hopefully once that, yeah, once that's run, I can show you. Um, I mean, to be fair, before we take questions, I think it's about to finish now. Come on. This is where the, the nice elevator music needs to come in. Um, creating new predicate. Is it going to finish? Maybe 25. Okay. Maybe it's not. Anyway, what we would expect to see or hope to see is now that that's fixed, we've remediated all the things that it's going to be triggered off by the attestagon attestation being queried by all of those rego constraints and what we should be ended. Oh, oh, hold on. Let's use Archivista CTL to take a quick look at that attestation. Um, and I'll get the digest and I will go to my pod and I will not try and use Vim shortcuts to do this. I'm going to do it the old version way and I'm going to create that pod. I expect one will still fail, um, which will come on to in a moment, but hopefully that'll look a lot better. Hello, TXT. Why? So, the reason why is because while with our original attestations, we're only just seeing what's in the source in, in, in, in the runner task or the job or whatever you want to call it. Um, in my, in my docker file for the project and building, you can see that I'm echoing hello world to home. Hello, TXT. I appreciate normalicious thing, but it's pretty cool that I can create attestations for supply chain purposes that don't just tell me what's executing. They tell me everything about what's going on in that invocation. If I go ahead and I delete that, um, and I commit it, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, oh no. Get a UBK, they said. It'll be great, they said. Jingle, jingle, they said. Um, if I commit that, if it lets me, or it doesn't because probably because of the dongle, if I, if I were to have committed that, um, you would see that, um, that final, uh, yeah, that final problem would no longer occur and we would be done. Um, I think you're probably all done with me talking as well. So I'm going to go ahead and open the floor for questions. Um, yeah, just so you know, my GitHub user is chaos in the CRD. Um, yeah, please, um, also reach out to me on Slack, CNCF Slack or whatever. Um, I'll get my final slide up as it's got a bunch of QRs. Um, and yeah, like, yeah, final slide. Hope you've enjoyed the, the, the, the arcade-ish, um, Pokemon theme. I had fun creating it. Thank you very much for listening to my talk. Um, yeah, not enough. Goodbye, sir. Let's play. Thank you very much. If anybody has one, it was a great presentation. I liked all the jokes. Thanks. You maybe repeat if there is a requirement for Tecton or not? Absolutely. And I can probably show you. So the only thing at this current moment in time, I'm depending on it being Tecton 4 is this. Let me show you. So what I need to create my attestation is to understand what, um, what artifact was built in my pipeline. In the same way, if you might be familiar with the project Tecton chains, it creates cells of providence for Tecton. Um, it needs to understand what was built, and you need to pass that out. Uh, which I can show you how it does that. Um, dash and default dash. Oh, yeah, more. So if I get this, and I'll put it to less so I can search through it. Um, I think it's around the bottom. I need to output, I forget what it's called yet. I need to output in the terminator state, a message. And you can see that my digest is char two five six BDD zero seven one, blah, blah, blah, blah, blah. That I think is the only dependency. So I say it only works with Tecton. Technically that's not really a constraint because you, if you in your whatever workflow engine or whatever you're using, you can do the same thing. It will work. Sorry, I'll correct myself. It should work. Um, so yeah, that's why I say it only supports Tecton pipelines, but the intention is that you could use this for like basically anything. I'm using this for builds. Like in my opinion, like maybe just straight tetragone would be better used, but you could technically use this for any pod that you wanted to have like a an attestation for. It doesn't need to be for a build. Uh, hope for balance is your question. Okay, great. Tom, thanks for showing us how we can catch them all with tetragone. Thank you for laughing at my jokes.