 Good morning. Good afternoon. Good evening. Wherever you're hailing from welcome to episode three of Kubernetes by example insider I am Chris short host of redhead live streaming and I'm joined by some of my favorite red headers I'm joined by the one and only Langdon White Gordon Tillmore and Luke Kynes is our special guest today Langdon You're back. Really redhatter. Normally a really red. Yeah, I've got I haven't got that yet It's only been a hard habit to break isn't it? Yeah, well, I try to keep the reminder with I have my oh You don't have the hat in the background today. I was I was looking for matching hats As you can see I have a Boston University hat behind me over my left shoulder. Yes But you often have the hat there so welcome to everybody for the the third edition as you said of the Kubernetes by example insider Although we we may be changing the name soon. I don't know It'll probably everything will change around the same time as the level of our swag shows up is my my current theory So, yeah, you had to do that Had to put the big game So today we always like to start the show with a little bit of kind of news from what's going on in the Kubernetes world and Unfortunately, our normal news host is currently on vacation. I believe or starting school And so we have Gordon here for us as a fill-in replacement And I'm sure he will do a lovely job. Maybe not as good as Mina, but you know, it'll be close It's it's man big shoes to fill, right? So yeah, I mean it is off with family in Turkey and hopefully she's not watching this hopefully She's exclusively with family and steering away from work. Yes. I am I'm filling in this month and I'll do what I can to match up to Mina's high standards But yeah, let's start with news. All right, I'll be judging really you'll be judging. Is there like are you the East German judge? Am I gonna get a bad mark from you? We'll see All right. Well, I'm gonna do things a little different than me to Mina would cover. Oh gosh 10 12 Sometimes articles, I'm gonna focus on four. I think and in particular, there's a theme here a security focus this month And a lot of security articles popping up over the wire as late. I guess it's no different than any other month, right? But But it's been a year this month than others. Yeah, I would agree with that And we also have of course this feeds in really nicely with our guests who are so happy has joined us But let's let's let's start with with the first article here So all of these articles can be found on KBE news So go to Kubernetes by example and navigate over the news section I'll put links in in the chat window here in a moment But the first article we're gonna cover is the what in the why of cloud native security And that comes to us a courtesy of the folks at container journal and the whole main gist of this article is That too many organizations too many still rely on traditional security efforts for their cloud native architectures, right? So You know traditional tools traditional approaches traditional policies even and this is of course a mistake for so many reasons, right? So the author then goes on to define cloud native security, which helps but he also breaks it down into Something I hadn't heard of maybe you guys have heard of it. Maybe I think he coined this phrase actually it's called the four C's So if you picture a cake Starting at the bottom layer with with the platform with with cloud, right? And then you move up to the next layer and it's cluster and then you move up to a third layer and it's container And then code is your final so this this four C's and and the whole his whole Idea behind this is look each of these layers is different and you got to treat each of these layers differently with your security approach Right, so it is a cool article and I will post that link up here in a moment as I mentioned But that leads nicely to the second article that I want to focus on And that is called. It's how to harden kubernetes systems and minimize risk And it's from help net security, but it's in you know These guys are essentially showcasing a recent report that was put out by our friends at the NSA And also the cyber security and infrastructure security agency say that five times fast But so NSA and CISA right and so Yeah, the report lays out the primary threats to kubernetes environments and actions that we should all take to combat these threats, but Chris Langdon Luke even How about a quiz and your opinion? What are the three's I'm keeping it interactive here What are the three leading types of attacks and kubernetes environments according to our friends at the NSA? Yes, and I read this and I've forgotten that damn it DNS poisoning. Oh, that's one. Okay. I Think like a hacker apparently Man in the middle scared No, well, well here. Let me give you that too. So data theft, you know, I'm very general, of course. Okay. That's really broad Yeah, yeah, I know I know they're going they're going broad with a lot of this But the second one was interesting computational power theft Oh, no, I read it. I read a paper about that this morning Is this like the coin mining on on Exactly yeah currency mining. Yes, and you know, it always comes back to cryptocurrency doesn't it but There you have it. So thank you NSA, but they spell out any number of ways that we can all minimize these threats, right anything from scanning pods and You know and clusters for vulnerabilities to running containers and pods with the latest privileges or the least privileges possible and You know using strong authentication firewalls, etc. All that fun stuff But the full report that's full article courses art is available on KBE news I again will post all these points here in a moment the third item the third article I want to speak to you guys about this morning interesting Is called concerned about Kubernetes security Check out these three tools. So tech genics puts this out They focus on these four open source tools that will help us maintain Kubernetes security I'd heard of one of them. I'm curious if you guys have heard of any of these four You probably have your closer to it than I am but cube hunter is one So finding security holes and we have or haven't Chris curious have have yes, okay That's the one I like has been featured in newsletter kind of thing. There you go. Yeah cube burner is another so stress testing I've heard of that one now so no buoy Which is determines overall security levels of a Kubernetes cluster by running a set of plugins and then the fourth one's my favorite one Just because of the name and I haven't heard of this. Maybe you guys probably have powerful seal Anybody heard that it injects failure like a Yeah, like seal like the animal was spelled the same anyway. So yeah, you know Langdon so in my mind, I'm gonna gravitate towards the animal then Well, I'm sure it's a pun right like right. It's like a has got to be yeah That's pretty good idea behind this. Here's the cool thing It injects failure into Kubernetes clusters and you know, basically tests the evidence and users how quickly can you fix this? Right. So what they call it complete chaos experiments, which is yeah, so it's like Netflix chaos tools Which are like still like blow my mind the whole you know, let's let's knock out a server and see how we react Let's knock out an entire service right a certain region and let's see how we react. Yeah, right. It's pretty wild There you go. Yeah, exactly. I think that's a yeah Like what if you're getting to the point where you can use tools like that you're you're in a good you're doing something right, right? You know, which I think is is rare enough. Oh, I forgot. I have a kid a Kubernetes by example coffee mug I could have brought that you to I totally It has been co-opted by one of my children. So that's part of the problem You know, I'll give you guys a quick hint with this with Kubernetes by example mugs and shirts and everything else Look to see them on the Red Hat cool stuff store soon. Oh, cool. Stay tuned I know you always like talking about swag LinkedIn, but I do I do like swag. Yes. Yeah, well, there you go We might even be able to I might even be able to get three bees your way Chris. We'll see Anyway, let me tell you got the fourth item on my list the final item and I think the coolest of the four frankly Because number one I love wired magazine and yes, I still I'm an old-schooler who gets the actual physical wired I have to subscribe to the magazine to get the cheapest way to get unlocked website There is that I can't do with ads though. I think you can I don't know But yeah, I still subscribe to the magazine Something about since the 90s There you go. Well, you beat me on that one But there's something about sitting on a couch or on your porch and reading an actual physical magazine that still appeals to me Maybe that speaks to my age, but anyway, let's talk about the article in here And it's cool because it's from wire, but it's also cool because it features Somebody near and dear to to the Kubernetes community and somebody appearing on the call to on the show today a new tool Wants to save open source from supply chain attacks So we've all been hearing more and more about supply chain attacks in the news as of late So in essence rate a hacker slips some code bad code into legitimate software It propagates and well before you know it destruction ensues, right? So we saw that with With heck this Russian solar winds cyber cyber espionage effort, which we're still plagued by I think we saw it with Not Petsha the malware attack, you know, thank you Putin for those and you know this this article talks about how we combat that, right and The big item on the list to combat that is something called sick store and code signing and who would know about sick store Yeah, yeah, but man if that's not a perfect segue for you Langdon and Chris, I don't know I can't do anything more for you Right, right I think that's pretty good. I did want to make a quick comment about the first article you mentioned Which was kind of talking about the three or four C's there I've been reading a lot of spy novels lately And one of the things that kind of keeps coming up is you know layered security, right? Is that you know if you have multiple layers of your of your security mechanism and this is talking about in the physical world, right? That's the best way to do it because you know we breach one and then you have to breach the next one, right? They're not dependent on each other and I think I Think in the back of a lot of technologists kind of mind, right? They knew about this concept right for a long time, but You know, it's really started to show In I would say more recent years is that we're really starting to have a much better You know effort towards that, you know, if you just go to like a firewall with SC Linux, right? That's kind of multiple layers But with that, let's transition to our guest for today And that's Luke Heinz who is conveniently quoted in the Wired article that Gordon just brought up and you know We invited him here today to talk about sick store and talk about kind of like why You know, what do we want to do there? But before we do that Luke, do you want to give a brief introduction to yourself as as I often joke? You know, it's very hard to keep track of titles and organizations inside Red Hat and now that I'm not even employed there It's even worse. So rather than me guessing and being wrong I will just say could you please introduce yourself? Tell us what you do for Red Hat? And and then we can kind of start talking about Kubernetes Sure. Yeah, okay. So first of all, thanks for having me on the show. It's great to be here really excited to take part in this and Yep, so I am Luke Heinz working Red Hat as already outlined I'm in the office of the CTO. Okay, and in the office of the CTO We have a department called emerging technologies. Okay, and our focus is One or two years out essentially technology that is not considered enterprise-grade as yet it's perhaps new idioms that have been discovered and projects of various projects are trying to collaborate to find consensus on a particular project that can become a solution to a particular problem set So I have a team of engineers that I lead in the security domain. Okay, and we look at all sorts of areas Predominantly cloud native. Okay, there's a lot of stuff around cloud native stuff. So container runtime security, but also things like trusted execution environments Trusted computing so sort of TPMs and so forth and of course software sign in okay and software transparency and supply chain security So yeah, I've been at Red Hat for just come up for six years now and Have a long history in security and open source as well Cool and in your history and security has security been getting better or is it about the same? Peaks and troughs really. Yeah, a little bit like the Bitcoin chart. You think it's going the right way and then it kind of changes You know It's funny you mentioned like the peaks and troughs because yeah, we've been in worse situation security wise than we are today for sure Yes, we've been it better too. Yeah, I think to be fair. There's been a lot of disruption Okay, so if we look at security, so so land and you made the point area earlier about lead defense, okay so Traditionally security, it's been a relatively simple ground. Okay to think about you had a Green zone and a red zone. Okay, so effectively everything outside there is just not trusted. Okay, everything inside is trusted And this is our citadel and we protect that so I used to work on Firewalls quite a long time ago. You know, you literally had a red interface and a green interface Good and bad good and bad cops on a thing Cloud came along okay, and then the principles are last sister C scale ability Hybrid Infrastructure hybrid cloud all these sorts of things and it just turned the whole thing on its head essentially where The trust boundaries were no longer easy to define It was a very mixed grouping of security controls that needed to be Fought about and then implemented and then of course software has accelerated softwares each in the world So we're starting to see Projects utilize a magnitude of dependencies that come from multiple different sources. So in a lot of ways Securities definitely got to be more of a challenge, but it's an exciting challenge I feel but I think in the olden days. I'm sounding like the old guy now Yeah, I would have said it was simpler. The attacks could have been very complex, you know, right? And the particular vulnerabilities that there were but the whole architecture was definitely easier to grapple with really So so I would slightly disagree with you in on one aspect Which is that one of the things that I thought as a consultant in those days about the move to the cloud Was it actually made people start to think? significantly more About the kind of like holes in their firewall or whatever like, you know I still remember working with a lot of banks and you know and basically they'd be you know Like most banks right they run kind of batch jobs of processing overnight or whatever And they would have all of these holes in their firewalls from all these different organizations. They work with But then when they kind of move to the cloud They did a bunch of things right they started to split their services up across virtual machines, right? They started to have to be conscious of like what was routing where and so while I completely agree with you It is way more complex I think some of that complexity Help to drive now we need people who think about this problem way more than you know It's just an extra job. This is sad and has to do And so which I think is kind of an interesting, you know, kind of ramification that while you're completely right I think in some ways it actually had a positive net effect Even though the complexity kind of went way up if you know what I mean, you know, yeah Well, I mean to speak to Luke's point We wouldn't have I mean we have things like let's encrypt now or more of the internet is It served over, you know a secure connection than not so I would say yeah, that's A win for security, but we've also evolved from the days of like Worms and such to now it's like a massive distributed denial of service attack, right? So instead of having to infect systems now We can just reflect attack against other systems and you know, we end up in the same outcome, right? So it's kind of weird. Yeah, like the scaling has given us more capabilities as well as The adversaries too, so yeah at the same time it's introduced Automation and agility to security as well, so right and companies like cloudflare too, right? You know who basically their their job is you know figuring out DDoS, you know, and you know in solving for that problem Systems are a lot more ephemeral as well. Yes, you look at the old traditional system It's you kind of you spend two days installing your operating system getting all the network cards working And it's like right. I just have to remember to run pseudo yum update every two weeks And you know, it's very monolithic and obviously now everything is very ephemeral All right, so let's bring it back around to Kubernetes and kind of open source So one of our kind of standard questions We like to do on the show is you know, and this may be in the pit, you know independent for you But it's like so what brought you to open source or what brought you to kubernetes first? I'm not sure which one you were involved in Beforehand, you know, if you were you know, what why why open source? So, okay, so open source goes we're going years back here. Okay, so probably I have to admit about 20 years ago Yeah, I know the feeling Yeah, so this was there was I Worked for as a software developer. Okay, and we were developing a speech recognition engine So this was actually for mobile devices. So this is stuff like harm OS and windows. Yeah technology and We had a couple of offices one in London and one sort of out in the sticks where I was Okay, and we needed a a simple point-to-point office VPN connection Okay, because like we had an exchange server and the sales folks needed their their email and stuff like that and I'd predominantly Windows have been my gig really. I didn't really know much about Linux at all. Okay, and I started to play around I got a very early version of rail red hat Linux, okay, and Installed it and then I remember sort of double-clicking on things thinking they'd be like an executable and nothing was happening and Quite confused this new alien world Like I had to total my whole frame of perception around operating a computer was specific to Windows I didn't even touch the Mac and so anyhow, I kind of fell into that Pandora's box and and fell in love with it And so anyhow, I had to create this sort of VPN tunnel effectively and then think about I think they used to call them roadwarriors people that will be dying in into your network And so it was I think the technology at the time was PP point-to-point tunneling protocol. Oh, yeah Projects. Yeah, so I started to get those working and I had to learn how to compile things Compile up a module for the kernel and so forth like that. So I got involved helping them with documentation Trying to sort of fix things that didn't work and and interestingly enough the Distribution where I was quite I was more prolific with the work that I was doing was rel 8. Well, this was actually called red hat 8 red hat Yeah, which was red hat 8 like years ago, right? I'm sitting Yeah, so So yeah, I kind of caught the I caught the buzz there really I'd be you know became a Linux user effectively. I joined the Linux user group and Yeah, and then it just sort of throughout the years I sort of gravitated to that ecosystem more and Security had always been something that interested me and you know, I started to sort of gravitate towards that area and and that's where I am today really so I've kind of got a fairly long history of Working in open source, but I'd say really the past sort of six seven years that's been accelerated where I've worked on projects and really started to Contribute code and it's going to become part of communities and stuff like that. So I guess I was more of a sort of Power user, I guess you'd consider right, right? Yeah, which is how we get mostly we start off as users That's but it's part of like in some ways It's like the drive for like Fedora, you know ambassadors or whatever to try to drive adoption Right, because the best way you get contributors is people using your software, right? And then then something annoys them and then they want to fix it and so they come along and become a contributor Yeah, going back to the red hat Linux 9 I Or 8 rather I think the next version of rel will be the first version that actually goes past the version numbers Yeah, so I'm hoping for some sort of big celebration as a result, you know I'm curious if they actually end up doing a nine because you know a nine is a bad luck number in some parts of the world That's why like Apple like skip nine Yeah, so mostly like some Asian countries, I can't remember which exactly that that part of the world So all right, so so kind of going back to Kubernetes, okay So you talked about kind of becoming you know a Linux power user, you know that kind of you know pulled you into the fold So what attracted you about Kubernetes and what brought you to that, you know that particular project? Yeah, sure. So I followed Kubernetes from pretty much the early days I think when it first started creeping outside of Google and getting known by various Get on the map of various people as an interest in technology And I'd actually been an open stacker. So another sort of cloud type infrastructure project so I've been working there for quite some time and I had a Community position as a project team lead like an elected lead within the community for security. So in there we would manage Sort of help within bargos and creating projects we sort of do all sorts of things really Documentation there's a kind of a multi-rolled group and there's quite a few people collaborating and things start to quiet and down there so security started to establish itself quite well in open stack and I've been following Kubernetes for a while, but I wasn't prolifically developing to the project and Somebody that I know Was on this security used to be called the product security team Okay, so now we've just renamed it to the security response team Okay, and they wanted to rotate out and they'd heard that I'd had a fair amount of experience Managing barcode responsible disclosure type vulnerability programs in open source projects So they you know asked if I'd be interested. I said I certainly would and you know and from there I started to get involved, you know, so I I kind of came into Kubernetes security without a very in-depth knowledge of the code base Generally with security, it's the same things that you see happening. It's you know, it's right It's it's you know XSS a tax SQL injections. It's you know, the language is different, right? You need to learn the architecture a bit more and so that really sort of got me more involved in in Really starting to ramp up on what I understood about the architecture of Kubernetes, you know What was a pod what was the host what levels of access to the container have to the host and you know What was the scheduler and I just you know because I was looking at these vulnerabilities that were coming in and Would have to sort of replicate those to make sure that they were in fact real vulnerabilities so, you know, I had to learn how to stand up a cluster quickly and you know pulling up an old commit and you know Look into the you know, just just all this sort of stuff ready to to validate the Yeah, yeah, that makes sense Yeah, I mean to your point about them, you know, it's like if there's nothing new under the Sun kind of right I mean, that's why I think it's a wasp right has that nice set of I think it's 11 You know top vulnerabilities you should watch where you know, like there's Like if you do a little bit of research as an average developer You can have a really good idea of the traps you're likely to fall into Which is one of the nicer things about kind of doing a secure, you know development But it can be tough All right, so so that brought you to the security response team And then you were largely involved with that I assumed primarily for a while What kind of brought the sig store kind of concepts Forward for you or what, you know, like what what it's where you're trying to scratch? Yeah, sure. So I Started to pivot to my focus to secure supply chain. It's about two years ago. Okay, and Had this idea of so I took an interest in these things called Merkel trees And it's nothing to do with the German Chancellor. It's this kind of This is kind of cryptographic Good thing you play when I won't make any jokes that I have to apologize about later. Yes. Yes. Yeah, you'll be so Yeah, I used to talk to these about Sir Brandon Phillips He used to have Cora West and he was at Red Hat for a while. Yeah, yeah Kind of turn me on to this principle of Merkel trees. Okay, and at the same time I'd been wanting to find a Some sort of system that can act as a source of truth around what's happened in a secure supply chain because a lot of the time you're you're you're relying on auditing systems that are susceptible to Manipulation effectively, you know, if it's like a logging system or you know Sys log or some sort of data store a hacker could one breach a system and then they could cover up their tracks Effectively and just this there was that that was one aspect and there's this kind of again this sort of spaghetti spool of dependencies and you know, and this is what they call S bomb now I don't even had the term S bomb probably did. I think bomb was a thing build a meeting with a thing. Yeah. Yeah Yeah, and and so I was just thinking it'd be great if we have this sort of Some sort of source of truth that that's that has an immutable structure again And so Merkel dreams the perfect example there so started to to experiment and and to prototype around that technology and came up with a Project that I called recoil. Okay, and recoil is Greek for record So I use with Greek words a lot of projects do as well tecton is a Greek word I can't remember what the translation is So I rate this prototype called recoil. Okay, and it's essentially a Merkel tree So I should explain for folks what a Merkel tree is so to put it simply Merkel trees are Actually leveraged in a few technologies. You'll find them in a blockchain the transactions are signed with a Merkel tree gets actually operates a form of Merkel tree. Okay, and BitTorrent is a quite an old Not an old technology, but it's a technology that's utilized a Merkel tree for quite some time and a Merkel tree is essentially You have these things called digests. Okay, and they're fingerprints Any type of artifact? Okay, so it'll be a long string of numbers and letters Okay, and then it kind of represents the integrity state of a particular And then the idea is if you change a single bit of that object it would change the entire string Right. So what a Merkel tree does is it takes these digests and it has let's say you have a layer of eight It adds two together and it hashes those which then go up to four and you go up the tree Till eventually have a root hash Which is a bit like a commit hash effectively and that's kind of a very good Representation of the integrity structure of that tree. Okay, so that's these things called Merkel trees and So I had this project recall which leveraged that and then built like a simple API over the top a restful API so that people could make conclusions into the into the tree and they could verify That something is in the tree and it's not been tampered with effectively So I came up with that idea and had a prototype and this is where it's a kind of classic open-source story really I had this code Wasn't quite sure what to do next, you know, so I thought I need to share this with some people see what they think and There's a an engineer at Google Dan Lawrence who I've been collaborating with around different projects Tech Tom CD and The open SSF the open Open-source security foundation. So I shared it with down and said down. I've got this thing. I'm not quite sure what to do next You know, you're interested and he said, yeah, I'd like to contribute to this So then there was two of us if you see what I mean and then another guy Red Hat Bob Calloway He got involved and started to refactor the code and improve things and it just sort of expanded from there really it just sort of blew up essentially and and this The problem was when we originally called RECOR But then we found out there's a company operating under the name of RECOR Okay, we have to do a name change and we had to do one quite quickly because we were speaking to the Linux Foundation A bit of a driver. So then you've got kind of trademark lawyers and you know, right So we came up with sick store. Okay, so we've signature store Okay, because we kind of took this concept to the transparency log and then it evolved To being like a sign in part of us an overall sign in system, which we have in sick store now because originally it was just the data store. Okay, and we then realized that great. We've got this way of recording Events, okay We want those events to be signed Cryptographically signed so that we have non-repudiation around who made those signatures who signed that artifact. Okay But we then realized that the sign-in tools that are around Kind of suck a bit really people aren't using them and then we realized what we've got a big problem there Do you see what we've got this wonderful? Transparency love thing now, but how are we going to get people to use it because they don't like sign anything so that was right, right really and and That was where Dan had this really great idea about leveraging open identity connect and stuff like that Okay, so that we could then handle the key management challenge and yeah And that just sort of kind of grew and grew and grew until we started to look at signing more and more Artifact types. So we started with containers and then somebody would come along and say I want to sign a jar file How can I do that and and I want to sign an s-bomb and and yeah, it just sort of blew up from there really so in a lot of ways It was that kind of Is that it was that magic of what everybody wants really which is the right idea at the right time, right? Right and and consumable, right and yeah, and usable and then there's a problem there You know, there's something that I learned from who did I steal it from I should give them credit brandon phillips again So I heard him once say that there's this analogy, you know, I've always stuck with it, which is In software you can have pain killers or supplements. Okay So a supplement is something that people will say, yeah, that's pretty cool. Yeah You know, that's that's interesting. I'll follow what you're doing. Okay, but you're not, you know What you have is a supplement if people forget to take their vitamins in the morning They don't start losing their mind and turning the car around going back home Right, you see what I mean. Yeah, whereas a pain killer is an immediate need There is a problem that's hurting and you need to solve it. Do you see what I mean? There's a there's a nice physical reminder to take your pill Yeah, very much. Yeah. Yeah, and and so it's a very good sort of spectrum to Evaluate is your project Useful, do you see what I mean? Are you solving an issue? Are you a pain killer? Or are you a supplement because I'm somebody that's I've wrote thousands of supplements You know, and it's only occasionally you're lucky to get a pain killer. See what I mean Yeah Yeah, yeah, totally. Um, so So You know, I kind of am curious. So do you see kind of sick store expanding beyond the kind of kubernetes realm? Like is there, you know, is it is it a more generic solution? Um, because it kind of sounds like it is Um, but I don't know, you know from not knowing the inner workings I don't know how like how dependent it is on that on that tool chain in a sense Yeah, very much. Yeah. So so we've sick store We have the kind of the infrastructure services Two of our sort of core projects. So one is recor that I described which is the transparency log And the other one's called force here, which is the the the pki the ca software sign-in solution Okay, and and those can sort of on their own Stand without the other services and still have use. Okay. So recor's been one that's attracted a lot of people that have these Non sort of cloud native kube type usage scenarios So there's many of them around recor one that's quite interesting recently is arch linux and looking at doing binary transparency And and their security team is starting to implement that and we've had people looking at um Uh, sort of firmware blobs or sort of drivers and stuff like that where those can be recorded in the transparency log and uh, other people interested in Signing documents using this as well, you know, so so it's uh, it's it's luckily a very customizable system recor You you get to choose what? Data sets you want to go in your you can design your own manifest essentially so we call it manifest agility So we always tried to make it so that We could support any type of schema that comes along that people want to use so a lot This was preemptive around the work that is happening around secure bill of materials s bombs and so forth So that we know we have the agility to to be able to work with different data types and different manifest types and so forth So yeah, there's a there's a lot of interest in People that are coming forward that are finding they could utilize the technology to solve a particular problem that they have within their particular vertical right right. It's actually immediately what comes to mind for me is one of the kind of You know blockchain is such a you know such a buzzword, right? But it does have a few things where I think there's a kind of real strong legitimately usage And it sounds like there might be a good overlap with this one, which is in particular like One of the problems people have is distribution of college transcripts So like if you want to go and apply to grad school, you need to get your transcript from you know, wherever you did your undergrad And prove, you know, you're you know what you did and one of the challenges making sure that that is you know If you want to do it digitally right is making that cryptographically secure And it sounds like this might actually be a really good solution for that kind of problem as well Yeah, very much. Yeah, we had some folks from a project called secure job Get in touch and secure drop is a kind of um It's kind of like a Dropbox for journalists and whistleblowers Oh, yeah, I need to share And I know other people were looking at utilizing transparency logs as a way to combat fake news Okay, so effectively what somebody does is they write something about I don't know xyz politician had an affair with a An alien. Okay. And then plausible That will kind of blow up on social media And then somebody will call them out the experts will round up on them and they'll just change the story. Oh, yeah, okay I've got that long Whereas we have something like a transparency log. You kind of it. You can read out effectively. Yeah So yeah, that's yeah, there's a lot of really interesting use cases around this technology definitely I expect to see a a whole lot of innovation over the next few years Yeah, yeah, no, it's uh, it is definitely an interesting problem. I mean, I know from even Like, you know, there's even something in hdp right that tells you when something was actually printed Um, you know, but everybody just manufactures that header, um, you know, based on whenever they generate, you know Sometimes they just generated or sometimes auto generated. Yeah, right And what I want to know a lot of the time is when did this actually come out right or when was it updated or whatever? and, uh, you know, even and not even from the, you know, the kind of You know changing the past kind of model as much as just I I'm interested for whatever reason in the history of this thing You know, and so that kind of data is really really important Yeah, I'd really I'd really like to see this leveraged for firmware transparency as well Oh, that would be great It's an interesting area because because I obviously work in the software brain And I I do I I'm very heavily focused on what can we do to improve software security? Okay, and then I look down and I go oh god, you know, it's just I don't you know There's that whole base the layer of in the hardware these firmware blobs where right, you know, it's it's just um Yeah, what we could do to to improve things there would be would have quite a magnitude definitely Yeah, one of the projects That was kind of initiated by and then worked on by some people in the boston redhead office and arcs Was one that I found yeah, interesting. So that was my team. Some of my team were working on. Yeah, that's what I was kind of wondering Yeah, like so like willy sterman and uh, I don't know is peter jones on your team Um But no, no, but he was one of the yeah, it was kind of a conversational star. Yeah, she's an engineer on my team. She's a great engineer Yeah, yeah. Um, yeah, uh, I what am I I'm trying to rope her in to talk on something, but I can't remember what exactly now Oh, yeah, you should do. Yeah, lily's right. Yeah, she's really good. Um So, uh, moving kind of a little bit to the side, uh, we know one of the things that we want to talk to you about Was the hacker one bug bounty program? um And so can you tell us a little bit about what what that is and why it's interesting? Yeah, very much. So just to kind of Do a one-on-one on what we do. So we're the our security response team and um If a vulnerability is discovered by security research or or program or or anybody, okay We like to run a responsible disclosure program So that means um the issue will be handled under embargo. Okay So that way we can make sure there's a fix in place uh, and and That vulnerability is not going to be in the wild for people to exploit while we Have a crazy headless rush to try and fix if you see So it's safer for everybody. It's safer for the users And uh, traditionally people would report for an email address that sends us an email. Okay, and then we would then Have our embargo program which we would kick off and handle friends from there and um This was uh, you know, there was a good volume of issues coming in but we realized that in a lot of ways People should really be rewarded for doing the right thing. So that was one of the main Drivers, okay, and it's just more eyes on the code as well. Okay. So, you know, there's this security Idiom that we have of the more eyes on the code the better the more secure the more people looking at the code So so yeah, we launched a hacker one bug bounty program Okay, and so how this operates is that hacker one have a portal where somebody discovers an issue about a security issue in kubernetes They can raise it with hacker one and hacker one then have a team that will do an initial triage of the issue try to establish if it's Actually a vulnerability or is it something that's previously already been reported and so forth and then if it is it's then Sort of escalated to to us folks in the security response team and I think there's about seven of us at the moment and it's a mix of red hat google amazon somebody from data dog and To give me I might have forgotten some of the others But it's a kind of a team where we we have like a rotor. So we go on call. So I've just come off rotor for last week and Yeah, it's uh, it's proven to be very effective the hacker one program. It means that we A lot of the noise around false positives is no longer taking up our time because we had that sort of first line support layer to to triage and filter And and those that do report something they get rewarded, you know, which is which is always good to see Yeah, totally. Um, yeah, I think uh They're kind of bounty programs in general that you know, they kind of been, you know They're slow to start right there, but maybe they're starting to actually kind of take But that's it's a really great way You know to kind of, you know, support open source, you know or support, you know, kind of change or whatever You know, because you know Some people have more money than they have time, um, you know And you know, especially, you know, if you have kids, for example, uh, I definitely You know often have more money than I have time and I'd like to see some changes But I can't do it myself. Um, but kind of in the reverse the you know getting the credit for discovering these things, you know, and You it's it's not just like money, right? It's also, you know, the the You know, Hootspot, right or the, you know Publicity or whatever around having discovered some of these things But you need you need some backing to kind of say, oh, yes, you really did discover a real Vulnerability, um, you know, it's not just me claiming on the internet that something happened And then as you say, right embargoes are really important. So one of the things that I thought was, um I think a lot of people who don't work in software companies in particular Um, especially even ones that are uh, kind of infrastructure layer software companies Like you don't realize how important those kind of security embargoes are We've talked about it, you know on the you know channel a few times, but basically it's like It was common, uh for me to be in a meeting or something like that or whatever and they'd be like, oh, you know, we We need to do this other release. Um, and somebody would say obviously, why do we have to do this other release? And they'd be like, oh, I can't tell you And and it's just par for the course like it's just you expect that to happen on occasion You know, you don't you don't really need to know you don't really want to know, uh, you know And it's just kind of like part of your culture And I think that's uh, that's something that people don't necessarily from the outside recognize or realize How important like not only it's kind of obvious. I think how important it is But it's I don't think it's as obvious how common and how Acceptable it is and so when you do a thing like a bug bounty program like this or whatever that it's real Like when you when you say it's embargoed or it's going to be a secret Until everyone's ready to release it That's a very very true thing in most in most organizations that I've worked with. Um And so that's uh, you know, it's something I think that needs we need to reassure You know people who don't have experience with it that this is true Because it's kind of uncommon, you know The right thing today. Yeah. Yeah All right, so moving on one of my more favorite type questions So from a security perspective, uh, what keeps you up at night about kubernetes? Hmm Yeah, that's a good question. So if I'm really honest, nothing keeps me up at night anymore Because It's um, if I do let it I'm just going to be a wreck. Do you see what I mean? Um, I guess Really the scary stuff are Container breakouts always a concern, you know, I think they may know from the main ones is um Where anybody can sort of access the host or attack adjacent tenants. I think those are the the big ones They're the scary ones essentially or Any sort of very nasty Sorry Privilege escalation, okay, which allows somebody to control a cluster You know, that's scary stuff where people can destroy pods or perhaps, you know Even more and I would say those are the two ones really and Yeah, it's it's interesting with security vulnerabilities you have the the magnitude of the The criticality of the vulnerability. That's one thing. Okay, and then there's the The marketing spin the kind of the the fud if you like You get around that vulnerability, you know, and they don't always marry up That's that's the thing you see. So for me, I think it would be like both of those on rocket fuel really a particularly nasty vulnerability That gets a lot of coverage as well. Perhaps, you know, some high-profile people or users of kubernetes Undergo some sort of significant nasty data leakage as a result of that. Do you see what I mean? So, yeah, I guess it would be An orchestration of things a very nasty high critical vulnerability that's found in the world and utilized in the world and and then Creates a media frenzy. Okay, and then there's a high-profile attack somebody actually Is successful with a significant harm because of that because of that breach that for me is the kind of the Yeah, that would that would keep me up at night. So the perfect storm keeps you up. Yes So I so yeah, I I mean, I think I I like that one But I also think that what six doors trying to do is one that particularly concerns me, which is the the kind of the solar winds a problem a little bit, um, you know, which is that An attack has been delivered and been successful and no one knows Um, and that in some ways for me, I think it's the scariest thing. Um, you know, where, you know, it's like The quote-unquote sleeper agent, right? Uh, where, you know, something something can happen eventually But that the you know, the infrastructure has already been, uh, you know, suborned or whatever Yes, to take it over and I think things like I said, like sigstore or whatever are or how you solve for that is that, you know You know, I've known I've known a few people who've done, uh, some of this kind of work in the past, right? And You know, it is it is part of an a successful attack To clean up after yourself, right? Um, and you know, and if you can modify those logs or whatever to show that you weren't there Then there wasn't, you know, was it, you know, does does anybody hear a tree falling in the woods, right? Um So I think that, you know models where You know audit logging or logging or whatever where you're trying to keep track of or have a way of knowing for sure Whether anything has been modified, um, you know, at least it limits the ability to The or limits the problem of an undiscovered attack or undiscovered control of the software system um So, yeah, that's my that's my big one. Uh, yeah, yeah very much. Yeah, and that's where, um The the transparency component of sigstore is Is so appealing Because we will run a pub public transparency log Okay, so that means that anybody can audit and monitor that log So they can check the integrity of that log to make sure we're doing the right thing Okay, and um, they can check the integrity of any particular entry in that log Okay, so then they can start to look for suspicious patterns things that are happening out of the normal if you see what I mean and so one of our our hopes and one of the One thing that we're really trying to encourage with sigstore is people to come along And innovate on top of our platform. So we have this transparency log. Okay, so We're we're talking to people about running these things called monitors Which will monitor the log. Okay, and we'll start to look for certain patterns and so forth. So that's where The recall the transparency log part is is very appealing because as you rightly say when there is an attack Okay, we're talking a key compromise Which is a particularly very nasty attack. You want to know the blast radius? Okay, so you want to know what else has been signed with this private key? Okay, and with Sigstore recall You can answer that you can you can perform an inclusion proof using the public key to find out is Exactly what artifacts have been signed and when time stands with that particular signature set. So Yeah, it's it's it's really sort of a nice Application of that technology, but in a lot of the ways it's it's not new It's sort of this is something that we borrowed from certificate transparency. Right. Do a similar thing. So traditionally a ca would Provide would sign a certificate for somebody. Okay, and you wouldn't really know what happened behind the closed doors of that ca Okay, you just have to Cross that they're doing the right thing. Okay, and then what happened was a couple of very high profile domains I think google and facebook where Somebody went to a ca said. I need a certificate for facebook.com And they got given one. Okay, and then you can imagine the amount of damage you can do if you are suddenly able to stand up A server Is okay. Okay. This is great facebook.com. If you see what I mean so this kind of Concerned and rightly scared a lot of people so they came up this idea of certificate transparency Which is a similar thing the certificate chain from the root ca And the certificate that signed for the particular domain is recorded into a transparency log And then what that allows you to do is for example red hat.com can then monitor the log It's been signed for the domain red hat.com. Okay now If they've recently procured a new certificate Meh, yeah, okay delete the email nothing to see there, right? If you hadn't It's like holy bleep You know somebody is Managed to get a domain certificate for our property to see what I mean and and so you get the same thing with Recall around secure supply chain transparency as well So one of the technologies that we're looking to utilize is open identity connect. Okay So what you'll be able to do is sign An artifact using an open identity provider Okay, so for example Google microsoft github. There's various people that provide these open identity provider solutions, okay And the great thing about that is you can then monitor the log for people signing things with your identity You're a bit like, you know, have I been pwned? Right, right. Yeah, right. Yeah. Yeah. Yeah. Yeah, and the great thing about an email some people will say Yeah, but an email it's not as secure as a You know a 5000 pound hsm Right. Well, yeah If you want to use that you can use that with sigstore But a majority of the open source projects your mum and pop small projects They can't afford specialist hardware and they don't like managing the keys So this gives them a system where they can sign something using their identity And they have protection of the crowd monitoring as well. Okay, and then the nice thing as well is These systems these these providers rather they they give you extra little security trinkets on top like two facts or authentication You know that you can use to protect your account You get that thing if if you log in on a new computer Then it will tell one of your trusted systems. Hey, did you just log in from this country with this ip address? So there's lots of nice secure existing security controls that we can leverage there And so that's why we look to use this open identity Connect and the email address And then we've got that same thing as certificate transparency we can monitor Who is using our identity to sign effectively? Yeah, it's uh, it's interesting. I mean, I don't know if either of you would recall There was a project from microsoft many many years ago called hailstorm And then a response to it. There was an open source project called The liberty alliance or open liberty alliance something like that Which is I think still around But the idea of it was kind of you would have it was it was like sso for the internet, right? And without all the problems of using facebook login right or using google login or even to these days Microsoft login Because it was a kind of a you know an independent service Even if it was owned by microsoft it was there was no way to get to your The rest of you, right? So I really want to see You know, this is one of the things that I think has been missing from the internet for a long time Right is that I have an identity on the internet, right that is that is protected by something That then I can use to manage with my twitter profile or my facebook account or my, you know Or my work account, right? You know, one of the things when I left for it had right one of the challenges I had was like Oh wait, I need to rejigger my entire life because there's a lot of things that I use both personally and professionally You know Identity is is fascinating. Okay. Yeah, I think about it a lot and a lot of people are a lot cleverer than me think about it a lot and it's always very difficult because You effectively You have two choices really, okay? One is tofu trust on first use So I just assume this is Langdon. Okay Looks and smells like him like set that key. I bring it into my key ring Okay, or like we do with an ssh server. Okay, you click yes to the fingerprint And it's ready to your allow hosts, okay But that's kind of that has its problems evidently, okay And then the the only other system is There's like a web of trust so us guys meet at the pub You know, we look at each other's passports. We sign each other's keys. That doesn't really scale That was the house problems and cove it disrupted that even more. Yeah. Yeah So the the third one is you need a kind of a trusted entity to a test To this is the You know, okay, and we try that with various our CA our certificate authority and it's and it's it's really difficult to sort of Get beyond these models right really isn't anything that is That's got traction that has solved this really. I really hope there is you see, you know I'd like for one day for this to all be disrupted. But in other ways, it's like the humble password You know people are coming up with biometrics and all sorts of systems to try and disrupt the password But nothing can quite cut it. Do you see what I mean, right? So identity is a really tricky one. You know, I would love to see some sort of Open decentralized identity system, but identity and decentralized a strange bed for those Exactly to connect. Do you see what I mean? And that that's the problem that the blockchain folks have a lot really is when they try to establish an Identity they typically have to leverage something off-chain I'm probably gonna upset some blockchain folks But it is it's just that it's the problem that we all have, you know, it's it's Yeah, I really look forward to the disruption happening there well and to your earlier point of You know kind of hitting the right place the right time and being consumable It's like we had a number of attempts, right where, you know, like we had some technology or we had some, you know Thing or whatever and it almost made it or whatever But it didn't quite it wasn't quite consumable enough wasn't quite at the right time or whatever So so I do I do also like I hope and I think it will be solved But you know, it's like even, you know, PGP signing emails, right? It's just you know, it just never took, you know Exactly. Yeah, and I mean, that's a really good that was one of our areas that we've managed to I think improve a lot really is Nothing wrong. I'm not talking about the the algorithms their strengths. Oh, yeah Or design or any of that, but it's the adoption at the end of the day. It's just Incredibly poor, you know, very few people sign stuff, you know, right? It's like I said for email PGP is wonderful. I can sign something. You know, it's coming from me. You have non-repudiation I need to send you something sensitive. I can encrypt it But if you grab a load of technical people at a conference and say, right, who's using PGP? Very few people And and it's the same with software signing as well project signing software So I think with sigstore, we've come up with this good balance between usability accessibility And good levels of security protections as well where we have that transparency and and that's that's a sort of I think that's the The thing that is really going to help Get our traction around adoption really it's not to say we're better than PGP or we're better than x or y or It's just uh, this might be more easily usable Well, and and you know, we're far enough along I think into a lot of encryption kind of solutions, you know of like signatures or You know, or actually, you know making things secure or whatever that at this point It's really about adoption And and the better is really about who's using the you know, the most of one thing, right? And that's To some extent, it's like we can go fix Any security flaws in a sense if we can drive adoption first, you know It's like if we discover that, you know, we're using, you know, whatever like ssh a few years ago, right switch from like 10 28s to 20 40 48 or 40 46. Yeah And um, you know, it's like that's that's fixable, you know, because ssh is completely prevalent You know, and so we need that we need the same kind of idea and then we can we can fix the We can fix the actual bugs, you know, uh once we Because right now nothing nothing is happening in a lot of places. No, I use just one example I really like rust. Okay, the rust programming language So rust has it's really interesting as a security person because you get this memory safety There's the compiler is very strict around ownership and so forth And that that fixes a lot of the the issues that we have in c and c plus plus effectively Okay But if you look at their packaging system Everything is pulled in untrusted So you have this kind of wonderful performance language with all these extra security guarantees, okay Around memory safety, but then there's again this spaghetti monster of dependencies coming right none of it None of it is verified or trusted. It's just pulled in, you know, and um So they're one of the communities that we're hoping to really sort of help them Come up with a sort of a sig store implementation that could help them really Right, right So I think with that and we are already over time But we should probably kind of wrap up there Is there any kind of closing anything you wanted to kind of add on I mean or you know, I think I I will definitely say go check out sig store And you know, if you if you want to help contribute that'd be awesome But you know at least start using it, right? We need to drive the adoption for things to get better Yeah, very much. Yeah. So just to tack on to the end of that Do come along, okay We you know, it's security, but we're a very friendly community We are really we're really welcome in here mentor new people that are interested that they need that you know We support people at all sorts of levels of Confidence in there, you know coding or documentation. So so do come along and get involved Awesome. Thanks so much for coming Yeah, and last of all, sorry, we will be at coupon. We're gonna have a booth. Oh cool. Okay, awesome. Yeah So you'll be able to come up And we'll show you how to sign your things Awesome. Yeah, awesome. All right. Thanks Luke. Thanks Langdon. Great And thank you everyone out there, please check out six store dev Figure out where you can implement it in your environment and what Luke know If if there's things you feel like you can improve upon right Thanks everybody. Have a good one. Stay safe out there. Take care. Bye. Bye