 Hello and welcome everybody to the CNCF Tough Maintainers panel today. I am joined by four fantastic Tough Maintainers here today and we're going to go through and introduce them here very shortly. We've got a lot of surface area to cover in this panel you're in for for quite a ride. So let's just go ahead and start with Marina and Marina the rules are when you introduce yourself you've got to hand it off to the next person who's going to introduce themselves. Hi everyone I'm Marina I'm a PhD student at NYU. I've been working on Tough for the past almost four years now doing research and development especially on kind of the academic specification and new features side of the project. And a fun fact about me is that I also like to dance and actually have a dance minor from when I was in college and I'll hand it on to Ustra. All right hi everyone my name is Ustra Ali and I'm a software engineer at Google on the Google open source security team. And I work on supply chain security and I've previously worked on cryptography and privacy preserving technologies. And a quick little side fact about me is when I'm not at my computer I'm usually outside hiking or doing some kind of combat sport training. And I will hand it off to Trishonk. Oh my god it might mean not to mess with you. So I'm Trishonk, I think you'll tell me sorry camera off but kind of midnight here I hope you'll forgive me. Yeah I've been on and off involved with Tough you know the Tough project several years now. I'm a security engineer at Datalog by the way. You see what did you go next? Sure I'm Ustra open source engineer VMware I've got a pretty long career in open source and now lately in supply chain security. I've got two border colleagues here with me and all of us are waiting for spring to finally arrive and helping get to at least the time of recording it's not happened yet. And I am Andrew Krug I'm a the lead security evangelist at Datalog. I work with Trishonk of course and happy to be here and moderating this panel today. So for the folks whose first time it is hearing about Tough I think it would just be great to cover like what what is Tough and like what are the goals of the project what problem does it seek to kind of solve here? Yes I can start with a little bit of background. So Tough started off as a an academic project at the University of Washington where my advisor Justin Capos was working with Justin Samuel and some folks from the tour project to kind of solve this problem of package manager security and they've been done some previous research on attacks on software update systems and so Tough was an attempt to address these attacks and create a compromise resilient software update framework that allows security even in the event that one or more signing keys or even the repository itself is compromised and I'll let let us go into more detail. Yeah so I'll just kind of go a little bit more into detail coming from my side of things so I got introduced to Tough as I started working on the six-door project which is a project under the open SSF that regards like software signing and we were building a service that required a lot of infrastructure components that you needed to trust and Tough was the like way that we started thinking about how do we keep that ecosystem of trust like compromise resilient like Marina mentioned and also like to be able to sort of like think about secure updates in the future so we really started integrating with Tough and adopting some of that framework methodology really from the start because we knew that we would eventually have to deal with things like having a trusted route having to update keys having to update parts of our ecosystem and having a sort of central repository where we were managing with Tough was the way to go. Yeah yeah I guess I guess that pretty much some said that basically signatures alone just they aren't very useful right like nobody is going to have hundreds of artifacts or something where you're going to somehow figure out which signature should be or who should be signing this you need to be able to you know somehow delegate the trust that that you can just check one or two things and then that kind of just flows from there and I think that like everyone who starts looking at this problem kind of ends up at this same-ish solution and you know doesn't have to be Tough but it's going to look quite a lot like it I think. I think one of the interesting things about Tough right is that it's actually a toolbox of tools and also it's a it's a framework on top of that and one of the problems that it seeks to solve is really being really really usable right it's it's not just like crypto for crypto folks one of my favorite quotes from Rob Fuller is you can't spell crypto without cry and I think that we can we can all use that to attribute why we haven't seen teams put more of these controls inside of their CI CD pipelines or or their software release process is that the the tools just really aren't understandable and accessible so does anybody have any thoughts on like how much time this project spends on on UX and usability oh what's the start boy that's that's a good question look so I think I think I think there's one thing that the OpenSSF Foundation has done well especially with Sixthorn right especially with Azra and friends is try to make this technology more accessible to the wall previously you pretty much had to have a PhD okay or close okay as Marina knows so yeah the technology wasn't very usable but look something like tough is actually much more useful not if you're an not if you're an individual contributor but something on the level of programming programming language uh yeah yeah exactly something like a programming language community repository like pipi or npm or you know github and so on and how do I put this initiative just wasn't there before but it's changing now you want to use stuff to to give you guarantees like okay look this is indeed the latest stuff that I'm getting I'm not being mixed and matched this is actually the Kubernetes projects key for example Nasra I can talk a lot more about this you know she been working about this so that's something that's been missing before before how do I put it before some reason uh what's the word for it high profile attacks people didn't really care about signing software right I think we're gonna all agree to this and you know with the Biden administration this has changed obviously but we still don't have good standardized tools and I think tough actually belongs in one of these tools but just wasn't too usable before but we're trying to change that sorry someone else should jump in here yeah I think a lot of the tough implementations were originally they came out of academia right so they were used as prusive concept to show how these things can work in code and I think we've done a lot of work in the past few years to kind of transition those from more kind of academic code bases to things that are a little bit easier to use in industry in practice and I know you see you can talk a lot about the work we've done specifically on Python tough our reference implementation to try and make it a lot more usable and just easy to plug it into new systems yeah yeah that's what we've been working on in Python tough especially but I think the same process is going on and go tough as well where we know we have something that that works for specific use cases but it maybe wasn't engineered with the with the sort of point of view that that maybe is it's good for security solutions but I will also say that well well it is true that this is like we're providing a framework or a set of tools it's I think we're also seeing the limits of that that when we're trying to look at the kind of generic problem that tough solves it's it's not quite easy enough especially for the for the repository case to kind of for someone to just take that and implement but if we talk about ease of use I think it's really useful to notice that what tough does for the client side is pretty magical and the fact that you get well you basically get no user experience because it just works and that is really something and it you know it it does make it worth it to have to work quite a bit more on the on the repository side yeah I don't know if there are a lot of differences in the the different implementations I guess we've all kind of approach this in similar ways so it's maybe some fairly low level libraries that aren't like full solutions and then command line tools no but you see you actually raise a good point which is that tough isn't just this framework this tool that you use but you kind of need to know how to use it right for your own use case that's the problem really I mean it's a strength and a weakness yeah yeah I think we start we've seen that from the six-door side as well where like we really do approach it like a toolbox which means that we do need to have some like context and understanding of how we're using it like you see said like the client side is fairly easy you know it just kind of works out of the box but when you're looking at the repo management side or when you're looking at okay what do I actually want to protect with tough then you start thinking of the context and you know I want to protect an ecosystem of infrastructure like six-door or I want to protect you know developer packages so there's all these different sort of use cases that when you apply to tough you realize that your management and your repo workflows end up different and I think that's part of the challenge that we're seeing when we look at like adoptions and integrations is that you really do have to sort of specify your use of tough within your integration or adoption or ecosystem yeah I think that's one of the challenges right with with creating a tool like this that we want to be flexible enough that it can apply to different languages and different use cases and I know that there's some really unconventional success stories of tough in the wild right like things that we would never think of but on the other side of that curve it's it's always a question of how do we make it flexible enough but also prescriptive enough that it's it's easily applicable yeah speaking of success stories like I would love for like marina to kind of jump in with the uptain story here like that's a really cool story of also seeing the whole feedback cycle of you know getting more taps and more enhancements into tough and having that life cycle complete yeah so uptain is the automotive variant of tough and it has its own name because it kind of uses pieces of tough that has kind of a tough inside of it but it also has a lot of other pieces built on top of that in order to work like directly in the automotive space which has a lot of unique challenges having to do with very small computers in vehicles and also a whole network of computers that only have one internet connection with which to to receive updates and so uptain I think was a really great project for the tough specification because we learned a lot about you know we got taps three and four so we got like a bunch of new features from this work with uptain that are now merged back into the tough specification and are kind of part of that toolbox for other projects if they have similar problems that they they'd like to address there and uptain has been very successful it's been standardized I think under IEEE ISTO and it's been you know we work a lot with the automotive community and we have integrations in various automobiles so that's been a very successful project any other particular questions about that well I think this demonstrates like the flexibility right we have everything from the obvious the the work that has happened on Pi Pi right maybe we want to talk just a little bit about more about that Python work as well after this but then just thinking of this inside of an automobile actually went to a presentation once where somebody was talking about running K3S inside of a car and it was you know this moment where I didn't know if I should be inspired or terrified but knowing that there's something like tough that we could put in that supply chain makes me more inspired than terrified exactly it's so important that we're able to update the software in car in cars especially because they're so safety critical that if there's any kinds of bugs you want to fix those right away and securely so it's kind of it's an exciting project to get to work on so is there something that we could bring back from from uptain or is it like the tweaks that they did is it really specific to that use case where where you want the manufacturer to control everything yeah so I think one of the big differences between uptain and these community repository like Pi Pi implementations is that everything is very much controlled by the automotive OEMs like the car makers they they can they decide per vehicle actually they the vehicle basically asks which updates should I install instead of in the typical software workflow where the computer says I want this package it's a little bit more directed in uptain and it goes very top down I think one of the big pieces from the uptain model that we can take back to tough is kind of improvements to compromise resilience one of the big things that we did for uptain was support multiple repositories so it allows updates I'm actually uptain actually downloads metadata from two different independent repositories and then compares those to each other before actually installing the update and this has to do with ensuring that you can have that kind of customizability with online keys in one repository and also use offline keys for greater security in another but this is a pretty flexible mechanism that can be used for just downloading from different packages from different repositories or just gaining that extra compromised resilience from multiple repository consensus on these different updates. If I could after that Marina I think in the future actually something like uptain is going to be important everywhere not just in the automotive context here's what I mean if you think about it it's uptain what it tries to solve is try to make as secure as possible cloud package dependency resolution right you have this robot sitting in the cloud being able to choose what software you get to install in your Tesla or whatever you like right okay and do it safely it can't just make stuff you know it can't just make up firmer images on its own it can choose firmer images but it's firmer images would basically develop by human beings and I think this is going to be a big problem in what we call community repositories know Marina yeah that makes sense I definitely think there are lessons from uptain that can that are more broadly applicable I think especially to the iot space but even just to yeah to community repositories yeah like jumping in on that whole like multi repository support um that's like exactly kind of what we have been thinking with this the six-door model so um like if you haven't kind of heard the six-door like you know mission statement I suppose it's to to sort of be like the let's encrypt of software signing and make it easier for developers to like create signing keys and uh you know improve their security posture through that um so one thing that we currently have like I mentioned before is like a six-door tough route that you know holds on to our six-door public infrastructure components um but alongside that like we also want to like extend that six-door route to be able to endorse you know your own public um tough routes so let's say you want to go endorse like you know your enterprise's recourse signing key or let's say you want to endorse some signing keys from like you know a large uh OSS project um what you can do is you can start you know creating multiple repositories and have uh like clients be able to say okay I want to pull from six-door route here and I also want to go reference um verification keys or other material in other repositories um so the six-door model of like you know we have a central repository but we also want to sort of distribute um signing capabilities to other people really falls into this whole multi repository situation. Six-door sounds like one of those things that like really really helps people get started right which I think is one of the challenges here but uh before we kind of go on to a variety of challenges with implementation here what I'd like to hear maybe from each of you what do you think is is challenging adoption of these technologies because I've heard it's in cars it's in pi pi why don't I just see this ubiquitously like npm uh for example if it's in pi pi it feels like it should be an npm not to not to pick on npm or anything but uh why do I just not see this everywhere yeah that is a really good question and I think you know speaking about this this software repository it's like npm I probably have um the most experience with uh looking at those problems with tough and I think the core issue was that um we like the the tough community maybe underestimated the the implementation complexity of a repository you know much like Azra has been talking about earlier um like just as an example like the repository only version of the python like pi pi repository which is um like this is a system where we would have tough running on pi pi where just the repository signs things on its own and developers don't have any kind of keys um just that proposal I think turns nine years this year so we've been looking at or people have been looking at this problem for nine years and it's still not running it's not pretty close there is a PR I couldn't stop laughing at that one yeah it's so true yeah yeah you I think you started it um so so obviously we underestimated the complexity of the of the community and you know lately I've been looking at the the real holy grail there which would be the developer signing which would kind of give us the situation where we have this um protected part from the developers that even like a repository uh compromise wouldn't you know that wouldn't compromise users machines which would be quite something so I've been looking at that and what it would mean for those um repositories and it is a far more complex thing so I think we need to rethink our approach here um yeah what you're really saying is that at a at any you know repository scale this really starts with like a threat model for the the workflow of the software itself like we've got to think at like a to b to c before we start to put the technology in yeah exactly like the workflows are going to be specific to the use cases we can't you know we can't go in well we've kind of tried that we've we've told them that this is the tough workflow and expected them to just implement it but that's not how it works um so yeah yeah I think you mentioned earlier it really does it makes the client workflow like once it's implemented the client doesn't even see it right and so the flip side of that is that there's a lot of work that happens on the repository to make that happen and the repository maintainers of these of like PyPI and others they're already very busy people and so it's tricky to figure out how to make this as simple as possible for them while still getting these the security properties of tough that that are really valuable so that's kind of that's what we've been trying to solve yeah and in addition the repositories really really do not want to experiment for good reasons I mean they're serving a hundred million packages per day or something they don't want you know new things to you know try something out they want something that's been proven to work and maybe we haven't quite quite offered that yeah it seems to be like the more complex like your repository management situation is like the exponentially harder the tough situation gets um with like our six-store case it's like it was fairly easy to kind of bootstrap a minimal uh tough roots uh key signing ceremony and really kick off our tough route but the the reason why is really because we have a very minimal tough route we're only fetching it like clients are only fetching a minimal number of targets and we only have really five or six signers involved in the whole process so we're looking at like a really nicely well-scoped area but even then we had in outage maybe like two weeks ago and that that's very stressful because like if your tough you know system goes down then like you know your entire client workflow becomes everything yeah um and so like little problems like that end up happening in repo management things like race conditions started coming up like when you actually pushed your repository do you need to think about race condition so um you know it it's turned out that like trying to expand the tough route in six-store to cover things like you know multi repositories and delegations and things like that that's becoming a a more difficult problem because the repo management side is is fairly complex yeah i think that's a that's a good way to put it a lot of these issues are shared but they just become so much more pressing if you have a hundred thousand people signing things really interesting engineering problems right because i think as a community we can all agree that we need this at the repository level but it's more of a question of how do we get how do we get there and it's it's about risk management to some degree not just security risk management but operational risk management so this will be a great story to tell in a couple of years i'm sure about how we made this map the way that we think about software in these repositories so just just kind of moving right along here um how how do you handle cvs between implementations or or like plug bounty type scenarios in the tough framework yeah this has been a really interesting problem because the tough project is made up of a couple of different components there's a specification and then various implementations in different languages um so far we have um not have any cvs in the specification itself imagine that follows a similar process but we have had cvs in various implementations and most notably we've had a few in the python reference implementation which kind of by the name right it's the reference implementation and so a lot of the other implementations have like been inspired by this first implementation and so even if it's not a specification bug specifically it is a bug that has you know spidered its way through the ecosystem and so we wouldn't want to just really you know release a patch in one project without communicating properly with everybody else and so that's definitely been a a challenging thing to handle and just you know i have this like list of email addresses for like you know i think it's six or seven different open source projects that have to be communicated with before we actually really cvs just to make sure that we're all on the same page and nobody's um code can be exploited in the meantime i think i think usually though is it is it fair to say that the bugs have been fairly specific to implementations and not the framework in general yes that's definitely true um they they they yeah they're implementation specific and usually it's not everyone that's affected but it's a couple implementations that made the same mistake which usually means we should clarify something which we have definitely done but it's not they're not technically specification bugs right so so maybe here's the lesson right which is that look security is hard enough to get it right so i don't want to okay look let me let me just say it don't reinvent off okay you could try to do it you're gonna mess it up because believe me we have okay so yeah yeah i heard i heard a couple of things there that i just want to kind of unpack a little bit i heard disclosure is really hard like in in an open ecosystem like this and of course like patching is really hard in open ecosystem so i would just kind of ask the question to all of you like when you thought about this like the cve process for tough or how you're going to handle issues like this do you take any learnings from anything like the the kubernetes project which which also like kind of has the exact same problem right when they get a critical vulnerability they have to kind of secretly work a patch in before they make the public release and there's all these stakeholders that need to be notified so what was your kind of process there yeah i think it's a pretty standard process based on the idea of we have you know it's still an open source project but we have a temporary little closed section of it that we make to handle these vulnerabilities and invite in maintainers of any projects that you don't need to see it or you know are relevant for the for the disclosure um but obviously like drawing the line there like who needs to see it is always a tricky question so i know marina has done in previous bugs you've done good work on trying to figure out like which projects we really need to get in the same room yeah i think one thing that we've started to also realize is the importance of like cross compatibility and like interoperability like that's one learning i think as we see like more adoptions the really nice upshot of like having people use things is that like you realize you need more robust and hardened like procedures in your underlying libraries and so we've seen like proposals now to like think about interoperability and that's also where a lot of the bugs have been coming from in some of the implementation so especially like from my side six store is like go base so we use a go top implementation um that's on the update frameworks um organization on github and we've realized that there's um interoperability bugs between like rust clients and python clients which is like especially pertinent in a global ecosystem like six store um so we've realized that like yes we kind of need to start investing some effort into making sure that all these implementations are not only like bug free but also like on the same page which would help with uh things like you know CVEs in reference implementations trickling down um so that that's one thing that like I think one learning in the past like year so that we've realized that this really needs to be like a p1 p0 yeah I think um you might be like the first first implementation running into actual problems with this so far possibly everyone's been kind of working on a language specific thing and maybe even thinking that it's not important like this is just an implementations for you know pipey eyes so obviously it's everyone's running the same client that's implemented in python and that's it but you've shown that of course it's not true yeah that's a really good point like a lot of clients are sort of siloed into their own ecosystem so it like really interoperability really maybe hasn't been relevant but like as we've seen like in this you know people wanting to adopt six store tools in a variety of different ecosystems means that six stores tools which means toughs tools have to be interoperable um so it's kind of this trickle down um requirement I think it's very obviously it's go ahead oh I was just going to say I think it's very cool that you're thinking about interoperability at the outset right and I think that that is a great highlight for just how far we've come as an industry that we're not kind of like backing our way into interoperability it's like a design tenant at the outset and and that's very cool and I don't know that we see that a hundred percent of the time yeah it's an especially tough problem just because of uh how many people need to be on the same page with these sorts of things like with um you know like coordinating sub-disclosures it's like trying to get all the projects especially open source projects like in the in the same like you know synced up uh version of things is very difficult um I think we all know that from experience I uh I'll just pause here and just acknowledge that you said it was a tough problem I know I know I was just gonna point it out too like it's brilliant I make this joke all the time in your in your sync meetings yeah so I think the upshots here is that there there really is like a lot of continuous improvement in the project and really kind of incorporating learnings and trying to to engineer ahead of of the problems that you're anticipating which is it's a massive change in the way that we think about and build software you know as a as an industry so what if if if I could add to that um Andrew I think I think the genius behind a project like six-door is to try to make this accessible to everybody previously before this you really needed to um now you can use stuff in a way you don't even really to understand how it works right someone like Kubernetes can come along and use the keys and don't worry about all that stuff behind the scenes right thanks to um all this great work by by asra and brina and uc and friends and so on but before this you really needed to do a lot of the work on your own and that's I think is what's the word for it a bit of a bit of a feeling of the project right we could have done this better yeah I think we're starting to see as well that like um the more adoptions we have the more you know we by force have to improve those those like you know APIs and uh some man-line tools that we're exporting so I think I think one really nice thing is that uh especially the more use cases you have the more you're kind of informed on like what the best and most scalable um solutions are uh and yeah it's it's it's really interesting to sort of see the evolvement of projects even in like the past year um with python top and go tough and rust clients like they're all really improving the more usage we have yeah great so what's uh what's next for the project like what's the the number one thing that you're all focused on right now that's unsolved I don't know if we're all focused on it but I think for me personally it's definitely the the repository side that we've already you know touched on several times that implementing a specific repository is just way too complex there is a like a like a level missing we've got a good good base and in the tough implementation and then we've got the the user project somewhere high above and there is just things missing here in between um that's that's definitely something that where I kind of feel that like in the tough specification the client workflow is quite well defined and it's it's easy to read and kind of grasp what it does and we kind of need good repository workflows for you know in the same way you know described and this is going to happen in this situation um but the problem there is that these different repository types that we've been talking about they they have different workflows it's it's very clear and different needs and I'm sure that's the reason reason why by the specification doesn't really talk about any of that so sorry go ahead yeah yeah just that that's what we need a good solution for or more likely we need multiple different solutions for these different use cases do you know you think yeah oh sorry it's just somewhat similar I um I think what I've been really focusing on is figuring out from these conversations with the doctors and with different implementations um which of these problems are new features that we should be adding to tough and which of these new features should be prioritizing to really help people make this process easier and and um and make that move along and then coming from my side too that repo management is is again the biggest key here um so we're trying to get basically like smaller projects and developers more interested and able to create their own uh tough um repository so again this comes down to how do we give them the tools and like package them the six store away um for them to be able to do some of this um and so like I'm in particular kind of looking at uh ways to leverage github workflows um since most OSS developers are probably familiar with github um and I know Joshua Locke as well has been looking at that sort of thing of how do you do repo management on github workflows um so I think honestly that that's like the the next easy step to do um to get developers like you know more quickly on board with that yeah so so I know I know what my next obsession in life is right it's um look um so I really like this new open ssf project it's called oh I think yeah what it's called exactly something like protecting the you know open source software repositories okay marina and I like to call it um community repositories so my mission in life is to see something like pipi sometimes npm and github ruby gems you name it protect it on the dock invisibly and you don't even know it right and we've had ideas we know how uh securing software repositories okay thanks marina yeah exactly that's why so so that's my mission in life is to try to help this project coming along they're using excellent technologies like poolgear okay so here's what I want to see in the future right okay is developers open source developers using throw away uh poolgear keys which is uh this technology that you have in sickstore um and you use stuff to securely distribute that it is you who's supposed to be using this key right now we don't quite have the technology just yet so we're not living in a world yet where we can download of you know pip install jango for example and get all the dependencies securely we just don't live in that world yet um I would like to get us to a stage you know everyone here in the room of course um get to a world where we use stuff to securely um get your dependencies the latest version of them make sure you've not been mixed in it's a particular attack that I don't really want to go and do right now but also more importantly how do you know that you're the one who's supposed to sign for this project how do I know that you're the jango developer right this is a big problem that the world is not solving right now we can we can we can sign things like you see is saying just because it's a signature why should I trust you so so we are just about out of time uh for today's panel um but I just want to make sure and recap you know tough is a it's a toolbox and a framework right and I put the url for a tough project on screen but I know that uh you all have a little bit more advice on on how folks might get involved with these projects we can throw some of those URLs up on screen as well if you have like just a couple of pieces uh that that you want to recommend for for kind of getting started