 We're in the usual warm-up period. Any thing you have to add to the minutes now is a good moment to add it. If you want to add yourself to the list of attendees, that would be a great time as well. Again, we'll be starting in a moment. Please go to the meeting minutes and add your name and any meeting comments, any meeting points you would like to make as well and we'll get through those as we get. I'll just paste the link in chat. OK, and we're off. So I trust I was out of the country. I am out of the country, in fact. So it made it a little difficult for me to attend some of the KubeCon stuff that was going on last week. But just as a matter of interest for those of you who were there or who were paying attention online, any comments that you want to raise on that, any interesting feedback, any thoughts? Well, we missed having you there. A lot of the stuff had gone virtual, so there were definitely a lot of people missing. But the big thing, at least from my vantage point, seemed to be a focus on it'd be a focus on supply chain defense. And I think that this is something that we will be able to capitalize on long run, despite the fact that it's not the main focus of this working group. But some of the things that go on through that may end up gaining usage in our little area of the of the world over time. Yes, I agree. I mean, the work I've been doing, you know, within Cisco, oddly, for the last couple of weeks has been supply chain defense work as well, or discussions, you know, how you ensure that the thing you're running is the thing that your developer produced for you and that it didn't get tampered with on the way. And I think, actually, we mentioned that in one of the sessions that we had. I mean, there would seem to be no consolidation and two standards available for how it's done. And I think it depends on what you consider to be end to end security, as in when is the very last moment at which you can check the contents of a container image to ensure that it's what you intended to run? Or I would guess a helm chart or anything else that's involved in making these things work. Yeah, and this definitely going to get broken up into multiple problems. The very first problem is going to be how to define what's in your what's in your artifacts and to do this recursively with the vendors you bring in as well. So you as your, not you as a human, but you as the company produce a artifact. And then you can say, well, these are things that we brought in and here's the evidence of what they brought in and so on until you've hit some bottom turtle somewhere that represents the actual leafs of the graph. And I think part of it is a lot of people will rightly so point out that, oh, this doesn't solve the supply chain security and they're right. But it is a component that can lead into building a strategy to help raise the bar and the cost of performing an attack. And so over time, these particular artifacts would feed into more dynamic systems, like things to work out like what's currently running in your infrastructure as opposed to where was this thing built? And should I should I gatekeep on it based upon a CVE database that eventually gets populated so that you can gather what your what your risk is over time and try to work out how to prioritize your resources to make sure that you upgrade things over in an efficient way. So I don't think it'll solve all the problems up and down the board. But if it helps people better, you make use of their resources in that process, it certainly does move the move the needle. Well, I have one question, an audience participation question, I think. Between the people who are, you know, developing a CNF and putting it together and the components they use within that CNF, and people are running the CNF. Whose responsibility is it to make sure that CNF is built of suitably secure, reliable components? Yeah, and that's a good question. I don't think there's going to be a single answer for that. I think that we're going to have there's a couple things to look at because one of them is I may have a set of artifacts and I may say I brought this thing in. But how do you know that I follow the process? And that I haven't injected something in the middle of that process through a compiler that's been infected, or, or someone just attests through a key that's been stolen. So there's there are definitely gaps that are there. There are some things that you can bring together to try to help towards this, like, you may say, oh, we're going to use a repeatable, a repeatable build. And we're going to repeat that build across multiple systems that have different owners. And you may so you could do something like that. Of course, that drives the cost up. So it depends entirely on how much you're you're willing to do towards making that and meet and how much people are willing to spend. Because every control you bring in has a has a cost. You can look at things like the recent executive order of the US administration about software bill of materials, where it's mandated that every software supplier will provide a software bill of materials with all those dependencies of the software when they provide, I think it's currently addresses suppliers of software to the federal government. But I think that can be adopted widely in the industry. So I think there's some responsibility of the software vendor, but probably it won't end there. And that would be just one step of of securing the end to end supply chain. Yeah, absolutely. And the the executive order also calls out critical infrastructure. So if you're running a power plant or something, or the question becomes what is critical infrastructure? The question that you do not know the answer to is the is is service providers are service providers going to be considered critical infrastructure in a cell? When are they going to be considered critical infrastructure? Because they may expand it over time as well. And if it is, at least in the United States, then the software bill of materials will be a necessary deliverable for for them to accept the the software. So the problem with that is that I mean, it depends who you're working with. If you are using an open source CNF, then in all light, you have to build it and the CNF bill of materials is under your control or alternatively, if you're not building it, if you're pulling a bill item off of the internet, then the supply chain security is completely out of your control, you've literally no idea what anyone built it from. It's impossible toward it. If you're using a closed source CNF, which almost certainly is built from multiple components, is your supplier necessarily going to want to tell you every single version, and every single patch of every single thing that they are using to build that? Because there's a certain quantity of intellectual property in that along with a lot of actual hard work to make it possible as well, to be honest. So is it appropriate in all circumstances or is it actually making more work and more difficulty than it would otherwise be to basically say, it is your job to ensure your supply chain security and via contractual terms. We indemnify ourselves against your flaws. I think the recent executive order doesn't leave it as an honor system, but mandates the vendors to provide this software bill of materials. So it's not up to their discretion whether they want to disclose it or not. I mean, of course, as you said, and it's not their interest, and they might be reluctant to do so. But I think this executive order exactly enforces that. It makes them provide this bill of material. I missed the context of the executive order, sorry. Which executive order? So the executive order is, I don't know exactly what's the scope of it, what software it covers, but it mandates that software vendors provide a software bill of material with the software they provide. I think it may be limited right now to critical infrastructure, but it mandates software vendors to accompany their provided software with the software bill of material indicating all the components included in this software product. Yeah, the first turn of the crank is if the government buys your software, you must provide a software bill of materials. And my understanding is they're putting that as a full stop measure. So it will be a hard requirement if we're at least selling it to the US government. But yeah, I mean, specifically for selling into the US government to be clear, right? That's not that that's US specific. And it's US government specific. It's not the market that we're dealing with. The second could be adopted by the market we're dealing with. But again, the question I'm asking myself here is, will that stop people providing software under those terms? Would they simply refuse to do it? Or would they be, you know, their ability to provide good software or innovative software be stifled by doing it? I don't know the answers to these. And you might say that I've got a stake in this that I don't want to be providing this stuff. And it's true I don't. But that's a personal thing. It just seems like a lot of work for actually not a great deal of value. But, you know, it's I don't know what the right answer is, which gets the best result for all of us. That's my question. Maybe I can add a little bit on that one in general. Hi, by the way, CJ joined this meeting for the first time, but have a background, a lot of telecom work with some of the larger vendors in that topic, as well as a lot of the open source stuff. You know, one of the things that I've noticed when talking to some of these kind of larger vendors is that what they are really pushing right now as a competitive advantage is this controlled supply chain because they have no interest at all in showing their source code, which means that they've added a lot of work from the black duck to the FOS ID to all these kind of source code scanning and, you know, safe repositories and a lot of process would be able to do that one. And that will be a lot of their competitive advantage when it comes to selling these, let's call it black box CNFs, which have some specific connections up to the management system so that they, through that system, can validate that that happens. For the open source or the more mixed ones, exactly as you all been saying now, it's going to be really tricky to do a similar type of source code scanning or secured repositories that you have to check into, especially if there are projects that are using, you know, tons of different supply chain pieces, like we can expect. So my view on this would probably be that, you know, you have to have a way of securing an open source supply chain. But in the proprietary supply chain, you know, this has been something that had been worked on for five or six or seven years for competitive reasons to push this thing in to make it as we're feeling right now in a very complex position of how it needs to be done. Does this make sense? It sort of does, but I mean the thing that I come back to is, okay, so I found, well, I can think of two problems here. One is that the idea that there is a single best version of some component, right, if I take a random source component off the shelf, say lib, there will be, you know, a glibc that exists in the upstream repository. There will be a version that red hat ships, which will be older than that and will be patched up to hell because that's what they do. There will be a version that canonical ship that is older than that and patched up to hell because that's what they do. So I'm not entirely sure that I see that there is a clear end to that supply chain, the source in this instance. The other problem I can see is that actually, you know, as a company trying to deliver something that's in this instance, we're looking at security but reliability comes into this as well. Let's go with security for a second. As an operator of CNFs, I'm actually not as interested in where is this software coming from as who stands behind it, who is the responsible party if it turns out that there is a problem in this and I might be able to get a jump on my supply as being terrible responsible parties if I am auditing the components of their software but I'm not going to go back to the GNU project for Glibc and say, well, you know, you broke my application and I lost millions of dollars and now I'm going to sue you. That isn't what's going to happen here. So I mean in the sense of this being a potential best practice, I think the question is what could you possibly get out of this? What's the what's the maximal value you can get out of having a well-traced supply chain? Yeah, but do you want to be aware of sorry if I go ahead? Yeah, I'll tag on something real quick on that and then please continue. There is a they do differentiate in the standard ones that are catching on. They do differentiate between the creator and the supplier and it does provide for path to provide that. So you might say the creator is GNU. The supplier is Red Hat and Red Hat is the one responsible for the contents of this. So there is something for that. Yeah, what I wanted to say is that even I mean if there is a vulnerability discovered in Glibc, don't you want to know if you're affected or not as the operator? On the other hand, if I'm trusting the vendor to tell me what Glibc they're using and be right and why am I not trusting the vendor to tell me that they're not affected? It's a good question but I think up until now it was not much of a motivation for the vendor to constantly check for all these vulnerabilities. If it's now out there in the S-bomb and they know they're under scrutiny, they have more incentive maybe to make sure they're covered. And that's one of those vendors. I have to say that you know my company's finances are on the line if something like that happened. So I mean I'm not in terms of future sales but in terms of typically current liabilities but it depends. I don't know the answer to this. I'm not arguing a corner here. I'm trying to get to the bottom of the question to be clear. Yeah, I'm not saying there was malice from the side of the vendor. I've been working for vendors for 30 years now but there is never to my knowledge any malice or active hiding of vulnerabilities. But maybe having an S-bomb there and having both the vendor and their customers be aware of that will make, you know, drive both parties to be more aware of vulnerabilities earlier and that's all. It's not like you know holding someone accountable to something they weren't accountable for before but just providing a tool and a common language to have that discussion between the supplier and the vendor. Yeah and I think part of it is where they're trying to shift responsibilities. Like before responsibility was entirely on the vendors and after getting burned by various attacks like solar winds, the one that attacks solar winds, the government I think is trying to push some of the responsibility. Like if there's a problem and then you should own it. Step one of ownership is like let's go trust the vendors and tell them hey you need to do something about it. If that fails you still own the problem so let's move it up the stack and get more more control. So I think it's I think part of this is a reaction to the various attacks where a company like Cisco may be an extremely good actor in this space but because there's a large number of vendors out there who may not be as mature in their processes they need a way to force that maturity and part of it is not just about security as in where does it come from. If you cannot produce an S-bomb that could be an indicator that from a process perspective you're you're not mature and you actually don't know where your software comes from internally. So what if I produce an S-bomb and it's not right. Absolutely trust and verify right. You can't you know if somebody's given me an S-bomb then it's effectively me you know making sure I can verify their claims that they're auditing for security problems in potential source components and interestingly we've moved a little way away from supply chain security here but still but I mean if they give me an S-bomb and that S-bomb is a lie or a fiction or out of date or simply made by tools that get it wrong then I don't know that I necessarily have a means of validating that the bomb that they give me is the bomb as it truly is in practice. Yeah and the issue of getting it right if it's a mistake like you're out you produce this output that's just wrong. I mean that's one thing. If it's somebody who's lying purposefully on that then they risk being caught and then it's it's literally illegal and they can then add on the appropriate consequences for that but that is definitely a major risk. Yeah that's definitely a major risk is like from an S-bomb perspective what if the tools don't produce the right output or the person doesn't figure it in the right way to do so. So a large part of this I think is going to be hinged on how easy is it to use the tools and how mature do those tools become over time and it's definitely it is definitely a major risk especially now in the earlier days. And doesn't this go down almost into the development tool chain that you know let's say there is the vulnerability somewhere in the code you know there are two ways that a company can do that either they are extremely strict with what code they let in and you know they only use their own code or they have this kind of qualification process that gives you six months old libraries because that was when the last library was kind of checked and then they can kind of build that application in their lab in a day and they're absolutely sure that there are no vulnerabilities and that it works or you do the other way that you let all these developers bring in their all all the cool code bring something that works and then you start with the source code scanning the application monitoring you're basically running it in test environment that customer into that process takes a month before you can certify it and say yeah this thing actually works and i'm guessing that that second option that a lot of the larger renders have been doing doesn't really work well when you're having thousands of different components that can go wrong in various times but the old the other option of actually securing each and every component that goes into that supply chain is also quite complex because then you're stuck in the fact that you're actually you know using old components with bugs and security fixes that might have been fixed in upstream versions a lot faster right these things are all potentially true and i think the other thing that's always missed is that just because there's a cv on glibc as an example again right if it's in a function i don't call then it's irrelevant um so um security problems with components built into a system do not necessarily respect security problems with the system they are a question mark they are not absolutely so what you're almost saying is that we should segment the type of cnfs and the type of functions if there are communication functions to the rest of the world or through other processes where if they contain these let's call it larger attack surfaces while other cnfs might be you know less valuable in that regard yeah and part of it as well yeah and part of it is also determining where to spend your resources like you're running a an infosec team you have limited resources limited time to do any given tasks so if this gives you a road map of being able to ask the vendors hey can you validate this verify it for me uh versus uh not knowing where to where to spend some of some of your some of your time and also to also look at the the updates like someone comes over and says we we made a patch with uh with glibc and this has gone into our systems then it also gives an opportunity to put out bug bounties for those things and say hey we passed this particular thing or we want to prove that this thing is not vulnerable to the cv e because we believe he's been cauterized uh if you can find a path that uses that particular code path or use exercises that feature then you get a much larger bounty than the normal bounty so i think that there's some interesting economic advantages that we can that we can gain here that are beyond just does this thing make it more more secure? Yeah but coming back to our job if we were recommending in this instance perhaps an interaction between the way that a developer chooses and incorporates components and the way that an operator needs to assure that the running system you know the the actual cnf they've been given is secure what best practice is and again they don't have to be perfect and they don't have to be comprehensive they just have to be things that we can write down what best practices could we possibly write down that we would all agree are as good as it's getting at this point in time. So um i've spent a little bit of time looking at some of these things i can write up a an initial draft of what such a best practice would look like and in the beginning it'll primarily be use one of the tools that analyzes the incoming packages the incoming artifacts generates an spdx or cyclone dx either is fine and uh and signs it if you're in the open source world use something like six store you can use that with post source as well but but basically you have something that is assigned at a point in time that you can then provide to other that you can then provide downstream and just as a first turn of the crank other turns over time will become more ideally more easy to produce something but also more rich in in the information that it provides but just as a first turn of the crank and this is actually what they did in kubernetes so kubernetes now has is now producing spdx as part of their bills and those i don't know if they're being signed in six store yet if it's not my my understanding is that's the goal and uh i know the people who wrote back code as well so i'm happy to go ping them and see if we can get them to come in and give some advice and i just wanted to add one thing to what you said is about the artifact i think there is various types of artifact and you might have a different way to scan our basically review the development abilities in each of them depending on the type yeah what's interesting with this we don't want the vulnerability information directly in the in the the sbomb we want it to be something that can or the sbomb can act as a key to those vulnerabilities and so as because the vulnerabilities over time they they change they tend they tend to grow so the sbomb is designed to be a static elements that you build you sign and it never changes from from then on Ian if you can hear us we saw the message on the chat yeah it's back again sorry about that it's it's going to be intermittent because i am in the middle of nowhere in the uk and it leaves you with rural internet problems sorry about that um yeah um i trust you're riffing on the concept but okay frederick you've just signed up for this part of things what we could do in terms of bill of materials of software built into a cnf as i say i think you know it becomes a broader topic of working out whether the software in question and any given category of vulnerability on that software actually makes the cnf vulnerable in turn which is ways are given the supply chain stuff back to the you know the build product making its way from whoever developed it to production running code without being maliciously modified or its configuration being maliciously modified those also exist and i think we mentioned earlier that yes there are a couple of signing solutions out there one thing i have as a question at the moment is whether any of those signing solutions or perhaps how far those signing solutions go to end-to-end security because again if i take a container and i store its table on my you know target node and that table is changed at some point in time by whoever then when it's unpacked and it's run then it could it's been tampered with it's no longer what i intend it to be so there are questions there about how far from an end-to-end perspective you can take that and obviously the start is the same question right if i'm building something then it the signature of a container is typically calculated by my build system but what goes into that build before it's signed is you know potentially alterable so same kind of thing with that security it's almost the question of how do i ensure that the bomb that we're talking about here actually is what's reflected by a signed component again more questions than answers in my head at this point in time yeah and these also these are some of the conversations that we're going to have in time like you could have a there has to be somewhere where some entity is performing that attestation and absorbing the risk and saying like i as company x declare that this thing is is accurate to the best of our knowledge and that they followed some process and they may even be required to provide in the long run some process information on how something was built which they could do with something like intoto which basically says I followed this process as I am this role and I follow this process and you can define the various roles that led up to it but the end result is that there has to be something there that you can that you can anchor to and if it's open source you're pulling in that that entity is absorbing the risk of having selected those open source tools and ensuring that the source coming in is to the best of their knowledge low risk to the customers part of doing that as well there's another risk that we need to be careful with and I want to flag this it's we want to make one of the things and I'm going to be going around the circuit talking about the specific the specific thing we want to make sure that the entity that is considered to be responsible or assigning for it is a is a company and not necessarily the name of a human and so the company may maintain an internal list of names but we don't want the legal name to be part of of the humans to be within the SPOM itself and I think your point would be that it's a responsible party I mean someone who's exactly responsibility whether that is a company or a human the fact is that you know exactly you know who's next to right but but the reason we don't want it to be explicitly a human is that that causes a risk for a nation to say factors to to target like what humans do I need to go shake down in order to make a change so if it's like if it's a large vendor or a vendor of a company you don't have that roadmap but you trust the company maintains that that's going to be hard with curl since I know Don is the amber that does that little module yeah but even with curl like you have the individual who makes it when red hat pulls it in they have people who are responsible for ensuring that the software that comes into the platform is yeah yeah separate the two things right it's not who does the work it's who's standing up to say that this works on paying of you know me losing money and paying it to you this works that's what it always comes down to that's what responsibility is that the consequences land somewhere they don't just fly around and ultimately if you haven't it's a form of insurance in a sense if I'm running a cnf and it uses curl and I don't check curl and I get it from a vendor who makes no promises about the quality of it or I just pulled it off the internet then when it breaks then all of the consequences land on me I bought that for myself if you think that that is not what you're doing if you think you've got a vendor on the hook you're saying that you need someone to effectively find in blood that they're on the hook saying when you want to see a responsible name in the bottom and then what you're saying in addition is you would normally find that as a company not a person yeah so it's the fixing liability not the you know economical liability for the damage it's both normally I mean I want to know the liability for it not working is not for us to choose or at least we could to be fair make recommendations because that's precisely what we do but it isn't necessarily for us to make the final choice if somebody says well it's all right as long as it's to fix it within five days then yeah that's great if that's what you choose then you know knock yourself out there are plenty of people running you know production networks who would say if my network goes down for more than 10 minutes then I will be charging you a million dollars an hour or whatever but again the the actual nature of the contract is fine and good it's really just a matter of saying I got this from you and I can prove that I am running exactly the thing you shipped which means that you are in fact liable for all of its failings and then you have those you know 10 or 20 larger companies that are the only ones that can play in that kind of game but it is you know pay a million when something goes wrong which of course the un-customers at the operators yeah again they don't want to pay that extra penalty but they still want to have that flexibility who needs to take that risk right and and this is the nature of things like clouds for instance that you can build a a reliable network from unreliable components or in a reliable piece of whatever you call a cloud from unreliable components I don't necessarily guarantee that the cloud the individual servers work perfectly and forever I guarantee that the cloud works perfectly forever when built from those less reliable components so I mean a an operator of software or anything else is effectively not saying that the software that they've been given in much the same way as routers die eventually they're not saying that the components they've been given are perfectly reliable and will live forever and will work perfectly they're saying that they can deal with whatever consequences are left after the guarantees they've been given exactly and if they have the source code worst case and if you're going if you're going with yeah so so basically we're getting even more into a kind of supply chain risk management thinking here where we should basically say you know five or six different categories for components do they talk with the outside world you know do they have more than than this many type of libraries do they have you know components connected to sign on authorization so we can basically create a bit of a you know risk management matrix based on a few things that you know do you have an impact if they are affecting other components in that secure chain of insecure components yeah it comes down to I would take this from there is a big picture to solve and that big picture is going to take some solving in terms of all right when I bring all of these exciting components together then where do the risks get the biggest risks get introduced and how do I mitigate them but we can also take it from the other way around right we said that it is useful to have a manifest for a cnf even if it all it says is hello I am cnf version 145 but it might also say and I'm built of these things and it is important that that manifest ties back with a signature because it's the only secure way of doing this to someone who owns the thing the cnf and some degree of responsibility on the cnf so we know that manifest is signed and we know that the thing that the manifest refers to is also signed to avoid tampering and change we could probably put best practices together around that which is you know there will be a manifest file it will contain at least and it may also contain this sort of thing I would you know I don't think you can solve the world problems I think if you could then we've effectively solved problems and a lot of other industries besides our own but I think you could at least nibble away at the corners right there needs to be a manifest it will contain these things we will pick a random format and we will call this the best practice to the time being and it gives us a place to improve upon when we think of other things we want to add so a bit of 80-20 on it the basic saying that we cannot secure everything since we are a communication setup and we cannot go to a total you know zero trust because of course the components need to be able to talk to each other well indeed the internet talks to them they'd be very bad yeah for some strange reason it's the most tough yeah so I mean all I'm saying is and oddly this is a pitch that I seem to borrow the same pitches internally and externally but it's something I've been saying to other people internally this week is it is not important to get to to understand your end goal and to write it down as a piece it's important to write down the start of this in a way that you can add to it so my thinking here is if we know there's going to be a manifest if we know there's going to be signing why don't we talk about the bits that we know rather than worry about how to use them to you know make the perfect solution so that we can build from them so we've got something to change as opposed to a document to write from scratch well there's usually a lot of vocal people on here and in fact I think it's been me and Fred Derek and maybe CJ that have been doing all the talking today so I should probably apologize but anyone want to pitch in on that are we going anywhere in the right direction with us I think there were some good there was a good discussion here and one thing to remember it's not just our problem as Fred Rick mentioned the Kubernetes community as a whole needs to deal with it or the software industry as a whole needs to deal with it so we don't I'm feeling that we don't have very good solutions right now but at least we are aware of the problem so I think as we make progress we should keep an eye open to towards the rest of the industry and what kind of best practices emerge there and see what makes sense to adopt from from the community at large I agree that this is not exclusively our problem and we definitely do want to be keeping the air to the ground for better answers yes anyone else speak to you usually have something to say on the subject or Oliver maybe okay running scared all right and now again my network is terrible but at the moment I am looking at the the agenda and I'm seeing that we've run out of agenda so did anyone have anything else they wanted to raise today mark your calendars for Detroit next year for KubeCon I know it's a long way out but it's a really great place to to visit do you have a date I don't know if dates have been published or not some sometime this timeframe as as usual okay um get writing your papers now seriously they've been pushing the the CFPs earlier and earlier because of the size so hello oh welcome back yeah sorry about that all right um again if there is anything else then now is your moment otherwise I will give you the last sort of 10 or 12 minutes back and you can go on with your days cool well thank you very much so see you next week all right thank you yep thank you everybody for coming and I'll see you all next week thanks bye