 then kick it off. I avoid video like the plague when I can. So I think people should have access to these meeting notes and you're welcome to add in other items on the agenda bashing. I'll just start out as I usually do with the total numbers and these I'm just reading these from here but it's kind of insane right now so we're up to 63 certified vendors. New ones keep coming out of the woodwork including some Kiwis in the last week and 74 certified products and you can see that we're up to 15 1 11 certifications. So I mean I think the fact that all these certifications are taking place via the GitHub repo means that it's pretty transparent how it's taking place and issues that come up in other sorts of things but of course anyone's welcome to speak up if you have questions or concerns about it but I'm pretty pleased with the process right now. So if there's not any other pieces maybe we can talk about Shanghai and then I guess we'd probably another month on Seattle but I think was it Doug or Brad who signed up for the intro and deep dive sessions and we could just briefly pop those out. Yeah actually there was I think a person from Huawei actually did this the signing up but yeah the Huawei IBM Google are all working together on the on the agenda for the deep dive intro. Great and I think I sent you a note with the link in it let me see if I can find it quickly hold on a sec. I'll just paste these in here for folks reference. So what would you like to settle on right now? Okay there's a link to the plan agenda. I'm not sure there's anything that we need to settle on it but if people have questions or concerns about the doc we do have a meeting every two weeks so I guess we have one not this week I think the next week to go over concerns or questions we have and then start passing through some slide decks. Sounds great so I'm planning to show up and maybe do a two minute intro but I'm totally happy to defer to the working group on anything else. Unless you if you want me to do more please please let me know ahead of time. Yeah no I'll add that to the to the list and I think Aaron might be doing the intro right now so I'll leave it up to Aaron to decide how he wants to balance that time with you. Yeah I'll work through that with you Dan. I'm happy giving up a broad overview but I haven't been historically involved with this group since its inception so I'm sure you can help me add a whole bunch of context. Okay and of course we already have a ton of Chinese conformance members so there's already a lot of interest there and hopefully a number of them will show up. In fact we'll reach out to our members to encourage them to do that. So should we move on to July's action items and HH do you want to chef for a second about where things stand? Kipi Hacker you're on I believe but are you muted? Yes I'm muted sorry about that. Sure. We've made a lot of progress in adding to the different components to be able to enable audience. I'm speaking to something in the agenda I mean there we go. We did the graying out of the links and we still haven't done it for the test yet. Sorry I wasn't expecting this to get thrown in that quick. The audit logs. We can circle back to you. Why don't we do Tim because I'm particularly interested Tim on this question of pulling the test to run from upstream. Tim can you go ahead and mute. Tim you have some two high-level questions about backlog and state and documentation updates which I feel like I'm going to answer as I roll through my agenda items so I'm interested in what specific questions you have not covered below. Where is the backlog like where is the canonical backlog for the one of the things that should be addressed by this group like we've unleashed folks to work on stuff but it's not been transparent as of yet like who is doing what where and what the actual execution backlog is. I know we've talked about it a couple of times in previous meetings but if we want to go into detail into where that is that would be helpful. I think the TLDR is I'm ramping up to getting us a backlog and I want help from this group in appropriately populating that backlog. So I'm hoping the update I provide will provide sufficient transparency on what we've been doing and why for now. But question why you like why you and I have that I have that question too yes I have that question too and then what was your concern on documentation update. Just a just a update with the state of getting documentation in place for the conformance tests so we can point folks at issues because I know that we had done a lot of effort to get a lot of documentation in place with regards to you know specking out and having a well-defined text and that was supposed to be auto published to the dock site eventually or to some location but I don't know what the state of that is either so that would also be pertain to the backlog. Can anybody other than Srinivas speak to that here. Okay the brief the brief part that I'm aware of is that there were a number of TRs outstanding to put that text in that I helped Srinivas get merged and as of right now I have no bandwidth to further assist in automating the creation of those docs and as we'll discuss below the way those docs were generated walk through a slightly different lists of tests than what Sonaboy has been running for this program. So there's there is work there to be done. The docs we're talking about are the ones that end up landing in Kate's conformance in the Kate's conformance repo under the what is it under the docs directory right now there's one for kube conformance 1.9 and kube conformance 1.11. This was this group's desire to have like human readable descriptions of what the tests are and why these are the things that define conformance. My personal opinion is it is not all the way there in being a consumable or be useful. I had initially tried an effort where I thought well if this is what the end results supposed to look like for describing like what all the different test cases are we could maybe work backwards from something like this where we write out all the test cases in human readable form and then figure out and then like implement those. But personally this needs to be pushed forward. I have no bandwidth to do it. I would be willing to help whoever does have bandwidth to do it. So if there's nothing there I think I want to just roll on to the rest of my agenda. Likewise this is Mitra. I have the bandwidth to be helping out with any kind of reviews in this area. I do not have the bandwidth to be doing the VR's and stuff. So I apologize like I've majorly distracted there and I've missed the conversation but I did Pink Shrini to make sure he joined the call and he's on now. Maybe Erin or somebody else could summarize what you guys are just talking about because obviously Shrini is pretty much involved in that. Yeah so my concern is the state of automating the publishing of documents based on the comments in the conformance test. That's not there right now. So right now it's like Shrini had to go manually run a command and then take the output of that command and then open up a pull request against the Cades conformance repo. How are we gonna automate this process? And then like personally I further have concerns that like the comments are kind of brittle but I don't have time to take issue with that. Yeah so Shrini I think I think you've thought about how to automate that process I believe right? Yeah I'm sorry I mean yeah Erin and I also talked about it a little bit. We need to solidify something on those grounds. I mean I'm open for any suggestions at this point. It needs to be automated it cannot be parsed. Right okay so obviously I don't think Vanessa had the solution right now on this call and I don't think this is the right form. Vanessa hashed through a solution. However I think Shrini will take ownership of making it happen because we need to automate the process somehow some way. Yeah I'd be happy to help with that. We have to just brainstorm probably one-off meeting like you know breakout session where we can brainstorm on this. I can set one up. Okay I'm looking for own not helped to be clear. Yeah I agree. Not to be a jerk about it but like I really need you folks to push stuff forward. Yes no no no Erin you're completely right. Shrini you will own this right? Yeah Shrini you'll own it. So real quick question this goes back to administrative year which was my first question is like where are we going to track this? Where's the backlog? Do we not have a repo that we can track issues in? We could use the conformance repo to track some of the stuff into triage and use it that way or we can use KK and just provide that enough labeling so that way we can at least if you just put it inside KK it'll get lost forever. But if you put in the CNCF conformance repo you can actually track it and it's small enough for you can manage it. One of the problems is the tools are spread out between KK and so that's one thing we need to know. I guess I would suggest that perhaps this group's interest and backlog should be tracked in this group's repo and there are like technical issues or bugs that relate to the sausage making over in KK that's fine but ultimately we should be tying back to like we did open a project board on this repo right? Do you something like that? I hate project boards for what it's worth but we totally do it whatever works for you folks. No I think let's just start off creating issues or whatever inside of our repo and track it in there and if we didn't want something more complicated than an issue we'll figure that out as we need it. So Srini can you open up an issue to track the progress or the requirement of automating the generation of our of our documents? Right. Okay thank you. Okay so if you folks don't mind I'll move on to my agenda items in here. I put in a lot of info. I'm not sure we're gonna have time to really drill deep on all these. This is kind of an intent to catch you all up on what I've been doing for the past month and get us discussing a couple things and then make sure I drive that discussion to the appropriately actionable places where I need decisions. I'm super interested in all this but could I just ask for Kim that we had one other piece of the how the Sonabue would pull the instructions from upstream. Should we add another separate item after yours? So I'm about to get into that. Okay great sorry. Yeah so let's let's the first item is refining the definition of conformance. So the definition of conformance is something that the Kubernetes project owns specifically within that it's SIG architecture. As I've been drilling into this I've noticed a couple inconsistencies. The first thing was that there were tests with the conformance tag that lived inside a directory called E2E node. E2E node is just node conformance tests which is a completely separate and orthogonal concept to conformance as we care about it here. So I updated our gatekeeper that guards the list of tests that the project considers as conformance to stop looking at that directory and I've had folks from Globent work on moving those tests over to a directory where they will get run as part of conformance tests. So wait a sec pause there for a second. You've had folks from Globent. Like where is that work being tracked? So I have opened up issues in the GitHub in in KK and I've been applying area conformance labels to that and I've been making sure this is sort of covered down at the bottom and in terms of how is this being done where the process is I'm opening up issues I'm calling out to people on GitHub publicly and then when it's ready for review or discussion by SIG architecture in terms of whether or not this is conformance related. I put it on a project board that SIG architecture takes a look at. Can we discuss there? I'm just wondering why Google is driving Globent versus this group. That's a really good question. I would so let me be clear. I mean, sorry, I'm actually driving Globent and I'm paying them, but several folks from Google have been assisting me with that, but it is not a Google task in any way and other folks who would like to get involved in managing that are welcome to. Ultimately, I'm certainly looking to SIG architecture on feedback for where we should be prioritizing. But I do want to emphasize that so far the first few months have just been them getting up to speed with Go and the PR process and all the other pieces. I also want to be clear that I don't want to drive all this. I just want transparency. So that way if somebody else asks the question, which will happen, and I've already been asked a number of times, that there's a canonical location or set of locations where I can point them to, and then it's clear to everyone what's going on and who's doing what. I also should. And I apologize, Aaron, for interrupting you again. But just we are sending those updates to this email list, the Kubernetes conformance work group email list, which is the best place to keep it. And then I'm looking forward to Aaron's discussion now on how we're going to track it more thoroughly than that. Yep, that's you put the words out of my mouth. I don't want to be the only person running this show. My ideal scenario is that this group collectively comes to agreement on a dump truck worth of work that we can just start to turn the crank on. But we're kind of not there yet. And I've been trying to help us iterate through that. So the next area of iteration is that I noticed that Sonoboy runs conformance tests by running a Docker image that Heptio has built that has a skip list inside of it, such that the set of tests that Sonoboy runs is different than the set of tests we list as conformance tests inside of Kubernetes, Kubernetes. I'm of the opinion that's it's Kubernetes, the project should be defining what the conformance tests are, that skip list should go away. And those concepts should be propagated upstream. So for example, I have the skip list in the in the meeting notes here, like I agree that if a test has a tag of like alpha or disruptive or flaky, it should never have been tagged as conformance. And if it was, we should go back and strip it out of conformance or figure out if maybe it's not actually as flaky as we once thought, things of that nature. But ultimately, my ideal scenario is that skip list goes away. And conformance tests involve just focusing on tests that have the conformance tag inside of them. Yeah, there's no there was no arguments. And in fact, Yago and Matt Liggett were involved in the creation of those things. In the beginning, and ideally, Matt Liggett and I had both agreed that we wanted to push that container, that single artifact that everyone can use, which is the kube conformance container, into upstream. And that's this that would be a canonical location that everyone could use. It just never happened. Right. So on that point, specifically, I have this ideal dream world where the same images being used to run tests for conformance and for the project CI and by end users. Practically speaking, I don't have the resources to support that within safe testing for at least the next quarter. But that's a direction I wouldn't mind heading, like in the q for timeframe. But in the interim, I just want to get rid of that skip list and make sure that, like, these are the sorts of things that should be used in a checklist of should a test be tagged conformance, yes or no. Why don't we in queue it into the backlog? If we're doing backlog portions here, and we're using the CNCF repository for it, why don't we in queue it for the CNCF repository in the backlog? That sounds like a great idea. Right. Are you taking an action? Okay, so then my next concern is specifically one of the things in that skip list is the word coop CTL. And it is not entirely so the historical context for why coop CTL is in there is because son of boy runs end to end tests from within the Kubernetes cluster. And for whatever reason, the coop CTL tests weren't working in cluster. And so they just decided to skip it. Right. So I think that those issues have been worked through number one. Number two, I think coop CTL exercises, like the coop CTL tests exercising a Kubernetes cluster are a great example of end user behavior, and are very illustrative of conformance related behavior, are already tagged as conformance. I think they should be conformance test. So we could go through a deeper review from subject matter experts to confirm that like nothing that isn't stable is being exercised. This is this falls back to architecture for this particular one, because they own it. The there was a meta topic that existed originally, which is one it didn't work for all cases for other people that Matt was aware of this too as well. We it works now because dims and I fixed it. So we can enable it should we want to. But there there are macro level concerns about whether or not that behavior is aggregate level behavior versus API coverage behavior. Right. I don't think this group owns that space. So we can defer to architecture for that one. Yeah, I have a point about that down further in terms of prioritization where I talk about like where do we care about focusing on direct versus indirect coverage? Because I agree, cube CTL probably indirectly cover indirectly covers a lot of functionality that we're not directly covering. Next up, huge thanks to dims. I don't think is on the call here, but he and someone else have been pushing on making sure that all of the test images involved in conformance tests support all of the architectures for which Kubernetes is built as part of the release process. And we think that that should be a mandate for a test being promoted into conformance. As a result of this, I know there's a power PC 64 conformance test being run out there now, which is great. And finally, the bigger issue where I kind of want to get this group's input and shepherd discussion is like refining what the definition of conformance should be. So I have an open poll request. There are many, many comments and discussions there, which I haven't actually had time to fold back into their quest. When I chatted personally with Tim about this, he was of the opinion that I should take all of this out to Google Doc, and we should go through one more round here, which I'm happy to do. I do want to make sure that I get this group's input and consensus. But my goal is to walk away with a document that has a bullet list that is so clear and explicit that it's obvious to both people writing tests and people who are reviewing tests, whether or not the test is being written in a way suitable for conformance and whether or not it's exercising a behavior suitable for conformance. To be clear here, I'm not talking about and don't care about anything related to profiles. I'm focusing strictly on the poorest of the barest of the core behavior that is super default. So the key thing that prompted me to start writing this doc to tighten up requirements was that years and years ago, the versioning, the version skew requirements were maybe slightly misinterpreted to imply that a cluster that was say one nine conformant should also be passing one seven conformance tests. That's not actually true. It turns out that client version skew guarantees are only one version back. So like the biggest change that I saw aside from refining all of the specific criteria was if you're conformant for like version 111, you only have to also be conformant for 110. You are allowed to fail the one nine conformance tests. I'm curious if people within this group like interpreted interpreted the versioning policy that way, or if this looks if this sounds like a change, silence is consent. I'm thinking. Okay, I want to give you an example of something that would fall into that category of not spending two releases, but does spend one release. I honestly can't think of anything. I can. Okay, please hear it. I'm curious. Yeah, so I was lurking for exactly this kind of issue. Right. So actually, kind of within Kubernetes, the only thing that multiple release versions skew supported for is skew between the cluster level control playing components, the API server and controller manager and scheduler and the kubelets. So the kubelets can be up to two releases behind the control playing components, but we don't even support any skew between the cluster level control playing components at all. Officially yet, which is obviously a problem for things like H.A. And we don't support downgrades officially, which is a problem for lots of other reasons. And keep control only supports one release forward and backward of skew. So 1.8 cube control will work with 1.9 and 1.10 cube control should work with 1.9 control plane, which is also different than the kubelet control plane skew. 1.10 kubelets are not supported on a 1.9 control plane. So obviously that's not ideal. There's been ongoing work for literally years to try to improve various aspects of that, but it is where we are. So Brian, can you elaborate a little because I understand conceptually what you said there, but what I'm trying to understand is aside from removal of a feature, in other words, it's been deprecated, do you actually have examples in mind where the functionality changed across two releases, but it didn't change across one release going to wrap my head around how that could possibly happen? Yes. I mean, there have been a number of cases. So the API machinery has been evolving heavily over the past few years. Entirely new things were added like our current API discovery mechanism. And the Swagger 1.2 was also exported, which is different from our our sort of custom API discovery mechanism, but for we effectively used the Swagger just for schema discovery and not for endpoint discovery for various reasons. That was added for keep control validation. And then it was evolved over time to Swagger 2 or what became open API. So there were a bunch of dances that needed to be performed to transition various details. There are also a lot of bugs in the schema information and things like that that had to be fixed. So clutches had to be put in and then we're later removed. So there was a bunch of complexity around that as well as changes in garbage collection and reaping and all sorts of things. Keep control had a lot of logic in it. It was basically a fat client. And these things were not officially in the API really. They were just in keep control. And it was sort of we didn't have the mechanisms in place for keep control to actually really reason about what version of the control plane it was talking to. Like there are all kinds of issues. So basically it was just through testing that we ensure compatibility and practically speaking, there was only one release of compatibility. There are still there's still very complex functionality that's in keep control. Keep control ply is probably the most the biggest and most complex that basically doesn't behave properly. If the fields don't exactly match between the versions, the behavior is very surprising. We're moving on. We're working on moving that into the API server itself to address that and other problems. But it's still going to be several releases before that transition is completely done. So yeah, it's a very complex set of surface area. There have been other issues that were unintentional in the tests, like tests that depended on event content, which wasn't actually supposed to be guaranteed at stable as part of the API service and things like that. But I consider those things as just bugs. Yeah, Aaron. Right. So like this is this is the level of detail. I need to avoid if I'm going to get through the rest of our stuff, though, I did want to make sure we entertained the question. The actual question I have for the group is where would you prefer we iterate on this definition? Google Doc or PR? Google Doc with a time out and then PR is the typical modus operandi for stuff like this. That's so applies to such a large audience. Okay. I will get to that sometime in the next three days. If Tim, since you had expressed an interest in being actively involved, if you want to kick it off, that's fine by me. But like it is important to me. I opened it up a while ago and then I haven't gotten back to it because I've been behind on other things. But I will start it up and iterate us on that discussion soon. Yeah, cool. Actually, it was great. Yeah. All right. Moving on. So let's talk about refining the definition of what improved coverage means substantively. Ultimately, I think what this group is after is to improve feature or behavior coverage. And there's no way we can have an exact measurement for this. I think a lot of the features or behaviors may ultimately be defined in such a way that even like very specific code coverage may not catch end to end behaviors or real world effects of things. But I do think that proxies are useful for us to, if nothing else, track progress and possibly inform discussions. But we will ultimately require human subject matter experts to really chew through this. I also don't think that there was ever really agreed upon definition of done from this group that we're going to reach a hundred percent X or a hundred percent API coverage, for example. I think the goal was just improve coverage. And I also feel like there's been this perception that great, we have conformance tests, but they don't really cover anything. And I'm curious what or how people are coming to that determination when they make that statement. So I'm trying to get us to to measuring our expectations, starting with less granular measurements and moving us to more detailed or more granular measurements over time. So things this group have talked about in the past were like client side API coverage information. This is a Meche's tool, which currently lives in tests for experiment coverage. There's server side API coverage, which Hippy Hacker has done with API Snoop. The version of that that I most prefer at the moment is the audit log review thing, which you all recognize is that wonderful sunburst graph that breaks down API coverage by API group by the endpoint and by the verb that we're hitting that endpoint with. The problem with this approach is that it relies on audit logs and the default audit policy filters out a lot of high volume and high traffic events. So it looks like we're not actually covering a bunch of APIs. The current modification of PR that's in flight for dynamic configuration of audit logs would enable this behavior and right. Well, so you also have the opposite problem, right? Of things appearing to be covered that aren't really covered, like every controller is going to pull the for resources that it cares about, even if those resources don't exist. Right. So system resources I wouldn't consider to be the same as coverage as user resources either. Right. So that's why I linked that there is a cap for dynamic audit configuration, but I don't expect that to land fully within this release. Most of it is landing in one 12 most of it. OK, very good. In the meantime, thanks to hippie hackers prompting and help and Catherine Barry from Google who has joined us recently. We're working on looking at API coverage from audit logs with the user agent included. So there was initially some work done to try and correlate like what test causes what API endpoints to be exercised by doing this manual process of like running the test client side and SSH over and like running that stuff over here. And like it worked, but it's kind of hacky. So what we're doing instead now is user agent gets logged as part of audit logs. And then the E2E client will set the user agent to the name of the test that is running things. So this way we can filter specifically down to like which test is running which API endpoints. Even without that today, Catherine put together a pull request into API state for a version of coverage called E2E coverage view, which allows us to filter API coverage by user agent so we can exclude all the API coverage that happens from things like controller managers sweeping through or scheduler trying to pull every single node and pod, right? If we have time, I'd love for us to show demos on those, but in the interest of moving quickly, I hope you don't mind if I just kind of move forward. The next thing that Catherine has been working on is what do we do when API coverage isn't enough? At some point, you know, we could like cover all the APIs with stupid crud tests, but we're clearly not going to be exercising all the behavior. So generally in the world of unit tests, what you do for this is you look at line by line coverage. Yes, that is a 12 blinking clock. Thank you. Whatever, it's fine. So in the world of unit tests, you look to line coverage for this sort of thing. So we're trying to run every Kubernetes process as a unit test and get line coverage from the Kubernetes system itself. Catherine has a design proposal out there that we are running through the community. We'd also appreciate input from this group. I think especially because at the moment, we're really interested in trying to make sure that we gather the data, how we ensure that the data is presented in a usable manner could probably use some input from folks here. But the idea is to do things like, you know, collect coverage from every node in the cluster and then merge that coverage information together. So even if tests end up exercising different kublids, we get the union of what code was actually covered. This is the sort of thing we can probably take to a discussion with subject matter experts to verify whether or not we've hit all of the appropriate corner cases for behavior. So with that, let's talk about how we can use this information to help us improve coverage and how we can prioritize where we should be improving coverage. And I think this kind of ties into an action that hippie had. If we look purely at API coverage and we try to go after the biggest least covered area, I think that's the wrong approach because it could result in us going after high visibility, but low value targets. I think that higher value targets of coverage are areas where functionality can be implemented by plugins or substitutes. So I'm thinking of things like CRI, CNI and CSI. And my goal is not to certify those implementations, right? Like, I think there should be a CSI suite, a CRI suite, a CNI suite to make sure that, like, a plugin is a CNI plugin. You are covering the direct area that William defined as profiles. Well, what I'm trying to cover is, given those plugins, does a Kubernetes cluster still satisfy the core behaviors that we expect of a Kubernetes cluster? Right. So I just want to comment on this. So this has basically been my direction since the start of the conformance effort. Since the purpose of conformance is testing compatibility, we need to focus on the areas where compatibility is at risk. Compatibility is at risk in areas where there are multiple implementations, right? So that's things that are explicitly pluggable, like CRI. And it's things that people swap out, like the scheduler or kubelet or ingress controllers or whatever. So starting with, you know, Aaron mentioned that node conformance earlier. Pod, the pod surface area is the number one most used API in Kubernetes. It also happens to be highly pluggable. So I think that more than justifies the focus on pod related functionality first. Sure, I agree to that. But that makes a ton of sense. I think the framing of the conversation piece is inverted. But yes, I agree. Yeah, that's like the pod is like the fundamental thing that makes the rest of Kubernetes work. So if we can't guarantee that a pod works the way it's supposed to, we can't guarantee the rest of the stuff. Just for clarity, though, too, there is behavior that is incongruent and inconsistent across different like C&I providers, for example, network policy is wildly different. So we ask you there's aspects that we need to be careful of there. Yes, we need to identify distinguished portable and non-built portable behaviors that are expected. This has come up in a variety of different contexts, since you mentioned profiles, windows, for example, there are a number of Linux isms, sadly, even things that are specific to specific Linux distributions like SELinux versus App Armor that are in the API. We're going to have to annotate those ideally in the API so that we exclude them. You know, when we have profiles, we may want to include them in profiles, but right now we don't have that. So we would just exclude those things. There are things that are going to be fairly subtle that should be consistent. Networking in particular is one of those like testing that pod networking actually works. You have addresses, IP addresses for pods that are routable to each other on different nodes is something that I don't believe we test. And I don't know that that will be covered explicitly by line coverage or API coverage or anything like that. But we will need to make sure that that gets tested. Brian, I'm certainly supportive of focusing on tests that matter as opposed to just a percentage number. I do seem to remember back in December at the live meeting in Austin, you had mentioned an effort, I think that was being looked at inside Google that would provide more of a base level coverage of just the just the kind of existence of the API and maybe whether they followed the rest verbs. Yeah, I remember incorrectly and did that effort go anywhere? I basically told people to stop working on it because the stuff they were doing was not going to be useful. It was like exercising a very small amount of the API server as to what they were doing. So we have multiple efforts underway, focusing on getting things that are currently in no conformance into the actual conformance suite so that we get more cop pod coverage is sort of one not entirely low hanging fruit, but not so doesn't require rocket science either. Existing E2E tests in other areas of commonly used stable portable features we're trying to get to the conformance suite. So those things are hopefully are usefully increasing the coverage. But I think just hitting every API endpoint has effectively no value. So I hand that if you assume that the Kubernetes code base itself doesn't change based on each provider, which I think is for the most part probably true that I definitely agree with everything Brian was saying there. It's all the points that's not it's not actually true. Well, I but but you but your assertion there about the biggest concern are the plug points CRI, CNI, that kind of stuff. I think in general, it's probably accurate. And my question was actually more of a tangential one to that, which is if someone is then going to claim conformance to our test suite. Do they have to say which plugins they're using? Or are we just comfortable saying gentlemen's agreements, you're not going to swap out your CNI implementation as right after you run the conformance test. There's a part in that way there's a part in the conformance process that clearly defines how to make the experiment reproducible to other consumers that is part of the CNCF efforts. So they need to specify versions as well as, you know, other details of how they created X, which will have that. It's true. OK, that helps. Thank you very much. Yeah. On those lines also and we'll have a process of why if if people can't reproduce the results, there is like a reporting process for that as well. So there's going to be a strong incentive to make sure things are reproducible. OK, Brian, can I just we can do this in our next meeting about email, but just one more minute on the very shallow broad coverage, which is we did have a case where a major provider of Kubernetes, their hosted implementation was not compliant because they had turned off a couple API features that they didn't think were necessary. And thankfully, those were caught by the original one dot seven conformance and they decided to turn those back on so that they could be conformed. So I do just want to ask here, because it does seem something like that something particularly hippie hacker might be able to do because it's kind of the inverse of the API snoop of instead of capturing which call is being made that we could potentially just go through the open API swagger definition and just blindly call every single thing. If you blindly call every single kind of views, there are many APIs in that that are not part of the conformance suite for a good reason. And there's not a question about which ones those are are not in the API discovery information. Like I personally would have no objections. If anybody from this group wanted to volunteer people to go work on writing tests to cover all of these stable APIs. Great. Go for it. So there's a meta topic here that I think is worth discussing, which is the first thing I brought up. Right. We would like a backlog. We would like folks to groom it and we would like to have people who are executing gets it, you know, give feedback in a regular interval. Right. Yeah, somewhere I have a doc with high level guidance on areas I think deserve prioritization. Those need to get translated into actually lower level details about what functionality actually needs to get tested and that can populate the backlog. But I am not going to waste my time redoing things that I believe have low or no value. This I mean, I think this is us kind of at least hashing out that initial priority of as we look to populate the backlog, do we collectively agree that focusing on the pod as the fundamental abstraction of Kubernetes above all else is the most important thing? OK. Yes, I would just say the next time you try to phrase that in an argument, invert the statement. OK, perhaps I should have started that and then explained why, but because like I don't care about all those pluggable things. I want to make sure that regardless of what those are, that the pod still works the way the pod ought to work. And that's I believe where we feel like we're not covering pods very well. OK, so now and again, this is the this is definitely trying to tie back to Tim's like how do we collectively get involved? How have I been doing this right now? So, you know, that that conformance doc tries to lay out both what are the criteria for a test to be promoted to conformance as well as the process to promote a test to conformance. That process basically is write the test, prove that the test works and then we'll talk about promoting it into conformance. I have made sure that all work related to this is labeled with a GitHub label called area slash conformance. I thus far have been driving those tests forward to LGTM either myself or with subject matter experts. I then put them on conformance test review board that lives in the architecture tracking repo of Kubernetes SIGs. If you don't care to remember all that, it's linked in the doc here. And then I show up to SIG architecture to badger Brian and everybody there about like, what is it going to take to get these things approved for conformance? Because usually right now we're at the state where we're still trying to iterate on really fine grained details of what conformance means, like what ways of testing functionality are acceptable, what functionality is acceptable. And so like right now, I know Tim saying like we've unleashed all these people to be clear. It is me shepherding and like two contractors. So we don't have like a whole herd of people who are bum rushing the project. I'd love for that to happen. In order for that to happen, I need a dump truck of test cases and I don't have that right now. I think like the ideal scenario, the current scenario is like, we're just kind of doing this back and forth, suggesting test cases from subject matter experts. And then we have Brian Grant and Clayton Coleman as kind of the SIG architecture bottleneck to figure out if that makes sense. I want to help us like grow this understanding to larger pools of people to get to that dump truck of test cases. And this is 100 percent where I could use this group's health. Yeah, and effectively we could rope in more of the API approvers and even some people beyond that. I think we need shadow reviewers so they can people can start to understand the types of issues we have concern about. And we can make sure that those things get documented like some things are documented, but they're like in the API convention document about this is not guaranteed to be a stable part of the API and things like that. So we need to collate a list or a list of pointers or something of those issues as we surface them. Like there are questions about, yes, this is a pod feature, but it's not guaranteed to be portable or in the past hasn't been stable or it's not part of CRI yet. Like they're all host of details that that one could get into when reviewing these things. We just don't have enough of them under our belt to have a comprehensive list of criteria. I mean, to be clear, a lot of what's been happening right now is I've been kind of correcting and refining the definition of conformance. We really haven't been going out there and writing brand new cases to cover new things. It's more been a process of identifying which test cases already exist, which look like they could be promoted to conformance or which test cases were called conformance, but weren't actually being run. But at some point, I'm going to run out of those two things and it's going to come time to like write some new tests. Yeah. And along those lines for people who didn't hear the discussion that we had in SIG architecture and maybe elsewhere about it, at some point there's going to need to be some engineering efforts around building new test frameworks that will enable us to run tests in multiple modes. Right. There are a lot of advantages to writing smaller scope tests, unit tests and integration tests that can be run outside of the end to end test framework that we have. They're faster, they're more efficient, they're more stable, they're more bugable and so on. Right. Like all the reasons why unit tests are good. But so if we had 100 percent coverage in unit tests, we could still have zero percent coverage in the conformance test because they weren't running an E2E. So we are going to need frameworks that can actually abstract that away and run the tests in multiple modes. You can directly invoke the functionality in the unit test and you can invoke it through an off the shelf cluster in the end to end test and still test correct behavior. So that's more of a rocket sciencey thing. I don't think it's something that contractors can necessarily do, but somebody is going to have to work on that and we'll probably need more than one such thing. So Ben, the elder has been taking a stab at it a little bit in test and fro. But that's very early stage still with the Kubernetes and Docker experts. But that doesn't cover multi node or any of the advanced scenarios yet, but can at least provide a baseline for some set of conformance tests we are hoping. But then that's still a work in progress and there are PRs in test and fro. Well, the Docker and Docker thing is that's what you're referring to. I don't really consider it to be a unit test. Maybe it could be considered an integration test. I mean, obviously it wouldn't actually require a cloud provider or actual multiple nodes. So that's a useful thing for just exercising the components themselves, although not in a real world scenario. I'd say that's like a whole another category of tests we need to be able to run that in both the Docker and Docker mode and on real clusters. They're typically partial integration tests of some kind because they require other information in order for you to work on it. But that's that's splitting ears at this point. Yeah, like I view all of that discussion as relative to improving the testing hygiene of the Kubernetes project as a whole. Ultimately, conformance tests still have to be run as black box tests that simulate real world scenarios as end to end tests. So Tim, since I've been trying to tie back to your point, like how do you think this group can help me accomplish the goals I just laid out above? I think we need to start with a backlog and have someone run through prioritization that can say like do A, B first and A, B. Because right now I have zero visibility as I mentioned before. Like what's the level of granularity we should be doing for say, OK, we've all agreed the pod is the fundamental abstraction. How do we break that up into the appropriate units of work? Because like one Uber issue say cover pods. That's just like the way we do everything else in Kubernetes. We have a large breakdown ticket which kind of enumerates the state space and the things that need to be done. And then we have a bunch of PRs the view of whatever that is. Did my internet break? Can folks hear me? I got you. You were a little laggy. I didn't hear most of it. Yeah, I was sure it was me. An umbrella issue and then basically enumerate the state space and that umbrella issue and link back to that umbrella issue. Yeah, over time. It's the way we've done it since epoch. Yeah, I mean, it wouldn't be that hard just to look through all the fields and pod and group them in some rational way like Leibniz probes, exact based HTTP based whatever readiness probes, same thing and just go through and figure out which things we have coverage for, which shouldn't be too hard because there aren't that many. And then think about other cases which aren't just reflected by what's in spec, like, you know, interpod networking and things like that. And crowd crowdsource that amongst some domain experts. And that should be a reasonably complete list, certainly vastly better than what we have now like some somewhere for some other reason, I had a list of some things like can two containers share, actually share a config map or can, you know, if one container writes in teacher, can the other one see that things like that? So just in case I haven't said it enough, like, I really don't want Google to be the owner of this. I think this group is the owner of this. And if you want to help out, I more than encourage it, I would greatly welcome it. I think that I apologize that I didn't really get a chance to demo it today since we're kind of at time. But Hippie and Catherine both put in a lot of great work to show some of the things that I linked if you want to take a look at them after the fact, it's great. And I'll make sure we have lots of pretty shiny data and graphs to show next time. But please. I had moved this back to a monthly meeting because they'd been a little quieter. If we're having a real active period right now, I'm happy to move it back to twice a month. But why don't we go monthly for another month or two and see if we keep running out of time like this? And I do want to remind folks that the mailing list is there and all of us are on it. So please feel free to engage there. So I just want to let people know that in terms of my sig architecture related work, I am prioritizing components test reviews over reviews of API changes. That is tests of existing functionality or I am prioritizing ahead of people helping people add new functionality to Kubernetes. Great. OK. See everyone in a month and on the mailing list. Bye.