 Okay, this is Dan Kahn from CNCF. We can go ahead and get started. I want to thank everyone for joining the call today. The first piece I'd like to just spend a second on the status for KubeCon-ClavinativeCon Amsterdam, which is March 30th through April 2nd. We are still planning to hold the conference. And so under the upcoming events, there's a link to this novel coronavirus update. We're investing a ton of resources to ensure that we're following best practices as to keeping all the attendees and staff safe and ensure their well-being. So it is still possible that this crisis is gonna end the Netherlands health authorities might ban conferences. It's a little hard to predict what's gonna happen. That has happened in France. It hasn't happened in the Netherlands yet. So I would just say that we are watching that incredibly closely. And we're certainly cognizant of the risks involved and challenge on planning travel and such. But as of now, and my expectation is that the conference will occur. And I look forward to seeing many of you there, not shaking your hand, but we're gonna be popularizing the concept of elbow bumps instead as the official KubeCon Amsterdam greeting. I'll also mention that I personally will be at the Linux Foundation Member Summit in Letahoe in one week. And I hope several of you or many of you will be able to be there as well. And then I will be attending the Open Networking and Ed's Summit in LA and then the CNTT meeting is happening the two days directly afterwards. And I'm planning on attending that as well in LA. And so of course, all of that is modulo the public health issues and companies and people's ability to travel. And so we are following all of that extremely closely. What I can say is if you do want any more information about Amsterdam, this link that I pasted in to the chat window contains all of the up-to-date information and always will. I'm happy to answer any questions on that or hear any comments or such. And then my hope for the call today was that Taylor could give a little bit of an update on the work that his team is doing on beginning to build out the CNF conformance test bed and talking about what is working today where the progress has been also some of the documentation that they've created for others to participate and that we could spend 10 or 15 minutes on that. And then if Robbie would be willing, I would love to then spend some more time on the CNTT collaboration question. And in particular, I have a question for Robbie and anyone else from CNTT that's here, which is just looking at these requirements that have been put together on the platform side. The kind of key question that I'm coming up with is that my assumption is that any RA2 platform, I think it's RI2 is the right phrasing, is going to be certified Kubernetes. And my belief is that the tests we associate with certified Kubernetes hit something like two thirds of these requirements that are on here. And we could go through and maybe we should even just do a side call and try and lay that out. But I'm particularly curious on the question of what about the requirements here that aren't covered by certified Kubernetes? I understand that right now, these are being implemented as a spec first and that's happening live on GitHub. I would sort of ask the question, is there a plan for a test suite associated with it? Who do you see writing that and how do you see coming about? I am familiar with the timeline document, but I'd love to see a little more context on that. And then related to that and sort of the question that I'm getting at is that we're trying to decide with the CNF conformance, does it make sense for us to also begin writing some tests on the platform side, or should we keep it to essentially the CNF side? Absolutely, that's fine. I'm happy to share also one slide on this. That's great, Rob. So we're gonna get to you in about 15 minutes if that's okay? Yeah, sure. Great, so let's just have Taylor start unless anybody had any comments about the coronavirus and KubeCon or any other agenda items that you were looking to discuss. Taylor, would you mind diving in? Sure. Happy to have you. This is a quick question on travel to KubeCon. So any updates on the flights? I mean, anything on like, for example, which airlines is good or bad? Any news on that? I mean, that probably could be asked or even more sensitive than actually staying out there in Amsterdam, I feel. I mean, all I would say as of right now is that there were some people who were planning to connect via Milan. Please don't do that because Milan is in the Lombardi region of Italy, which is requiring people not to have traveled through for the previous 14 days. So you're probably aware that, I mean, US airlines in particular have been canceling a number of flights to Asia, Korea, Japan, China, and Northern Italy. But as of now, there's not been any impact on flights to Amsterdam. So I think all airlines are equally good that we'll get you there. Anything else I could add on that? Okay, Taylor, please. Sure, Dan. So I can share my screen a little bit here. So this is just in the nuts to start. So on the CNF conformance initiative, there's just three main areas, the highest level for what are we covering? And that kind of goes into it. And I was talking about with platform versus application level testing. And if we break that down, we're going from high level stuff that could potentially go into slides and understanding where it applies, the categories. And then it goes down further in depth on the documentation towards the tests themselves. I'm gonna jump into that. So if we go in here and I've posted some of these into the chat, I'm sorry, in the notes, I can drop the link here in the chat. So here's the read me just to start off of the GitHub repo. And this ties in with the categories, which we've been talking about for a while. These go into cloud native principles that we would look for in CNF or cloud native network functions, so compatibility, statelessness, security, and what are those main? And then these can go further. You can look at, this goes into the actual categories and what type of things we're talking about. So compatibility of CNI plugins, how it uses the Kubernetes API. And there's different tools for looking at this, like API Snoop allows us to look at an application's usage of the Kubernetes API from alpha through GA endpoints. And then we break this down on each of these categories. So what are we talking about? Security type test with privilege mode and access to different things. So this is to be able to have an idea of what we're looking for. Some of these may overlap. Either the tools usage or the test could overlap in the different categories. But this at least gives an idea of the overall structure. And then the other part, if I can go back here, just read me, implementation overview is to try and provide some of the questions or answers to some of the questions that we've been having. So we're leveraging upstream tools. So mentioning some of that, you can get more of these if you look on some of the other documentation, but like OPA gatekeeper, Helm delinting tool, prom tool, these different things that we're using upstream for building the test. And then what is the actual framework that we're talking about? So this goes in a little bit about this, the way that the tests are written, trying to make them easy to read and understandable so you can go in and know what each one of them is actually doing, tying right back to those categories. And then the testing that we're doing right now has primarily been using the CNF test bed tool chain. And there's some links in here and how that could be leveraged. And this creates Kubernetes clusters on packet using upstream Kubernetes. And the tooling itself is pretty open to supporting other providers if desired. So if you're interested in seeing something else leveraged with the CNF tool bed tool chain, then please open the PR for that. But it's not documented here, but we'll have pretty soon to some testing that we've been doing with kind. So that wouldn't allow say performance testing, but if we're talking about development and being able to contribute a lot of the tests or validating functionality, then that should increase the capability to do that. So we'll be adding some documentation for running the conformance suite in kind. That'll be an alternative to actually running it on physical machines with packet. And then you can see the installation here, a few prereqs before you actually get in and you'll be gonna skip past some of this. Once you're in there, then there's some quick start on if you have a Kubernetes cluster and you can get right into the first set of tests. Beyond that, we have a usage document that goes through testing specific pieces, including the entire categories we're trying to make it. So if you're focused on one area, maybe security or compatibility, you should be able to run all the tests within there. You can also run specific tests. So continuing to update the documentation across the board. And at this point, I think we're ready for more people to get involved and try to follow the documentation and give feedback on repeatability. So that's one of the main goals that we want. And we've covered quite a bit here as far as from the highest level all the way down to contributing tests and trying them out. Another effort that's happening right now is getting a test suite that actually runs each of these tests continually. So we'll have that up soon. And any tests that we add will be running in a CI pipeline. We have, I think about 10 tests right now that are either in completed and ready for say they could be run automatically or they're in peer review QA before they roll out. And the way that we're doing the testing if you go in here on the project board, there's two main areas. There's a specification if we have stuff that we're not ready yet to write tests, we're still trying to figure out what tooling. So there's research on these, including sample CNS, Nikolai from NSM projects, been doing some research on using a prox from OPNFE. We have other tests that are about using it during tooling. And once we're ready with something that we want to implement that goes over in this project board. You can see there's quite a few to do and those will keep getting at it as they come out of that idea state. And then planning out what we expect to see the input and output of each of those tests, that's development. And then you can see there's a good number in progress and then going through QA and peer review before they get completed and moved over. So if there's anyone that would like to help on any of these, including the IDS stage for potentials, then please reach out via CNCS Slack or via issues on the CNS performance test. Does anyone have any questions or comments? Would it be possible to have a demo maybe next time just to see exactly how the running, is that something possible? I'm sure, I think that we could get a demo together. Are you suggesting, would you like to see like a one-on-one or are you wanting maybe for the whole group? Probably the whole group would be useful. One of the things that I wanted to understand which maybe the documentation is not clear to me. So there's seem to be an installation phase for a platform press, right? And then on top of that, you add the test suite on top of that. Is that the case? So it's a question about how it's set up or what test we're actually going to have on? How the infrastructure is put in a place first and how do you deploy your infrastructure? And is that an instruction on the page will show you how to deploy the infrastructure first and then to the testing on top of that or does it assume that you already have a cluster up and running and have a CNT? Okay. So right now, most of the documentation for this are going to be either on the read me in this installation section or more extensive test after the platform is up is in the usage document. We'll probably be moving this over into its own install DoxTune. And one of the areas I think that we need to do is I have a section for platform setup in the pre-rec setup before you deploy any application of CNF that you want to test. So if you have a, we have several simple CNFs that we're going to be using but if you have a CNF that you want to deploy there will be a section that's focused on that and then the actual cluster setup. So this installation area here is would be the platform setup. And right now, similar to what I said on the implementation overview, this is using the CNF testbed. So this is kind of just the overview, setting up Kubernetes on packet and how do we do that? We'll use the tool chain that's on the CNF testbed. That actually I'll click this link. The CNF testbed documentation actually has information on setting that up. So this is reproducing, it's saying the CI environment for CNF testbed. This walks through actually bringing up a cluster on packet that you can test with and it talks about the different, there's some differences on machine types or if you want to do different things like CPU isolation, there's other things. So this gets a little bit more extensive on the CNF testbed side. On the conformance side, we've mainly say you can go use those, you can use the tooling and here's the commands to run without getting extensive. But I think we'll end up with a section that goes more into this and one of the alternatives besides packet with Kubernetes deployed to packet will be using kind so that you can use Kubernetes and Docker for doing testing. Does that answer your question? Yes, thank you, Dan. Tara, could you just spend a minute or two talking about the sample CNFs that you're using as you develop the tests? I think it's also, if we could spend a couple minutes maybe chatting about our thoughts right now on scoring for bronze, silver, gold and giving some context about what's involved in getting to gold, I think it's gonna wind up being important for this group. Yes, I have the same question that, because I guess the whole test switch should be set up for those gold standards but there's no concrete API deflation for those. So I also want to know how to check if it complies or not. I can go into that. I'm on the wrong repo here. So the sample CNF right now, similar to the any type of gold standards for certification or where the points are that's still to be determined. So the focus on any samples are to create ones that allow us to cover the tests that we're working on right now versus trying to build one or that will complete everything or validate extensively everything in production. So trying to build up to that and then get feedback on each test. So the idea which we're working on here you'll see some of these would be, these would go into some sample area. Met seven are gonna be upstream. So this one here that I'm showing on this particular branches, it's using core DNS. So this would just be a layer seven example. If you looked at the VCB use case, ONAP and various other projects implemented this would be like VDNS. So it would be one CNF and then entirely use case. So it seemed like at least one valid one will probably adding others like we have a serving gateway from an EPC use case will be adding. But the idea is to take something ideally upstream and be able to pull in if we have Helm charts with this one we can reference an upstream. We have not on this particular branch but another one would be using a CNF that has invalid Helm charts or maybe some other pieces. So we're talking about components that could be used in a use case and focusing there. What are the smaller components that we can say these are the features and how is it implemented? So application that's part of the use case. The packet gateway or serving gateway or MME from EPC would all be examples that we could put in here. And then the next part is how are they used? And this is something where the documentation is here. I think it's actually maybe if I go some of it as I pointed out was here but we need to update. How do you use a CNF? If you wanna use a CNF to actually test I think this is something that's partly over in the usage documents and other stuff but the general idea would be you're gonna put your CNFs in the CNF directory there's a structure that we're saying you need whatever your Helm chart, whatever other pieces and there can be some static testing of that CNF and then there's deployment. Deployment level run time test and it would go through here and do the static test deploy it and then run the run time test. That's the simplest case. And then what would still be in planning and trying to get feedback on is how would we test a CNF that's maybe in a use case or an example? So an example of that would be the serving gateway in APC. So we could say we're testing the serving gateway someone else may have developed a different vendor may have developed MME or some other component but this vendor has delivered a new serving gateway so how do we validate that? There's probably going to have a need for deploying a few other components to test the serving gateway itself. So that's something that we're still working on but the first part is how do you isolate and run individual CNFs? Does that answer the questions around sample CNFs and kind of what we're thinking before looking at gold, silver, bronze? Yes, so I think so everything is still ongoing that because even for the same CNF we have many different implementations and so maybe we need different test suites for the different implementations to check the same feature in maybe the gold standards something like that. So I guess maybe it could be very difficult to achieve a unified test suites to finally achieve this certificate program, yes. So that's not the way that we had been envisioning it we were envisioning a single test suite that you can run against your CNF that the resulting score that that test suite gives you shows you whether your CNF is currently not so great bronze, silver or gold. So and then we're presuming that there's gonna be a ton of debates about what those scoring ranges should be but not that different test suites would be needed or differently the key phrase is sweet where there are many, many different tests that would be needed. And Taylor and his team are working hard to leverage a bunch of different upstream tests but what they're doing is combining it all into a single test suite that can be run with a single command. Taylor is that a reasonable summary? Yeah, that is. And what I may be hearing in addition to this and please let me know if I was hearing this there's potentially other test needs that would be out of scope. So I don't think that this test suite is gonna try to validate say an SC standard or maybe a protocol standard that doesn't have to do with verifying how the CNF is deployed and maintained in the life cycle. So if we talk about the functionality or implementation of a particular application of CNF then you could get into details where you need some type of integration standard that could be out of scope. That's probably an endless amount if you're integrating with any API for lots of different services and applications there's could be additional tests. So the focus here is does it the CNF conform to cloud native principles and standards? Can we validate that? And if we look at it from that standpoint then you should have one test suite that can cover all of that and validate that the CNF is gonna behave this way and you're gonna get the benefits if it's behaving to these standards. Yes, I understand that we're testing for those principles but still if we don't have a unified specifications or APIs, something like that I think it's very difficult to find a way to cover all the scenarios for a single principle. I mean, because there are many different implementations to achieve one of those principles. I understand and that's definitely something that we wanna look at. So there's say on a specific category if we're looking at statelessness there's many different ways that you could handle not having states saved in a CNF. So how do we test for that without saying we're gonna test that the user is using one of let's say a hundred different implementations. That's probably not the way that we need to go. So then for that one we're gonna look at what are some things to not do? And of course there could be many. So I would love to get you involved in the conversations around those. And then if we're looking at something and potentially you're saying an implementation that you think should be cloud native but it wouldn't pass then that's probably something that we need feedback to say how can we improve the test or maybe add a new test. So I'd love to get you involved in this. Yeah, sure, thank you. Yeah, I would emphasize that point that although we are using these this core DNS based CNF as a initial guinea pig to ensure that the tests are passing when we think they should pass and fail when they should be failing the real validation is gonna be to reach an alpha level for this CNF conformance test suite and then get the vendors and the operators in the space to run it against their own CNFs and to start giving feedback on oh, this test is too strict or doesn't test for the right thing or is unrealistic and then to also begin to help us with some of the scoring questions. So I really would emphasize that this needs to work in the real world and not just theoretically. Yes, of course, I think we need to discuss about that more later. Yes. Okay, well, if there aren't any other questions for Taylor on it, I would love to hand it off to Ravi who can maybe give us just a little bit of an update in general on CNTT and feel free to take the screen, Ravi. Thank you, Dan. Sure, I had one more comment to make on the previous topic and I'll make it quick. So a couple things that are very low hanging fruits that we can use to test almost any cloud native application out there is like we can go and we can go kill a pod and see if it affects service or we can try performing an upgrade or a downgrade using the common tooling. And so there are some very low hanging fruits that we can start with that should cross cut most CNS. So, and I would expect that some of these things that we start with would probably start with some of those areas. Anyways, we don't have anything else to add at this point. It's a really good discussion, thanks. Sorry, guys, for breaking Greg is speaking from over. We are quite new in this group. Maybe it was discussed before. Sorry for asking that question because something what I'd like to understand and please correct me if I am wrong is that we are talking here about telecom services. So it means that we have to handle somehow existing protocols like we have diameter, CIP or SS7 and actually this conformance what I've seen is related to behavior of component itself. But the question is how to confront that with using or doing what is designed for. So I mean the CNF can be tested. Yes, but the question is that is doing what should do? Is handling SS7 or diameter traffic correctly? Did you discuss this topic before? That's a good question. And I think that it has been discussed to start. And I think that most likely that needs to be split off from talking about is a component cloud native conformant and then how can you use it? So we can say this application is designed and developed and works in a certain way. And then does the application do what you want in a larger sense when you start putting all those components together? I think those are two different things. And whether or not it's part of the CNF conformance suite to say test that is the next part or not would be to be determined. We definitely from the standpoint of any application that's gonna work with current standards then that is gonna come up at some point. It's what is covered in this test suite to be determined. Okay, thanks for asking. I already discussed this topic to Dan last week because right now I see that maybe it's not the right time for influencing the white paper which you are going to release but I would like to see the way how legacy protocols will be handled in CNF because... Okay, so I do have... I think your audio cut out but... Can you guys hear me? Go ahead. So I think Greg actually going back to your point about the functional testing. One of the things that I personally tried to do and understand in CNT is where things should be tested and who are going to test them. And just to give you some context this slide I'm showing in front of you this the purpose of that slide was to understand the new verification program launched by LF and the question was what test we need to include in that program and those tests where they coming from and what does CNT role plays in regards to those different testing needs that we need to perform when we talk about containers and containerized platforms. So I get to this slide and where the question mark is I'm gonna talk you through it in details now is where I don't have a clear idea about which projects in the industry will cover those aspects but in essence what we're trying to do in CNT trying to understand the relationship with the CNCF performance test with OVP phase two and with the other projects existing in the industry. Now CNT as you can see on the screen if we look at the right so the left hand side of the diagram we do have the reference implementation which we talked about maybe in the past where the intent of the implementation is to create an instance in a given lab where this instance reflecting the specifications we create in CNTT and that kind of information in that document will be specify what are the lab requirements and also it will specify what kind of infrastructure we would like to have in terms of capabilities and that capabilities could be whether it is any to waking capabilities or to the storage capabilities or what kind of capabilities infrastructure we'll have to have and that information the expectation is to be feeding into a community installer and that community installer I know Taylor talked about the CNCF testbed and the way that infrastructure can be installed and those installers the idea is to take that information as an input and the result will be installing the cloud infrastructure into that given lab that specified and available. And any vendor if they would like to provide their own cloud platforms they should be able also to consume that specification and install their cloud platform in that given lab and that lab could be their own internal lab or it could be packet.net or it could be a community other community labs like open FV labs it does not matter which lab it should be as much as well as long as the installer supports that kind of lab infrastructure. Now, the other part that we were looking at in CNT is the reference certification or reference conformance the name has been changed lately and that is really a set of requirements that we say as a community that would like to test against and that could be a part of the infrastructure testing but some of it are also the workload testing and the way we see that is once the infrastructure or the cloud platform is instantiated in a given lab would like to make sure that this cloud platform is compatible or compliant to the CNT specification we specify in our documentation and that will mean we need some way in the industry in an open source project that will be able to create those test cases for us that we can run against the given cloud platform and determine if the cloud platform is compliant and conformant to the CNT specification we said. The actual test requirements what exactly we would like to test against and what exactly we expect to see in the cloud platform will be written and determined and specified in our CNT reference conformance documentation and this work stream, the RC2 is being established as we speak this is still discussion with CNT to create that working stream within CNT but again, that will expect it to give a requirement to the different tooling and testing projects within the community to create those tests and run it against the cloud platform and this is all from the infrastructure point of view I'm not talking about workloads or CNFs at this point yet now when it comes to the CNFs when we have actual cloud native function that we would like to test I think there are different areas where the testing is needed one of them is we need to test the orchestration aspect and the manual aspect of this cloud native function and that includes for example has it been packaged in the right way or does the packaging is compliant to its standard or does the package compliant to the TOSCA or HEAT or help chart or whatever packaging technology that need to be supported and that's testing is not, I'll call it offline testing because you don't need to run the actual CNF to know if the CNF actually is conformant or not and just to clarify this is not what I'm talking about here in this diagram so that this thing will be covered by other projects my view for example I know Onab in the LFN they're doing some testing around containers and what they test against is the compliance of the packaging against the definition that Onab expect to be onboarded into Onab and the other kind of testing is the cloud native principles now this is what I'm really showing in the bottom diagram here and this is kind of testing against the categories that Taylor highlighted which is around compatibility, statelessness, security, scalability, observability and so on and so forth and those are in my view are very generic and those testing does not necessarily care about what infrastructure lie underneath because they're all impacting the actual workloads and how the workload is being designed and how the workload is being architected from cloud native principle perspective now there are other aspects of that that CNTT in my view will be impacting and this is around resources and hardware consumption now if the container and cloud native function is being designed to take advantage of let's say acceleration capabilities from the infrastructure and that acceleration capabilities was not specified in CNTT as prohibited acceleration to be consumed just an example I'm not saying we prohibiting I'm just saying if we have an restriction in CNTT to on the way that a given acceleration resources consumed or how should we consume this is where I feel the CNF conformance test and the testing that they do specifically around hardware resources and scheduling will be impacted is to kind of support those test cases so in my view CNCF conformance test has two important point one of them is very generic to any cloud infrastructure and the other one is maybe some requirements coming from CNTT to specify we have this on specific requirements from the infrastructure and would like you please to test against those so that would be a subset of these cases that just to cover the requirement for CNTT now this is in my view still workload testing now the other testing that I feel is still need to be covered which is I went mentioning here in the diagram let me see if I can use the marker I don't know if I can use that but this box here that says community tools and tests for CNF in my view this is where also another area where I don't know exactly who's going to do this testing but going back to Greg point from Ovo that sometimes when we talk about performance testing when we talk about runtime testing how the actual CNF is consuming that is the hardware does it use the API that we allow it to use or does it use what kind of it working does it expect them how does it consume that and it working from the infrastructure point of view and addition to that how does it perform and to know how does the CNF perform we need to understand and have an understanding of the category of that CNF what functionality does it perform does it do in order for us to understand how do we test that CNF so for me this is a much bigger question and this is the hardest one at least in my view because there's a lot of information we need to know from the CNF itself what functionality it performs before we can actually do this testing but other aspect of this testing is still possible because if the CNF is consuming let's say storage using a specific API and that API is not being determined and specified in CNTT then we should be able to test against the API or the interface again interfaces is another topic that's also in my view we need to be able to test against does the CNF use for example if interface or if XDB interface to consume the infrastructure and this is kind of we need also to find information about how to test around those interfaces and APIs so I appreciate that overview I didn't totally get the very specific piece of information I'm looking for which is the testing for RI2 that the test development hasn't begun yet has it no it's not done yet okay and so you're envisioning that first the RI2 would be finalized and then are you envisioning that there will be software tests for it or is it that a like a certification provider a company would go through each thing at a time and just manually confirm that you can create storage that you can do networking that sort of thing no so the expectation is that the RCU work will start as soon as possible in the CNTT once the work storm is established and that will be a list of these cases that we need to see implemented to test against CNTT specifications now some of those testing as I mentioned we would like ideally to understand how that relates to the CNTF conformist tests that you guys are doing and potentially they mentioned the discussion we're having that those requirements can be included in the CNTF conformist tests or we need to find other ways of creating those test cases to be able to start using them against a given workload or against infrastructure okay so that doesn't I mean we are considering now adding this function adding an additional set of tests into the CNTF conformist test suite that would potentially then be able to satisfy some of the needs of the RCU work okay yeah good it sounds like there's so the two main areas that I'm hearing are a platform and then application and then if you look at each of those for the platform there's probably a lot that could be generalized similar to the current principles CNTF application tests that are happening in the conformance and those ideally we could look at what's missing from stuff like the Sonoboy Kubernetes conformance that wouldn't cover it what other things that we want to look at so these are general tests and then you have the specific tests that you're caring about for whatever different reasons so one of them would be does an EPC does it have the implement in a way that conforms to standards for integrating so if you're saying you want to support FIF or some other interface or something those set of tests if we can say these can be grouped and what are they trying to accomplish then I think it may be easier to determine if that should be it's own separate test suite or if it should be a subset if we have those split out yeah that makes sense so one question teller that I want to understand which is not good to me so are we saying that the CNF conformance test will cover platform testing I think that's to be determined okay and that's something that we want to look at what is what are we wanting to test platform wise so that could be trying to figure out if we say what's desired for the RC2 what are you looking at for a telcom platform and then trying to split that between here's things that are protocol or integration specific to they're an implementation so you may get rid of the MMA and have something there could be an APC could be implemented completely different and not have any of the current components at some point in the future so it's really what do we want right now though so right now you do need integrations because those are out there and you have these parts that you expect the platform should work with existing so if we can figure out what that is at a container platform level the difference between a specific thing like supporting an APC and then what's not specific then we could see oh it's needed so if we park the APC discussion for a moment because I think this is a long shot and long term if we say there's two kind of testing right there's workload testing and there's platform testing now in the workload testing they also I think you already covered the cloud nativeness of that workload that's not necessarily CNT specific but there might be and there will be some requirement coming from CNTT that we expect the workload to comply to and that's around resource consumption for example and around interface and APIs towards the infrastructure so if you how do you consume the infrastructure and those are where I see more requirements coming from CNTT towards the workload testing now the other aspect is the platform testing now how that platform is really being implemented what kind of plugins it uses what kind of protocol it supports and so on that's also some of that requirements will come from CNTT and if I hear you correctly what you're saying is we need to look at the RC2 when it's being created and see which ones will fit into workload and what more this case need to implement to support them and what goes into the platform testing and what workload what sorry what this case is we need to create for those platform specific requirements Yes, for the workload I think we're covered on what we're doing for cloud native workload testing and of course there can be any number of workload testing beyond cloud native or that we need to do the same thing for the platform so what parts of the platform testing should be about cloud native conformance versus which ones are I don't want to say vendor but whatever you would call the other parts that are non-generic these are things that must be met to fit existing standards okay then what are those so if we can split that up and come up with a set of tests that we want to look at for the platform then we can then look at existing tests like okay does the CNI conformance and Kubernetes certification do those cover those if they don't can we contribute and get those updated or do we need something in addition maybe the CNF conformance we add a set of platform tests we don't want to start down that road of adding platform tests and saying it's going to be official until we decide if there's something else that already makes it so you expect this box here to be covered by the CNF conformance test and that will have the cloud native principle testing as well as any extra requirement that CNT might need but this is something still to be determined I believe it could be that that could what you have there on the screen the CNF conformance we could have a set of platform tests where we may say it's better to contribute to the Sonnaboy test for Kubernetes maybe extend and have staffed with the CNI conformance testing for the CNI spec we need to determine what's needed on where that new box is that you created and then decide on where the testing should be I do think that there should be a separation between is this cloud native on the platform and is this cloud native on the workload versus does okay it is cloud native if you've developed a CNF in a cloud native way well can you actually use it following other standards those other standards should be tested and I think we should determine where those should be tested which may or may not be part of the CNF conformance it could be another set of tests yep like I said thanks I think CNTT is going to care about that an aggregation of tests that are covering many specific areas when you're looking at what is going to work for Telcom you're going to care that it covers many different tests and if we're focused in each of those categories then I think we'll have more confidence in the results of each yeah so I'm good all right we're at the top of our I think we should stop there but Robbie we do look forward to continuing this conversation thanks I appreciate everyone's time today thanks everyone goodbye cheers