 Good morning everybody. As Arpit said there's been I think a theme going through the keynotes we've heard so far in terms of the importance and the challenges around integration, the importance of interoperability and the ways that we need to solve that as a community. So as we sort of continue to discuss this along our journey we have a panel on compliance and verification. We have a number of different perspectives including a leading open source vendor, a long-standing interoperability and compliance test lab, of course an operator and an important standards organization that has long supported the mobile ecosystem. We have decided to actually do the format of this panel a little bit differently as you might guess from seeing this interesting setup. Inspired by the TV game show Family Feud we have divided our panelists into two teams. People will buzz in. First person to hit the buzzer gets to speak. We also are limiting answers to a minute so you will actually be able to see the timer countdown and you as the audience is actually going to act as our survey says. So I hope that you can get into the spirit of this panel. So our team blue is made up of Beth Cohen from Verizon and Henry Calvert from the GSMA. Team red is made up with Tom Nadeau from Red Hat and Lincoln LaVoy from UNH IOL. So panelists come forward. So I am not Richard Dawson but I will do my best to lead us in this game. So let's play open networking feud. All right. First question to our contestants. Given that operators will still have to do a fair amount of in-house testing given that we're not going to probably get to one size fits all for our infrastructure and given that there will still be some things that are specific to any given operator. Where do we get the business value? How do you define it in what quantitative and qualitative descriptions or metrics? So first panel sorry first question. What is the business value of compliance and verification for you? Survey says so let me make a statement first which is that open source and standards are not free. So us as operators obviously need to make an investment in you know joining these communities and the reason we do that is because it affects our bottom and our top lines in a good way. So for our bottom line it means that we can be time to market is faster because we can rely on the shared resources of the open standards and the open source communities and the labs to support our efforts across the operator community. There's a lot of hard problems that the operators have to do and sharing the burden is only going to help. As Amy mentioned you know infrastructure is hard and it's not our differentiator. So all right time's up thank you Miss Cohen. All right let's throw the question over to our other family. So how do you think about describing and either quantitative or qualitative fashion the business value of a compliance program? Do we need to buzz in? Why not because it's fun? At least I didn't say that was easy. So no it's about creating like a commonality I think right and getting things that level of interoperability. I mean there's been a huge amount of talk so far about like CNTT what it's attempting to deliver and will actually kind of allow that ecosystem to build itself up where you can actually kind of undo that siloing of a VNF and the NFVI and the direct linking that it has to have to allow the portability it's going to make this all kind of be able to grow to scale. Okay team Red has still 22 seconds on the clock. Do you want to add anything? Yeah I just want to echo some of the some of the kind of the points that you just made but there were also echo echoes in the presentations from the beginning. Matt for example mentioned this about the the advantages of this kind of consolidated thing. The other thing is that we're that's important is the integration perspective. All right so sort of two I think general themes here on the one hand open source and standards are not free but the investment does impact the bottom line on the other hand sort of the portability and interoperability of integrated systems. So survey says this is you audience team blue. You should respond to that. All right team Red. I also think survey might be saying I need more coffee which unfortunately is not an answer on the board. All right so second question contestants. We are used to in the telecom industry programs around standards based certification. We've done them we sort of know how they look as we've been looking at how to do compliance and verification in open source where some of the code is there but we've still got interoperability challenges. One could definitely argue that we need to think these programs through perhaps in a different way but also learn from what we have done in certification in the past. So contestants what do you think are the most important sort of characteristics of a compliance and verification program in open source and how is it either similar or different from what we've known in the past? This one I think actually is really easy. I mean this has been a fundamental driver of the way we've structured OVP from the beginning which is it's about the community it's about a community process and that that's everything from the the entire story arc of what is certification and what does that mean right like it's the setting of the requirements where those coming from getting the community buy-in getting the and when I say community buy-in that really does include the customer buy-in too from the operators that those requirements are going to meet their requirements and their needs it's getting the community buy-in on how that testing is implemented and designed in such a way and then it's also making sure that there is a review process and a submission process that actually aligns well with the open source and then lastly from that is who actually does the testing right so traditionally and being a lab I'm saying this traditionally in the lab it was always in a single lab whereas we've tried to move to a process where it can actually involve the entire community being able to submit the results into that program team blue I think yeah from the GSMAs had to hit the buzzer didn't you right that's going to lose three seconds from the GSMAs point of view looking at a bit of the history you know the both words that are coming out that I just love our collaboration and consistency and GSMA has always really driven the operators to be interoperable to drive scale okay so if we have consistent profiles we call it profiles in our language in in the operator community if we have consistent profiles that are actually known it reduces the number of test cases that have to be done on the boxes or on the systems or the software or the stacks or the distros that you're actually putting out there it reduces the number of case studies because they become learned and they become well understood and people can drive through that very quickly that can only happen through collaboration if you're actually doing it in a silo you're doing it for yourself you're not doing it for others and that is what we're trying to bring together with cntt is an actually understanding of that knowledge to make sure the consistency is going to drive less testing that has to be done so let's go let's actually let one more person buzz in for this question yeah just add to that one of the things that's important to add to what my colleague said earlier is that is that not only do the specs explain what needs to be tested but they can actually be implemented by the vendors so it's important that the community includes not just the the the operators but also the the vendors and they do it in a way that you know we can we can test what needs to be tested and then make it work i want to i want to add to that operators and testers operators and the vendors and the entire community has to participate it fails if we don't do that that is essential right okay so let's go out to survey says first answer that we heard was one of the important aspects of doing an open source compliance program is that it aligns with sort of open community processes and that the governance of the program is open and the test labs themselves are open answer to about the importance of the repeatability and the way to take learnings from that answer three somehow split magically between team blue and team red is that it is about sort of the involvement of the vendor community alongside the operator community so survey says answer one survey is still saying i want more coffee all right survey says answer two survey says the somehow split team answer three all right honestly i think those are all right answers and they all actually dovetail into each other if you might notice i just worked in the word dovetail which is the name of one of our compliance projects in opn fe um but that actually they do dovetail in together that the way that we look at things from a compliance perspective in open source is for it to be a more open process that we want everyone working together that we want to take the learnings public that we want testing to be able to be done in a variety of ways but that we also want to get consistency and interoperability and take things from there all right so question three panelists i think this one i i hope this one is a bit provocative so we've been hearing a lot about cloud native and i think there's a recognition that cloud native is an incredibly important thing that we you know sort of both in terms of the technology but more in terms of the process around things like ci cd dev ops etc so if we are thinking about networks is actually being moving to something like dev ops and ci cd constantly changing that the applications are going to be constantly updating with you know small changes constantly into production which i know is still a lot of times a goal for many of us in telecom um so if we think about ci cd if we think about cloud native if we think about dev ops as important drivers for the future of the industry is a badge based compliance program actually sort of the right thing for us to be focusing on um you know there's a certain assumption of stability and version control and version numbers perhaps built into that so third question is is a compliance program incompatible with or compatible with and if so how with ci cd and agile and um you know revolutionary cloud native networks um i should have guessed that actually it would be red hat it's critical given the the nature of of how people do business today in software i mean we saw several presentations with stacks of various components and testing that integration is is paramount to the thing working and um and and and so the ci cd pipeline is actually a way uh that you can do that in a distributed way with different vendors contributing uh also integrating ovp tests for example in that whole thing so it's it's really critical to to the thing being successful i think i'll steal the remaining time of um i actually think it's part of the ci cd process right and so like there there's always going to be a driving force to actually move the software through that badging program creating the value that that has there from the industry into the customers as part of going into that ci cd process and stuff like that and if you look at like that combined with the nfvi layers and what is kind of coming out of that's where we're actually aiming you know actually i'm going to because i think there was something interesting there so as the host i get to arbitrarily change the rules on stage let's let's actually give you another team read another minute on the clock to talk about this particular right so you've got to start from somewhere right like you've got a day zero thing that needs to come out so if you've got a badge that you're using to kind of set some bar on that initially you're getting to kind of a base state that you know what it is and where it's going to start from and then from there you start running your ci cd processes internally and relying on the upstream stop be doing similar and and again back to that community thing contribute back to the community so that those upstream processes start to align with like your needs and making sure that the community processes are fulfilling their roles too and then as you're pulling in new devices new vnfs if those are badged and those have um you know some level of testing coming into it before they come into your space then that even better suits your environment initially to run on top of something else that was bad so it's kind of building that complete ecosystem all the way through and the consistency he hadn't even gotten to the end of his buzzer bat that's okay extension she's stealing it i'm stealing it all right bat you have now stolen the question so i asked all the question so i would argue that the cloud native is going to make everything that much more difficult so the reality of the work of of the operators position is operators have become integrators it's not what we officially do for a living but that is what we do to support what we do for a living and the not that i'm against testing i'm not at all of course against testing i love testing but i think that um all the cloud native stuff that's coming is just going to make testing just super hard so that's the challenge you're stealing from your own teammate i love this give people buzzers and look what happened university challenge i'm used to um i think i think there's a definite once you've got your infrastructure sorted out and and you're developing on top of that then you know the repetitiveness becomes easier but it's a massive cultural change for operators who really basically do murphy's law they put it in thank you you can reset the clock you can have some more time you deploy something and you don't touch it because you don't want it to break at all which is just not DevOps not at all but at all we do that and matt brought it up he but he basically says you build your capability that you actually got and then you're you're developing on top of that and that will bring experience that will bring customer experience that bring other value benefits that you can see very quickly so i don't think we should be afraid of doing deployments and patches and upgrades every day um you've just got to have a confidence infrastructure confident in your infrastructure that it's going to work and it's going to be there yeah i was going to say that just to add to that you can't agree that it's a feud i have to agree it's a reality the fluidity the fluidity of the integration and which you're just talking about the again the many pieces parts and the evolution the constant evolution the quote earlier you know embrace it or die right i mean that's the reality and the and the cloud native makes that like you were saying the it invites even more of that and so unless we can embrace it with a consistent ci pipeline and testing strategy it's going to be difficult making that work just because it's hard doesn't mean you can skip it right so actually i am i'm i'm actually intrigued uh sort of by what you said beth uh because on the one hand so you're saying sort of the introduction of cloud native is going to make things more challenging from a test and integration perspective on the other hand if i talk to sort of a cloud native developer they actually will say sort of the things like automation and c i c d and the you know building and system testing maybe and from the beginning should make things easier um maybe we could get some disagreement once again as the panel debates um you know i'll actually i'm just going to you can still use the buzzer but but tom i mean do you agree with beth that that going cloud native will make it more challenging i i do i mean i don't i don't think we're gonna go back to a world of manual testing you know i mean the the presentations today what we're seeing in the industry um and again it's it's getting more more complicated but it's but it's getting more complicated for a reason and again because it's hard doesn't mean we can't um can't test it and and make a work right but it requires that we do that team team blue a rebuttal i want to add something so in controversy which is that at the end of the day it's about the operations because that's what we live and die on and if we don't have that cell phone work 100 percent of the time we're in trouble doesn't matter right all right we have five nine six nine s l a's and you know our operations people freak out over this stuff because they they want it to be so reliable they want to know that it's gonna work so this is this is coming down to the difference i think between like test and production versus a c i c d pipeline that's actually testing before deployment though right and how do you do those best practices go do both anything from our from our other panelists see i'm getting nice as this is going on and giving everyone a chance to speak yeah i i mean we've seen this as as as software brings flexibility into networks okay and that can can be a double-edged sword and we're we're discussing this all the same but it was the same when we brought vaulting vaulting is a highly configurable uh software application on a device and in a network effectively and and and the very different variants we got around the world is phenomenal and i've spent the last three years sorting out all those differences that's consistency if we can get to a consistent this is the consistent infrastructure we've got to provide then the other experience applications on top are going to come and the value is going to come at the end of the day but that consistency is so important so to get the base core infrastructure right get the testing procedures for that right and then heart content go for it you know on the software on the service and software layer you know fine one final question and this one we are going to do properly families feud style um what do you think is the most important priority for the sort of next thing to do for us as a community in our compliance and verification program right we are going to do the timer this time so the most important thing in compliance and testing is testing the real things so this this is the biggest challenge upstream right now in my opinion are the the vnfs and the cnfs in ovp testing and so on um are not the production vnfs and cnfs and to your point if you really want to solidly test that thing before it goes to production to meet all of your your specs we need to do that otherwise we have to do it back in a lab you know a private lab or would not so doing that upstream that's that's a key ingredient I would add you need I'm going to steal it steal it steal it time so I have lived the the vendor who you know we were in production the vendor hadn't done their job and we had to take it into the lab and say no guys what you told us was wrong and that you know that put egg on our face with our good right so just to add just to add egg on your face yeah but I mean like I think that the biggest next steps is to get the community resources in place to do exactly that to actually be able to test these real things on commonly identified infrastructure commonly identified you know stacks you know and and we've all said that that's the whole point of like what is open stack what is opnfv what are the integration projects and the testing projects that we're doing there but we're not yet doing that fully as a community right otherwise we wouldn't be putting the efforts into something like cntt and stuff like that so let's take that let's actually then you know build the reference architecture build the reference implementation to actually stand that up and then give that as a community resource to then bring in the rest of the parts and the pieces to that pipeline to actually make that happen and we can tick all the boxes at the same time that we talked about at the beginning of this in terms of like how do you do the open programs of allowing that to feed back to like a badging program so that when Verizon or AT&T or others bring in something that they're allowed all right team blue any last rebuttal or addition to that rebuttal just for beth you know i've spent 20 years in operators mainly in asia when something goes wrong it's the operator's name that's out there in the press yeah and they suffer badly that is that is the real reason everyone gets very techy about the performance that is actually delivered i think we need to collaborate we've got to collaborate we've got to set the right expectations those expectations um will lead to success and my last statement is we need more suppliers in the industry yeah we have got a drought of suppliers we've got some brilliant suppliers out there don't get me wrong but we need more diversity in our supply chain let me let me add to that we have a whole bunch of partners and vendors that we work with who aren't participating in this in these tests at all yeah that was my point that's a significant problem for us yeah all right so all right survey says um now i'm trying to remember what everyone said um test the real thing test the real thing um more resources into the community to build the tests um more vendors more vendors uh so related to resources and performance testing so survey says answer one come on come on yes yes survey says answer two survey says answer three by the way she's the customer so really the right answer is always that one all right and the host is biased now answer four okay so ending on a slightly more uh collaborative and serious note i hope that you have had fun with our format we wanted to shake things up get a little bit of back and forth i hope that you have enjoyed the way that we've set up the game show i have ideas for next year i think next year i want to play the weakest link um maybe not but you know we're sitting up here dividing on teams and buzzing in i know because it's you know because it's fun because we want to express different opinions and we want to debate but the the reality is that we do all collaborate with each other i work with all the people on this stage every day we are all really in the same boat as an ecosystem um you know to make this successful really does require all of us working as a community and so on that note we've got a little bit of time left so i'm going to give one minute on the clock for any last thoughts around community collaboration and sort of success around this in open source and we'll start with henry and one minute on the clock uh all us panels for one sort of final thought i think the activity we see in tt of course is bringing the collaboration uh and uh the the consistency together that we actually need in the testing environment it is there's a lot of effort that has to be put down that's completely we respect that but the benefits are known and they are actually are large um we want to there is a wider community we have to reach it it's phenomenal how how big that community is there are vendors not turning up uh and and and everyone needs to be talking off the same message for this to actually work a bit of unity is good competition is always our problem okay it is the downfall that we actually have so let's see if we can just take that while we develop um the the testing and the conformance plans to get things right let's just put competition to the to the side a little bit and work collaboratively together all right you ended early well done it is so much fun to hear that buzzer isn't it time is money bad uh so what i'd like to say is that um there's there's a lot of um as as uh henry mentioned there's a lot of competition that's really isn't the competition you know amy mentioned earlier yeah telecoms compete they don't compete about infrastructure because nobody none of our customers give a crap about our infrastructure they just want it to work and and we want our vendors to make sure that it works for us because as henry said it's our name on on that and you know if it goes down who called you know who do they call they don't call red hat they call verizon we're not in the press release you are i know it's exactly right so so you know we need to all work together to solve this problem because we're not going to solve it uh separately it makes no sense all right let's reset the timer for mr levoy so the buzzer is fun um i'm gonna build a little bit on what henry said which is is talking about kind of that competition and like can we create some processes in environment where we can kind of put that aside for a little bit and i'm thinking specifically around like the plug fest events and stuff that we have right in terms of like let's create the community resources that we need to be able to enable this testing create the environment where that testing can actually take place and can take place where it we try to set aside or push away some of the competition for a little bit and actually work collaboratively to solve the problems and stuff to actually build this and pull in that those that best practice that lesson learned contribute back to it and then push that back up to the community resource pool i.e the open source programs the ovp programs in the way that that testing is structured such that we're actually kind of building that complete picture that then the devop teams and everybody can pull internally to their processes at the operator and stuff that like is being done at red hat and similar right all right finally mr nadir he's taking the bucket he's going to steal the buzzer be beware if he shows up in any of your sessions later in the week these will be great for the next conference call um yeah for me it's really simple there's there's really three ingredients to this recipe right there's we're talking about collaboration uh consistency and using the real things and and those three things combined i think will get us you know the the cake we're trying to make um and and and that's this the kind of the theme that runs through everything we've been talking about today so for me it's just really simple okay how did you do all right with that i'm going to now declare myself survey says um so survey says all of y'all are awesome and great sports um we have a challenge ahead of us but i believe in the power of community and ecosystem to do it and um thank you audience for waking up and giving us some of your participation in that as well so thank you all to the panel thank you to all of the contestants for playing open networking