 Hi, welcome to this presentation about testing for NFV. So I am Jose. I work for Exxon and doing some mainly OpenFV Contributions, right? I've been there for two years and here's my colleague Trevor from Intel who is also very active in the community. And today we're gonna talk about, I'm gonna do a short introduction about OpenFV and then we're gonna talk about what testing projects we have and the role of testing for NFV platforms and we're talking about the CI, what's the role, the purpose of the CI, continuous integration, and also Trevor will talk about performance testing. So who do you know about OpenFV? Who have heard about that or knows roughly what it means or what is it for? Okay, I see that the director knows about that. So basically OpenFV, just to make it short, it's an integration and testing project, right? So we're trying to build a platform based on different upstream components and based on compute storage and network utilization, right? So we try to build an end-to-end platform bringing together all these components, working together with different communities and so on so forth, right? So this is kind of taken from the NFV reference, so you will see different components that we will be touching today in the infrastructure, right? So OpenFV is about integration. It's about open-source communities, open-source components, and open-source features that are developed upstream. So we are trying to put everything together and try to make it work with different flavors, with different deployment options, and so on so forth. So we are working, for example, with OpenStack, which is our base deployment where we add features on top, right? And of course many other things like OBS, FIDO, KVM, and so on and so forth that you will see later, right? So in short, so we have this platform which is OpenStack-based and there's gonna be more in the future, but if you would take OpenStack and we start adding things like SDN layer, for example, we have open-source communities that are doing SDN, open daylight owners, open control, and so on. So we put that together in the platform, we deploy it, right? So we have OpenStack plus SDN. Then we have different components, different flavors with different versions and so on, right? So we have now the FIDO community. We have a different obvious visualization, so on, that are also probably integrated in OpenNV and allows us to have some kind of many performance enhancements or things like that, right? And on top of that, we also have features. So it's not only about SDN components and so on, it's also about what NFV brings us, right? So we're talking about different features like service function chaining, VGB VPN and all the NFV features that you can imagine so that all potential operator will demand in the end, right? So all this together is OpenNV, right? So in the end, what we're trying to do is to build an end-to-end platform that together with our continuous integration pipeline and our testing mechanisms will kind of be the reference platform for NFV, right? So all the way from computer storage and network visualization up to the manual layer, right? So when it comes to testing in OpenNV, since we have seen that we need to touch different components, right, in the reference, so we need actually to create different testing projects, right? So we need to really focus on different things. For example, this is the ecosystem we have in OpenNV, right? So if we start from the bottom, we have, and we will talk about this later, we have our virus infrastructure, which is the reference infrastructure that we have by default in OpenNV, defining a reference in physical hardware infrastructure, so which is spread all over the globe, right? And we use this virus infrastructure to deploy our components, our, you know, OpenStack plus all the additions that we talked later, and then we run this testing, and there are different flavors of testing. Of course we need function testing, right? We need to prove that something works, so that's what functional testing is for, right? We have a project called FuncTest, and where we are trying to also integrate different upstream test frameworks, like you probably know Tempest, so we're integrating Tempest, we're also integrating all different frameworks like Robot and other things that are working together in a framework that triggers all the testing for you, right? So once we run function testing on the platform, we also need to run performance testing, and there it comes, you know, you will see a bunch of projects that are dealing with performance, because NFV is not only about function, right? So we want to have performance. So therefore we have a project called Jarstec, which is, Trevor will talk more about that later, but basically it's a project that is dealing with performance from the NFV point of view, right? So we're trying to create virtual instances, and trying to run and measure the metrics when running, for example, traffic generators and so on, right? Then we have other projects that are targeting different aspects of different components in the platform. For example, we have VSperf, and Trevor will show an example later, but VSperf is trying to provide performance metrics from the data plane point of view, right? So providing different v-switch topologies and so on, and measuring performance on top of that to see what happens. We have similar things for storage. Have a stopper, which is taking a look at the storage performance, traffic measurements and so on, right? What happens if you replace your storage backend by a different one that probably gives you more performance, right? So this is targeting the storage, and we have also the projects here that are consuming these results from Jastik, for example, to give you an explanation of what's happening. So what happens in performance is that, okay, you run your tests, and then what? So you have to understand what you're testing. You have to understand what the results mean, right? So therefore, we have a project called QTA, which is trying to provide a quality number, so consuming these results from Jastik, which means that, okay, I have this performance metrics, I have these results, and I have to understand them. I have to say if it's good or it's bad, right? And that's really a challenge. So we'll talk about that later, but that's really the challenge here. And at the end, we have bottlenecks, which is a project trying to find, as the name says, bottlenecks in the platform. That means that when you're running performance testing, you need to know stress testing. You also need to know if your network is behaving as it should be, or you have a bottleneck in the storage because whatever happens in the platform. So that tries to find these kind of points where something can go wrong, and we'll give you some information about how to fix it later on, right? And here, we have something very interesting that is part of our testing ecosystem, which is this databases on top, right? So we have different databases in the community where we are trying to store all the results that we run every day on our continuous integration pipeline, right? So whenever something happens in CI, whenever there's a deployment and testing, we push all the results to these databases through this test API that we have also here, right? So why? Why do we want to store results? Okay, you can say, okay, you just run it manually and see if it's working or not. But if we are storing all the results, we can keep the history of how it has behaved during the last months, for example. We can show some kind of improvement throughout the time. We can also show the status of what the deployment and this point of time is, right? So we can also use it for release, which is actually what is it for. So in our case, we are basing our deployments on or we are releasing our deployments based on these kind of results that are stored in these databases, right? And one key thing here is that we go all the way from basic health check up to service. So what happens is that first, you need to test if your system is behaving good. So you run a health check, right? So if that passes, you go to the next stage, go to the smoke testing. You run smoke testing, a couple of test cases, a couple of basic test cases to verify that everything is correct, right? And then you start testing actually the NFV features, right? So that's where we have here the third tier. We call it tiers, could be also categories, right? So we test the features that we want to provide with OpenFV, right? And of course, features and components go together. And then once that works, so once you verify that that's working, you start measuring performance, right? But not before. And of course, we also have tests around VNF, open source VNF and so on, right? Good. If you have any questions at any time, you can just interrupt me. So what we have here is that in OpenFV, we're trying to provide different flavors of different types of testing. So you know about functional. So for functional, you can test anything in your platform. You can test infrastructure. You can test features, different components. You can deploy a VNF to see if your manual is also working, right? So you can combine many things. Of course, there's more and more. But functionally, basically telling you if something works or not, okay? It works, it doesn't work. That's it. But then we have also performance. As I was saying, we want to measure performance of everything, right? It makes sense to measure, run traffic on network storage compute and also from the virtual layer point of view, not only from, you know, pure hardware. So we just want to measure how the virtual layer behaves, right? So you have good performance and so on. We create instances. We run traffic generators to see how it behaves, right? The same applies to stress testing. So that's how my, yeah. How long my system can resist when I am stressing really network, for example, running really high traffic demand to the network to see how it goes, right? And it's not only that. So we have many, many things in the NFV area that if you want an operator who wants to deploy a VNF, so of course they need to have security, they need to have upgradability, scalability, backup and restore, right? So all these things are kind of the target of testing for NFV, right? So I didn't display all of them here because there are many, right? And the customer always demands certain things. It's not always the same for everyone. So we try to target everything. It's a challenge. But we're getting there. And I was talking about flavors before. So since we're integrating different upstream components, we're putting OpenStack plus SDN, for example. So the way we're doing that, it's what we call as scenarios, right? Or you can also call it or understand it as a flavor. So flavor is basically for us, or scenarios for us, just a way of selecting different components that we want to deploy together, right? So we have, for example, a super basic scenario. It could be pure OpenStack, nothing else. So we are deploying and testing OpenStack. But this is not what we want, right? We want to build an NFV platform. So in the end, we need to combine different things. So we want to have SDN, for example. So we deploy OpenStack plus Open Daylight. And you can imagine, okay, so that's something you can expect the same results as if you have the basic scenario with OpenStack. Things always go wrong. So when you combine things, you start seeing some failures, right? And that's a cool thing in OpenFV is that once we have this deployed and we test it and we see the failures, compared to the known SDN version, for example, we always try to report the bug fix, the bug, sorry. And then we try to work upstream also to fix it if we can, right? And the cool thing is that whenever we have a fix, it gets deployed and tested on OpenFV, right? So that's one of the advantages of working also with other communities. So we get the bug fix and we test it. We just apply it in OpenFV, we test it. And that could go even more complex. We can have SDN and we can have also different features that come with SDN, for example. This is just an example, but you can see probably a different v-switch that comes by default, an NsKVM version, including SF Service Function Cheney. So many things combine together. So it's a chaos in the end, right? You need to test everything, all the combinations. So you can see three here, but in the end, I don't know if we have up to, I don't know, 40 different scenarios. I don't remember the number, but it's quite a lot. So we need to test everything. We need to make sure that everything works together, right? And therefore, we have CI, right? So we have continuous integration that is trying to make all these things happen, right? To test everything together and deploy everything, all the combinations together, right? So in OpenFV, we have also what we call federated forest labs, which is a bunch of small data centers distributed all over the globe, right? So we have different data centers provided by different companies and different vendors. So everything is kind of different hardware, but we try always to follow the specification, which is one jump server where we trigger the deployment from, and five other servers where we deploy three OpenStackController nodes and two compute nodes, okay? So this is kind of the forest specification we have for OpenFV. So all the CI resources where we are validating our releases are run on this hardware, okay? Any questions so far? So what role plays CI here? So that's very important, because whenever there's a new patch or a new fix or something, it has to get tested, right? So therefore, we have all these jump hosts that I was mentioning before connected to Jenkins, our Jenkins server. And what we do normally is that we have our deployment ISOs, right? So basically OpenStack plus all the additions, they're basically an ISO which is built before. We download it, we deploy it on these forest pods, and after deployment, we run tests, right? So you see that here we're pulling a Docker image. So we are basically building our frameworks, our test frameworks on Docker images. So this Docker image gets downloaded and automatically Jenkins will trigger the test streets, right? On the previous deployment, of course, right? And as I was saying before, FungTest is about function testing, and after verify that everything is okay, then we run also some performance testing, right? Which is some value that we want to add to the platform, right? So this happens like every day, and we have loops for all these flavors that I was saying before, all the scenarios. So we're deploying all the possibilities that we want to provide, right? So that happens every day again and again. And so if there is a new fix, a new bug fix or something, it will get tested, right? So like if you had the fix today, it will be tested maybe tomorrow, even today, right? And also what CI brings us is that whenever there's a batch set here in Garrett, right, we have some verification jobs, so which means that we want to have a healthy CI, we don't want to break it. What happens sometimes is you have a new patch, and then you kind of destroy the deployment or the testing, right? So we try to have this kind of controlled, it's not an easy task, but we try to have some verification jobs. And so if you can say that the new patch will not break the CI, it will be plus one, right? And then Jenkins, for example, next step is that Jenkins, whenever there is a new change in the repositories, right, it will detect a change and it will build a new ISO or a new Docker image. So it's constantly checking for new things. And whenever there's a new thing, it will build a new image, right? And that will get deployed later on and tested, okay? So that's kind of important because the code is continuously evolving. So we are, you know, writing code like every day. And of course you want the new code gets tested, right? So from the installer's point of view and also from the testing point of view. So we are also building frameworks in OpenFV. So we're building deployable artifacts, but also frameworks, test frameworks. So we always want to test a new code. So that's why this is very important for us, right? Can be quite a lengthy process. Is there any best practice that you just covered about probabilistic forecasting where you say, oh, this is the patch, I should probably test these three things first in order to have faster cycles? Not yet. It's there. Something that we have talked about is promotions, right? So if you can, for example, if there is a scenario that can, you know, you can prove that it works always, it doesn't make sense to run test again and again, right? So you want really to test the new stuff, right? So that's not in place yet, but it's in the pipe. Yeah, that's a good question. So we try to not test or not to run complex test if the basic tests are failing, right? And yeah, I mean, especially for function testing, right? So it doesn't make sense to run, you know, VNF deployment if you cannot create or ping two VMs, for example, right? So that's covered, actually. Yeah, that's why we had this kind of... Yeah, I mean, if we had this kind of deployments, of course, we would need something in place like this. But our deployments, you know, or our Farrer's labs are targeting five servers. So we are not kind of doing that work. We should, maybe, but we don't have maybe enough hardware, you know, together to make that work. But, of course, this makes sense. And we have to try to not to test things that don't make sense to be tested. Or if something is already working, why should you test it again and again and again, right? So that's why we have this kind of idea in place for promotions, right? Or for, you know, making CI more kind of efficient, right? They cycle. So we have kind of a big list of scenarios or flavors, right? And then they get tested one after the other. So yeah, what happens is that sometimes until you're a scenario, so we have a scenario owners or feature owners and they want to deploy their scenario to run the test cases and see if it works. And probably you have to wait one day to see, you know, to see that happen. So maybe one of these cycles takes one day because of the, I mean, we don't have infinite hardware, right? So we have to be maybe more efficient to manage this, as you were saying, right? But yeah, we have this kind of loops or cycles, we call it. We have daily loops normally. And then we have, we try to have weekly loops to test more advanced suites, right? Like more, maybe performance or trying to deploy VNFs or, you know, more advanced testing. And this is also very important. So that's my last slide. Try what takes over. So this is about, so we are different communities. We are open FV, and then there's other ones, of course, OpenStack, Open Daylight, FIDO, OpenO, and others that were, you know, we could benefit from each other. So the thing is that we are consuming OpenStack, we are consuming Open Daylight, and so on and so forth. But what happens if we try to, you know, get the latest pass from Open Daylight, for example, and we test it in OpenFV and we provide feedback to the community, right? So, I mean, that's something that it's ongoing. It's, we are already working, started some work around Bifrost in OpenStack. Also, we are running some Open Daylight testing in OpenFV and also trying to, maybe for a patch set verification from OpenDaylight could be run on our CI. So these kind of things of cross-community testing helps us to, you know, speed up the process of integration. So if we have this quick feedback, we could probably react before, right? Instead of waiting for a fix which never happens, right? So this kind of collaboration is very important also and it's just started, right? So we are still, we have a lot of work, you know, ahead of us. We still have to do a lot of work with all the communities and we started that already and we hope that it will work out because I think it's fast feedback for both communities or for all the communities, right? Especially when it comes to integration. So I will pass this over to Trevor. Sure. Thanks, Jose. Okay. Okay, so I'm going to focus on performance testing. It's obviously quite a vast top subject scope but this is kind of the flow. Hopefully you can get through some of these these areas, the meanings of performance testing in NFE, some of the approaches, the tools have actually got a few examples, like two examples of different tools and then two examples of actual test results and then sort of more forward-looking, what are our ambitions? And so really I think in this presentation I want to make two main points. So the first point I'll make at the beginning and the second point I'll make at the end. Obviously performance testing is really important from the perspective of being kind of the basis for competition. So comparing vendors A and B and then we need to ask sort of basic questions or answer questions about say can the platform deliver the required networking performance for a specific network service but it's also very valid to do performance testing of components like the switching technology or comparing offloads for acceleration or different aspects of the hardware platform itself or the NFEI which is the hardware, the virtualization and the VIM or what about the VNF which is the application or is it the full-scaled solution? So these are all I think valid kind of meanings of what when we talk about performance testing and then there's all the tooling and the methodologies and I think if you look traditionally so it's a situation where we had very fixed function type network elements and we have people who test experts and we can give them those boxes and they can use very trusted kind of proprietary tools and they can go off and test those those functions and tell us if they meet the requirements and those I think those methods and tools are still valid and useful but they're not really sufficient and so if you look at you know why is that the fundamental thing that's changed here is that we've now taken a general purpose compute platform and we are putting it into a heterogeneous compute environment why is it heterogeneous because the network has shaped the different requirements in different parts of the network different applications need different resources all of that means it's not a homogeneous environment and that really complicates things a lot because it means it means really that the system configurations are very important and they're complex and everybody's configuration is different it's really that we see two of the exact same configurations and it's not just that but their test infrastructures also differ and then you know the methodologies of testing a virtual network function is is different to a physical function so I think my first point that I want to make is that performance testing tools and methods are not just for developers and test experts so in the past where we sort of gave everything to the test expert to do performance testing I think everybody has a valid reason a valid perspective for wanting to do performance testing whether I'm a developer or whether I'm a user and so that means we need tools we need methods and there must be open source so that we sort of cater to the needs of all these different sort of meanings and and viewpoints of of performance and why I'm why one type of a you know user versus a developer is doing performance testing okay so they're very they are of course different approaches and valid methods for doing performance testing so you know first off there's always a system under test there's always some kind of a workload and there's always some kind of a stimuli so it gets confused are we talking about vnf testing are we talking about nfvi testing pre-deployment testing is obviously a very different thing to testing in service where my test tooling is essentially integrated with my with my platform then there's the topologies the different software versions which may be reasons or restrictions of why I'm choosing those different versions and the configurations of those versions like we put a lot of focus on data plane testing because of the network and performance of the network and all the switching technologies but of course if we introduce the control plane and the interaction between the control plane and the data plane that adds a huge amounts of complexity not just to the actual system under test but to the test environment itself and the tooling I need to to to do that testing then it's also you know is the component like maybe a switch or some sort of a subsystem like say a storage subsystem and then there are different traffic profiles and workloads which which I've mentioned the other point I think is that this one about deployment automation versus control so I don't mean in terms of the tests the test is set up but I mean the deployment so if the deployment is highly automated I take I use open stack to do that deployment that takes away quite a lot of the control I have of the configuration the topology the options available to me for that that test setup so that is something that has to be considered then the test objectives so the the table on the right at the top so you know before the VNF virtualization or NFE testing there was the the three the three by three matrix very well known matrix showing speed and accuracy and reliability against activation operation and deactivation of the service so we've we've added scalability onto that so that's how we think of our test objectives and we measure coverage from a performance perspective then the test methods and metrics themselves so just mention so there are you know quite a lot of people have already put quite a lot of thought into this they are the IETF internet drafts which are useful and relevant and specifically aimed at at NFE testing so just just put down ones that I know about there and then the Etsy NFE testing specifications so there are at least three NFE test specifications have been published redeployment testing interop testing and on VNF snapshot use cases and recommendations and then but then there are also drafts which are being developed and those are available on an open site Etsy site and and there's also a issue issue tracking tool which is available for anybody to go and give feedback and make make comments on those specifications so encourage everybody to to at least have a look at that and if you're interested in a particular aspect you know make some give some feedback about it okay so talking let's talk about the performance testing tools so as Jose mentioned he mentioned the Pharos infrastructure so I think if we talk about tools it's not just the tool but we have to think about how it fits into the overall test infrastructure and and so we we do you know today we do we do testing in in in our in our lab environments which are both geographically dispersed and and technically diverse and and and we want to be able to also do those tests not only in different different hardware environments in those labs but across different labs and those tests actually running across different testing environments then as far as the test frameworks I think they they you know that I I'm aware of the many layers I would say that sort of address many layers of testing so I think you know ultimately what are we trying to do we trying to test the performance of a network network service and that would include a full MANO stack so I would like to be able to say take a a test workload or it could be a real VNF and I'd like to tell my MANO stack look take this go and run these this test suite and then return the results and the MANO stack does the deployment of that that that test workload or that VNF and collects the KPIs and returns those to me so to me that's sort of the sort of the top layer then there's you know often we refer to as VNF characterization which to me means now I have my workload which is it's a representative of of some some kind of a VNF and it contains a control plane and a data plane and in fact we have a project which is just started in IPNFE which is about creating developing sample VNFs which can represent real real VNF or network services for the for the purposes of testing the infrastructure and then there's NFEI which I think today is very much focused on data plane performance I'll talk a little bit more about that one of the projects for doing that then there the components and the subsystems I mentioned that maybe it's storage subsystem well we have a testing project for that v-switch testing etc and then there's the idea that we can do some iterative configuration of the of the platform deploy the platform and we can analyze the performance and learn something about it be it for understanding a bottleneck or issue or optimizing performance and then there's the idea of analytics where we post process test results from the the CI pipeline and we able to take that data and essentially create information and create interpret that information and come up with some sort of wisdom about what what that means and then I think finally there's the there's more which is more like the in-service testing where we've integrated into the for statistics and events monitoring and they are we have projects that are working working on that as well and then also just I would like to you know say that so traffic generators so we that I'm aware of there at least you know seven or eight different traffic generators which are used by people in the community and I think the assumption or is that you know one traffic generator is another like another traffic generator which is really not the case so traffic generators all have different capabilities accuracies and suitabilities and they certainly vary in their complexity and cost and one of the things we're doing is we we're doing and comparing traffic generators to really understand what are the suitabilities and be able to show the the the variation between those and I think you know so there's there's obviously the proprietary tools like Ixia Inspirant and Xena which are hardware and also they have virtual versions of those and I think there will always be a case or a need for those but more and more I think the open-source tooling is becoming more important and I'll say say more about that okay so here is I wanted to just explain two so I'm just going to I'm not going to go through all the details but you know I'm happy to discuss it later but two two two projects which are performance projects in opnfe so this is the first one I'll talk about second one is the yardstick project that has I mentioned so this is called the vs perf project and it's a modular test framework combines a traffic generator virtual switching vnfs and network configuration test cases so the thing about this is there's no open stack involved so you'd say okay well what's the point of that well the point is that once we introduce open stack as I said with automation comes lack of control so we lose a lot of control as to what are the versions how the of the components how those components are configured what the topology is so with this tool we able to essentially specify through a configuration file exactly what version of v switch of dpdk of qmu kvm and then we able to and we're able to create a test vm and we can then the tool will essentially go and source all the from the repose it will do all the builds it will do the deployment it will run test cases and test suites and then it will come back and give us give us reports so it's it's it's it's automated and in fact this these tests run on a on a daily basis so I'll just if this works are you connected to the internet so maybe just show you what can I show you here um results that's what I wanted to do so if I look so if I look here I can see so this is my showing test results from um from my daily builds up here so if I click on this one it's saying this is my daily danu build and I click on that and my last build was this one if I click on that and then I can actually look at the output um and if I have the console and if I go down I can see okay these are these are this is the output of these tests okay so it's showing I can scroll down here um printing these out um so one of the things we're doing is um now looking at okay how do we how do we create better visualization for all of this test data that's been generated so it's more consumable and and more useful and then we have a prototype there was a prototype done but I think that's an area area we're really trying to focus on yeah so this uh this environment where this is actually running now it's running in a in a opnfe lab in oregon it's one of the pods um it's it's well it's specified in my configuration so I'm based on on well this first of all there's the versions of those components that or the the pieces that I'm it's kvm it's cumulus it's ovs dpdk this in this version and then there's my topology of how my how my virtual network looks so I have one vm two vm three vms I um you know my traffic is flying through these kind of ports I'm specifying the ports and so each one of those will consume certain resources and there's I can I guess control at um at a pretty fine level how those resources are allocated with this tool which with open stack I cannot do so that's that's really I guess my point here is I'm able to really sort of decide exactly what I want to test and it's very but it's very much data plane oriented network performance so as contrasted to say the yardstick which is assumes that there's a open stack environment so my my starting point is I have an open stack um environment and now I can I can uh do my um specify my my tests uh or test configurations in a yaml file which has passed converted to a heat template and then is deployed through open stack and then I have these scenario runners which execute commands in that vm that that has been deployed and run the tests the test code that essentially is in those um in those vms and then the output is written and stored in a database and then again there's a visualization which maybe I can show um which looks something like this okay it's lucky so again this is just oops what does this mean okay okay okay so there's some you know there's some kind of a visualization and again I think this is very much a work in progress okay so yes so that yes that's showing performance over time in a timeline okay so um it's a good question you ask because I'm gonna I'm gonna now talk about colorado versus danu okay so this is an example of a test this actually isn't using any of the test frameworks but I went and I wrote I ran this test it's a tcp vm performance test and on the left here I ran it with colorado using fuel to deploy my platform and that's the result I got 1.4 say I then went and I took um some newer versions of the components and I used cola to do a deployment and I got a pretty different result like 10 times different so the question is why right so I think I don't want to say the point here is not to say colorado is bad the point here is to say the explanations for why I'm getting this data is really really important and it has a lot to do with my configurations and it turns out that the explanation for this is that in the colorado in this in my colorado configuration tso offload was not available to me which I didn't really understand or know at the time now if I'm a developer maybe I understand all that stuff but as a user it's very difficult for me to know that or figure that out okay so so yes so what about danube so I've graded out because I didn't actually run it yet but by having the explanation by doing this kind of testing okay and then being able to explain it then we can make sure that the performance in danube is going to give the results that we want right under scenarios that we care about so again what's important here it's not just the data it's the explanation of why I'm getting that data okay so here's a second example which is actually also is with colorado and this is a so it's a colorado was actually also a fuel deployment and if I do a fuel deployment I end up with a essentially a single logical tenant network and two OVS bridges which are deployed by default and so that topology puts certain constraints on on my on my performance and on my configurations so in this case the the test data is showing on the left so this is identical two identical systems under test the difference is that on the left I'm varying my source MAC address and on the right I'm not varying my source MAC but I'm varying the UDP port so again the data looks pretty different so here again I'm not trying to say this is good or it's bad I'm trying to say the configuration makes a difference to my results depending on what I'm trying to test and my test objectives and I really have to understand what those configurations mean and again I have to have a good explanation for why does this data look so different and and that's really really important okay so this is a picture which is showing or trying to show testing act or performance testing activities so starting on the left so under the heading of developer testing so I think with develop you know I'm in the development world of activities and typically we have CI and it's testing on the master branch it's the latest code and we're running these performance tests with the CI toolchain but as a developer also sometimes I want to go and look at a previous version previous release and then by the way I also want to try and predict the future and say what do I think the performance is going to be right so I want to change change a version of something or make a patch and I want to try get a clue as to what's what's going to happen so that's really important and then of course I'm also doing more standalone kind of testing like I explained earlier maybe I'm it's about v-switch or patch kvm patch or something that's another very valid kind of performance testing that I'm trying to do so so what so then if we move you know one step to the to the right um under the testing artifacts so what comes out of all of that I think is it's the test tools that we develop it's test cases it's all the data but then also it's what I'm calling performance reference scenarios so we should be able to come up with meaningful scenarios which which are we can use as references for performance and and the configurations that go into making those well configured scenarios with performance ranges of what's good performance and what's bad performance right uh yeah I think yeah I mean scalability absolutely so for me that I sort of consider that as like kind of a stress test so I have this basic level of performance and then I I somehow stress my system which can be a scale my system by adding you know additional vnfs and whatever it is so I think that is a very important dimension of of this absolutely so if I look at you know then under the column of release testing okay that's you know on the stable when we do a release it's a stable we have a stable branch and then as as explained we do um the concept is we do integration we do continuous integration but we also do continuous deployment and continuous testing and we testing in these diverse environments which will greatly improve the stability ultimate stability of of the system right but now but now let's go you know to the far right where it says user testing so now I'm a consumer of all the stuff right of the platform uh my perspective is is different again and which and here I think um yes maybe the tests I really understood I'm really interested in maybe they're specific to vendor or to to certain they kind of my test and then we have the opnfe test suites which I can run but I think we also um as a user I would like to be able to in a very easy sort of automated way be able to test those performance standards or is it possible that you know we can say okay here here are some performance standards and as a user I can easily go and verify that my my deployment can meet those standards um so for that to happen I think it has to be very very easy to do I can't have to set up an extremely complex test framework and be you know an expert tester I have to be able to literally press a button or go and deploy a script and run with this in this VNF and be able to run those tests so so that's I guess it's you know I'm it's partly a question you know is this is this valid you know is this desirable is it feasible that we can reach that my vendor you don't know anymore okay so I super outperform on these six and I'm really slow on these two that's probably not important until you realize that those two other ones you actually wanted to be and right but right shows like yeah yeah yeah and and you know so it's always simplification always leads to people they are just going to run the test on time right but okay so very valid points but maybe if there's any place that where we can kind of do this would be maybe in the opnfe because it's kind of vendor neutral right so maybe there are some yeah general valid tests and maybe we can provide some some stress tests or or customizable aspects of that and and and it's done in a vendor neutral kind of way so I'm not sure but it seems like that's something that would be a very worthwhile goal I think yeah yeah yeah yeah yeah yeah yeah use the scenario so I don't just stupidly like run something we call it's a wonderful thing right right right yeah I mean very valid I do think this is possible but it's to be seen you know how how much we can how far we can go with this but I've just seen so much how yes I mean they develop you know all that all our testing is we produce all this data we have these complex tools and it's very valid but as soon as I'm not an expert test I'm not a developer I'm more one to go and try out this platform I want to deploy it I want to know something about is this within the range is this a good configuration is it a bad configuration it's very very difficult to to ascertain um so I think you know that really leads to my second and final point which is really you know data is not enough we need good data but we need good explanations for why the performance is either good or bad if we have that that will lead us to our good configurations that will lead us to good performance yeah I think I think maybe what does go with this is some ideally some kind of debug capability right for that performance but I think it all starts with the good explanations because even on the analytics I don't believe that we will be able to do analytics of the data until we have the good explanations because we don't know how to program the analytics so that's what it all starts with and then when we smart you know we smart enough then we can start becoming much more automated um have with the analytics and and more kind of predictive in in in what the performance is going to be and more smart about okay my configuration is not good why is it not good what's what's wrong yeah yeah yeah yeah yeah right I have no idea I mean what I what I feel is if we do use if we do this stuff and we show it really works we will figure out how to deal with those kind of issues that's what I feel just was hoping the idea solution would be to plug my application into that into the scenarios and then you know give more color on the data what it actually is right yeah I haven't thought too much about that so my my concern comes from the fact that I use the right now for feedback which makes it a part of the application so it's like okay that's two no's right yeah yeah okay so I think that's about it so I think in conclusion Hasey would you like to so yeah so you realize that NFV is more challenging than a basic cloud solution right so if you take a open stack out of the box and you test it and of course if you want to to have an NFV platform that contains all the features that you as an operator want of course that's one that's not the same thing you need to run through different processes different testing tools or different types of testing right that makes things very complex right so the requirements are different and complexity is higher than a basic cloud computing platform so to say right so you can say that just working is not good enough it means that if something is work that's that's fine but you need something else you need to that you know performance you need to show that it really it's really good so it's not just that it works and that's it right and around you know areas of evolution so of course what we talked about before we need to evolve ci right we need to kind of work on promotions we need to work on making ci more efficient because we have a lot of flavors we have a lot a lot of combinations a lot of way of deploying things open stack plus all the other features that we are adding on top of that so we need to improve the efficiency of that right so and of course this cross collaboration you know reusing things that we are producing in open evi and using them in other upstream communities also very important to provide feedback right and of course we also need to work on things like as Trevor was saying performance testing automations so we need to understand results so we have plenty of results stored and we show graphs on open evi dashboards but we're lacking the understanding of all of them so we need to really work on that and once we know how to do things manually we can maybe automate things right so I wanted to add to your comment that it's really difficult to say if performance is good or bad when you have different ways of deploying things and we you have you know different hardware where you're deploying meaning that you cannot you can have deployment with 10 compute nodes you can have deployment with 1000 compute nodes it's really difficult to find a way a common way to to save something is good or bad performance wise right and I think that's it right we have more questions in the meantime so I just wanted to to announce that there's going to be a reception this evening right at the living stadium so we're going to have fun you need to register first so this is the real call to action yes that's how to make things work just go to have some beers and yeah you can play basketball maybe okay and RSVP is required but in a few websites have you integrated some way of sort of collecting statistics about what are you running the test about like interrupts and the CPU load and the caches and Q-lengths and things like that do I have to do it by hand no there are I think so there are we do have some methods and tools for doing that but I'd say it's again it's very complicated and very fragmented not very easy to do I think so if you look at projects like barometer and bamboo right those are all about will be about well the first thing is standardizing those metrics because it turns out I mean a lot of those metrics are even collected today but they're not standardized so let's say something like CPU utilization there's no standard definition for what that means so there is a work item maybe castan knows more about it in the it's any fear about standardizing metrics right in the but but the good thing is that I think there's good collaboration between it's it's an FV on that and projects like barometer that are going to implement those specifications of those definitions to be able to provide results of those sort of detail like metrics per interface on interface and at the platform level in a in a in a very standardized kind of way and again I think where to come back to your question it's I do think we need more open source tools I mean a lot of it I think is either it's proprietary or it's very it's not proprietary it's very it's very manual so I really have to understand how this tool works and what it really means to be collecting that metric so I feel there's a lot of work that can be done to improve that situation I want to do a remark as well so when we are talking about performance how to measure performance it's really a challenge for us for example when it comes to storage performance right you cannot just give numbers of running you know some traffic for five minutes doesn't make sense right so one of the challenges we have is that we would need to run storage traffic for I don't know one day or half a day depending a couple of days and I mean we have limitation of course we cannot run that like every day and that's also one of the challenges when it comes to performance right so CI resources limitations right so something we need to address and it goes together hand with hand with making CI more efficient right but of course yeah if we had more hardware we had you know a lot of bots that we can deploy things on that would be perfect but I mean that's there's some cost behind and it's not always possible but that's real I think a real challenge also we're seeing in openfv we have some combinations maybe not all of them but dpdk for example is there and and sorry for example I'm not sure we are testing that very configurable because once we do the deployment we have all the scenarios so we deploy the scenario which could contain vpp for example right and and we run tests on top of that so yeah it also depends on the yeah we don't have this kind of comparisons right or at least when we deploy our hardware it's kind of static so we don't run you know changes throughout the time to see if it behaves or it performs better with hyper threading or not or things like that we don't sorry based on our specifications so I think that's not kind of we don't have a rule about saying you have to disable hyper threading or not so I think it's something that could be interesting for us right so we have this kind of yeah that's the problem so you need to administrate or changing these things right and and you cannot automate that so it's it's kind of challenging as well but it could make sense it's a valid point yeah of course okay that's it I think yeah thanks everybody thank you