 Okay, I think we can get started. Welcome to the last session of the day, brave people, last session of the event actually. My name is Gio Kunz, I'm working for Ericsson and I'm active in OpenNV in the testing community. And in this particular talk I'd like to talk about the OpenNV testing community, about the OpenNV testing ecosystem to kind of make all of you aware of what we have built in OpenNV over the last three to four years and advertise it and encourage that you take a look at this if you are looking for fairly complex, not complex, fairly evolved testing tools. So Emma also did a huge part in preparing this presentation, Emma Foley from Intel, but in general this whole presentation is like a joint effort of the OpenNV testing community, so I'm just here presenting it, but the entire testing community contributed to this, of course. So in general I have two goals for this presentation. The first one being, as I mentioned already, I'd like to create awareness of this fairly evolved set of test tools that exist in OpenNV, but outside of the OpenNV domain or the OpenNV community, why is that? Well I'm attending OpenStack and related events for some time and it happened to me actually fairly often, like when I joined discussions about testing tools, discussions about how to implement, let's say, resilience tests for OpenStack and I mentioned, hey, we have some tools in OpenNV, yardstick for instance, which implements some resilience tests, did you look at that? Then I got feedback, no, well we didn't really look at that, we do know that it exists, but it's being developed in OpenNV, so it's this weird telco-NV domain and I didn't really want to look at that. And I think that's a little unfortunate, so I think that despite OpenNV having like a focus on NFV, the test tools, most of the test tools, most of the test cases that we developed in the methodology around this, is actually valuable for everybody, so even if you're not working on telco-clouds, using these tools should have fairly significant value for everybody who's operating any kind of cloud. So from that perspective, the target audience here is really users who are, well, outside of the OpenNV domain or telco users who have not yet joined OpenNV. And as I mentioned, the OpenNV community spent three to four years building these tools, so there has been significant effort put into those and I guess we should just leverage those across communities. And then the second goal of this presentation is to ideally trigger a discussion on how to evolve those tools in order to enable them to address emerging use cases. So when OpenNV set out four years ago, well, basically you had your data center open stack deployment and now new use cases emerge like edge computing, cloud native computing, and this has certainly some impact on what the tools need to do, how you can use the tools, how you should be able to use the tools. And we have some very initial ideas and initial prototypes in that direction. I'd like to briefly show you a little bit what we have in this direction. And then, of course, I'm interested in getting also feedback from people outside of the OpenNV domain what you think are requirements you would like to see implemented in the tools to address your use cases. So since this talk really targets people who have not yet been involved in OpenNV too much, of course, you need to briefly explain a little bit what OpenNV is doing. And maybe oversimplifying a little bit, but in a nutshell, OpenNV is an NFV focused integration and testing project and community. So what we mainly do is we build compositions of the entire software stack, which is composed of well-known but diverse upstream components, so operating systems, OpenStack, V-switches, SDN controllers, and different combinations and configurations. Integrate all this, feed it through our CI pipeline to build deployable artifacts, and then deploy those artifacts in our federated lab. So we have different companies have contributed lab resources. We deploy those deployable artifacts using different installers, for instance. And then at the end of the day, we'll run the test tools that I'm going to talk about on these deployed platforms. So while this is basically the entire workflow of what OpenNV is mainly doing and contributing, in this talk, we're going to focus on the testing and a little less on the reporting part, although this is also quite interesting, but mostly the testing tool part. When I say we build like a full stack, no surprises here, but this basically gives you an overview of the components we have in the project. And they get integrated, as I said, in multiple different combinations and configurations. We typically refer to a stack composition as a scenario in OpenNV. And as you can see, we have the obvious management and orchestration layer, typically OpenStack, but recently also Kubernetes. We can run VNFs, tech applications on top. We have different components targeting compute, hypervisors, storage, surf as a backend, and then, of course, being a little bit more network focused, a couple of SDN controllers and a couple of different data plane components. And all of this we run on our distributed community lab infrastructure. And yeah, so the different test tools, I'll come back to this figure, the different test tools will focus on different components of this entire stack. We'll see how that will look like. Now, what is the test ecosystem, as I call it? What does it look like? So there are basically three columns here, three big areas. First one on the left-hand side is functional testing. There's basically one project in OpenNV that combines all the functional tests that's called funk test. And that is a fairly evolved and fairly flexible framework, actually, for hooking in plenty of upstream test tools, like coming from OpenStack. As you can see here, RAVStack for compliance testing, tempers, obviously, but also things like rally, patrol as a plug-in. All of these tools you know from an OpenStack perspective, but it also has more OpenNV-specific test cases in here, where we deploy VNFs, so TycoApps, using some orchestrator, and then even run application-level tests on those things. There are also Kubernetes tests in here. So it's a fairly wide range of different test frameworks that have been pre-integrated. And that, I guess, is the core value of most of these tools. All of this has been pre-integrated in funk test, and it is fairly easy to run. But this is a property of all of the tools. That's why I'm trying to advertise this a little bit more. We have in the middle then, non-functional testing tools that look at target or aim at measuring system performance and system characteristics. YardStick is a fairly generic tool that has a wide range of different tests. We will look at all of those tools in more detail. Bottleneck is a load generator. Then we have two tools, of course, focusing on data plane performance measurements via SPURF and NFV bench. And there is a tool measuring storage performance. And then all of this, at the end of the day, feeds into our compliance program. And there's another project called Dovetail that basically takes subparts of those tools and builds a compliance program around this. So we'll take a look at all of those tools in a little bit more detail. This was kind of the high level overview. So funk test. I mentioned already that it is a fairly flexible framework that is actually built around an internal framework called cross-testing that allows to hook in basically different test components and describe how to call those and make them run. We have, let's go here maybe, it focuses, as you can see here, it focuses on the API level layer of either your OpenStack or your Kubernetes deployment. And it basically treats the system underneath as a black box. So basically just talks to the API, obviously. We use it in OpenNV basically all across our CI pipeline across different stages, from patch set verification to gating the OpenNV release. And in order to support that, funk test has different test tiers defined. So starting with HealthCheck, which is like a simple health, a set of simple HealthCheck tests that eventually for tests that APIs are accessible and you can spin up a single VM and stuff like this just to make sure that a system that has just been deployed onto one of our labs actually supports the bare minimum of functionality. If the deployment failed for some reason, if you can't spin up a single VM, then it doesn't make sense to run the more involved tests, obviously. So for that purpose, there are different tiers defined. Being a functional test tool, the reported metrics are, of course, fairly simple in the sense that it's either a pass or a fail. And that's the important thing for most of these test tools. It has been pre-integrated. So it's really a simple artifact or simple to use artifact in the sense that funk test is a Docker container or rather a set of different Docker containers. So in order to not provide a huge Docker container, it has actually been split up according to test tiers. So there's a container for the HealthCheck part, for the smoke testing part, for instance, VNF testing. You can basically pick and choose what you need and you can run the corresponding tests using the corresponding container. And then, as I said already, it is built for extensibility based on this ex-testing framework, which by itself is a framework that, in case you would like to build something similar, you could take a look at because it gives you, oh, is that fairly easy to use, framework for building your test tools, aiming at also doing this across communities. So this is, of course, mainly a test framework used in OPNFV. But if you're more interested in, like, the OLNAP community could use this framework as well to host their tests, basically, in a fairly generic fashion. Good. Yeah, then going forward, yardstick, moving from the functional test to the non-functional test. As I mentioned, yardstick is a fairly generic framework that includes a wide array of different tests, ranging, for instance, from doing performance characteristics in terms of measuring networking performance but also memory performance, disk and storage performance. And it includes resilience tests, basically. This is what I was using as an initial example here. So there are a few tests in here that really target the OpenStack Control Plane API resilience. So what those tests are actually doing, they go in, basically SSH into your control nodes and start killing services. And then it basically measures if the system comes back within a predefined amount of time. So there are SLAs defined. And if the system basically stays accessible and the services recover within a certain amount of time, you have basically passed this test. So all of this is already available in the yardstick today. So as one sentence description, it targets an infrastructure verification. It also allows to deploy some simple VNF. So you can also do VNF characterization using it. It's being used, well, you can use it in your CI, obviously, to continuously monitor the performance of your build artifacts, of course. But you can also use it after deployment, of course, if you have it in your lab or in your production environment. So before you go into production, running performance tests on all those different components allows you to verify that not just that the system barely works or actually works, but that it works with the performance that you expect. That's what it's supposed to do. It's also released as a Docker container. Again, the collected metrics are performance metrics. I can't list all of them here, because it uses really a wide array of different tools, traffic generators. Then you have, of course, bandwidth latency due to these kind of things, packet drop rates. But then memory latency has its own metrics, right? The storage performance tests have their own metrics. All of those basically get reported actually to a database. And then there is, if you like, also a graphical representation of this. So you're not just measuring. You also store and report the data correspondingly. And you have a visual representation so that you can basically identify trends over time. And it is fairly extensible, of course, also. There is different tests or implementers, different scenarios. So this is extensible as needed. Moving forward, bottlenecks. So as the name already suggests, bottlenecks is running low tests on your system in order to identify performance bottlenecks. It is actually a staging manager, so to say. It defines its own set of tests, of course. But it leverages other OpenEV test tools. For instance, there is a simple test in the ArtStick that brings up two VMs and let them ping each other. Now, bottlenecks goes in and orchestrates this basically by iteratively increasing the number of VMs that get deployed and verifies that all of those parallel heat stack deployments that are actually running in the background, that all of them succeed. If they don't succeed, then this is basically a failed test. So bottlenecks, as I said, uses different test tools. It uses the ArtStick, but it also hooks into StorePerf, the storage performance test tool that I will talk about later. And it also, as you can see here, focuses mainly on using the API of your cloud infrastructure control plane. The reported metrics, past failure criteria, it also comes as a pre-packaged as a Docker container. And as I mentioned, it is extensible, obviously, in the sense that it already makes use of different other test tools that exist in OpenEV. So if you are running a test tool in your private lab already and you think that it might make sense to implement a higher-level test methodology that, and you need some orchestration to run the test for some longer period of time or with different parameterization, this could be a test tool that allows you to exactly do this. Yeah, I think I forgot to mention this. It's a load test, and it's also a long-duration test. So it helps you to verify that the system behaves on the high load, but also running for some time. So what we typically do, what you typically do in a CI pipeline, you deploy your system, you run the tests, and then you tear it down, and the next CI deployment run starts. And this already covers a lot of area. But obviously, there are a box that only appear when the system runs over some period of time. So this is also an area addressed by bottlenecks. Then moving forward, and this, of course, is very NFV-focused data plane performance measurements. And there are two tools that I'd like to mention, VSperf or VSwitch performance. It is targeting measurements of a particular VSwitch. It could be OVS, or another one supported as VPP, the packet processing component of the FDIO project. And it allows you to set up a system where you have this VSwitch or this data plane component, hook in some VMs on top, and then orchestrate using some description, a test, a traffic generator to send certain traffic patterns and traffic flows through your system. So it could be just in and out to the VSwitch, but could be something like PVP, meaning physical, into a virtual machine and out again, or even through the VSwitch to one virtual machine, to another virtual machine and out again. So different compositions of traffic flows is possible. It's already pre-configured. You don't need to worry about this. It is used for, or its main purpose is verifying, it's tuning, it's verifying that the data plane component that is being used in your system, maybe OVS or VPP, is tuned to the maximum performance, of course. So it does not care about open stack in that sense. It really focuses on the VSwitch performance mainly. This is a tool that, to my knowledge, is not available as a Docker container yet, but you have the source code available and scripts to build it and to deploy it. And of course, it has not, as I mentioned already, the different traffic patterns that you can program. So those correspond to different test scenarios, basically. You can extend this using the config files. So in that sense, you can also bring in your own, the composition of VMs, VNFs, and traffic patterns. In the same category is VNF bench. This is slightly different than VSperf because it is used for measuring system performance during production. So it targets, for instance, a Kubernetes or an open stack deployment. So here, this tool basically also orchestrates a traffic generator, TREX, in this case. And it allows you, if you have an open stack cloud, to automatically deploy a topology consisting of traffic flows across different VMs that get automatically deployed and started and then hooked together. So this is basically for monitoring the performance of the full stack, the entire stack, not just the data plane components. It doesn't really care if you're using OVS or VPP. That's more something for VSperf. This is really like, what performance do you see in a full stack production deployment? So you can use this to monitor your performance during runtime or pre-production, again, in order to verify that the system behaves as expected. This also comes as a single self-contained docker container. So TREX, the traffic generator, is already included. And you basically just need to get the docker container. And you can run it then. Finally, Storperf is measuring storage performance, specifically block and ephemeral storage on a VM level. So basically, it spins up a VM with storage test tools inside it, FIO, for instance. And it then runs certain traffic patterns or load patterns, rather, against your storage subsystem. So this is also the main use case for this tool is also after deployment of a system in order to verify that the system behaves as expected. But you can also use it in your lab, or in your CI pipeline, obviously, to verify that continuously the artifacts that you build provide the performance that you'd expect. So it also, well, as many of the performance characteristic tools, it measures the performance after some initial stabilization phase. And that's also what it reports. So for instance, it does not detect that performance actually stabilizes after some initial warm-up phase. Then this is also, for instance, considered to be a failed test. Again, it comes as a docker container. And it has a wide variety of parameters that I couldn't really list on the slide. Things like number of VMs you want to have, Q depth. The access pattern is a random or sequential writing to disk, stuff like that. And this is, again, just to mention this as an example, this is not particularly NFV specific. So I guess everybody who operates a cloud might be interested in what the storage performance of the cloud actually looks like. Good. So as I mentioned already, there is another project called Dovetail. And now in OP&V, given all those test tools, we have started a compliance program. And the purpose of this compliance program, of course, is to have a test suite that you can run in an automated fashion against commercial deployment, not the OP&V deployments. And if all of the tests basically pass, your commercial platform is compliant in terms of key APIs, behaviors, and characteristics to this abstract reference platform that's being defined through all of the tests that have been selected to be part of the compliance program. As every compliance program, the goal is to simplify, for instance, selecting vendors or selecting a commercial solution, because you basically, instead of asking a lot of detailed questions when reaching out to a vendor, you can basically, hopefully, replace a lot of detailed questions with, are you compliant to the OP&V Compliance program? And this would simplify the entire process. But also for vendors, it provides the benefit that you have some alignment across the industry in terms of what do you expect as capabilities of, well, in this case, NFV cloud in terms of APIs and characteristics. Good. Well, in Dovetail is just the tool, really. We can go to this slide. Dovetail is really just the wrapper around all of those test tools. And the test tools themselves, they do all the heavy lifting also in terms of integrating tests and maintaining tests and so on. Dovetail really just picks, select the tests, runs the tools, and then collects the test results. So in case you're interested from this entire range of tests we have in the community, the compliance program covers a subset. But all of the other test tools and test frameworks that are not highlighted here, they remain candidates for later releases. So the latest release we have was released in September. We call it 2018-09. Future releases will include more tests. So we'll aim at increasing the test scope over time, of course. Good. So this basically concluded the first part of this talk, which was mainly aiming at giving an overview of the open NFV test tools and make you aware of that those exist. Now the second goal, as you might remember, was about triggering a discussion about how we need to evolve what we have in order to address emerging use cases. And I already mentioned it briefly during the introduction. When Open NFV started a couple of years back, a typical data center or typical use case was a data center deployment, medium to large scale, centralized data center, single data center in that sense. And this data center, of course, having an NFV focus in Open NFV was running VNFs. VNFs, in that sense, are really just legacy network functions running in VMs, so nothing too fancy. Now in the meantime, of course, things like edge computing and cloud native computing, you're aware of this, have emerged. And this certainly has an impact on what the test tools need to provide and what you would like to test. So the question really is, how do we need to evolve those tools? And we have some initial thoughts on this, some initial prototypes, but also in the context of this presentation, it's also meant to collect requirements and good ideas that we can take back to the test community and the test tools to evolve them correspondingly. In terms of edge computing, of course, there are some fairly obvious impacts, but still, they need to be covered somehow in your entire CI pipeline and end-to-end test chain. And one obvious thing is the test topology. Now, as I mentioned, typically we are deploying one single OpenStack instance on a set of servers, and then we run our tests. Well, in an edge computing use case, not uncommonly, you would like to consider, for instance, a centralized data center and an edge data center, or multiple edge data centers, right? So your whole CI pipeline and tooling needs to be able to deploy multiple OpenStack instances and configure them correspondingly to act as maybe a central side or an edge side. And you need to be able to set up the connectivity between those OpenStack instances. Then once you have done this, there are the obvious networking effects that you need to somehow model, right? So latency on the control plane or data plane, because you might have tests that really check, does the system scale? Does it behave well under loss of connectivity? Things like this. How does it behave when there's just limited bandwidth? Or how does jitter or packet drop affect the protocols? And finally, of course, hardware resources. So until now we are running this on fairly standard data center servers. But edge deployments might just have a fraction of the hardware resources that we typically have in a centralized data center. So not just the number of servers, but it's also the capabilities of a single server. And that needs to be somehow taking into account when running those test tools. So there's one deployment methodology in OpenAV and that is referred here as XCI. It's actually a CI methodology. But it can also deploy OpenStack instances. And there's one specific flavor in XCI that basically deploys a single OpenStack in a single VM. Fairly simple. So it's called mini flavor. So it's actually used for debugging purposes. But in the context of edge computing, we can obviously do the following. Well, you have your single bare metal server and you run XCI ones. You get your controllers and your compute VMs. And then you run it a second time. And obviously now you have two OpenStack instances. You need to connect them somehow. And then you can run tests that we still need to implement in the community that are edge specific. Fairly obvious. Next step would be to emulate or simulate the networking environments you have in this case. So one prototype we have is doing the following. You have your two VMs. And you simply put a VM in between. So traffic is more or less transparently routed or directed through this VM. But this red VM in the middle, it has a traffic emulator inside that can add delay, jitter, packet drops, what have you. But all of this is basically, again, it's nothing new, but it needs to be done. And well, the goal of the Open UV community is to provide this tooling in order to test these systems in edge use cases. So this needs to be implemented, needs to be done. We have this prototype, but more work is needed. So again, obviously we are also asking for contributors. And if you're interested in that, grab me after the presentation. Ah, yeah. And then once we have this network, let's say, environment set up, there are efforts looking at evaluating how, for instance, Keystone as one single use case we basically picked, how a federated Keystone would behave in case of, yeah, certain connectivity issues, packet drops, jitter, things like that. So this basically covered the early prototypes and challenges mostly in terms of edge computing. Now in terms of cloud-native computing, I already mentioned that FungTest, of course, has Kubernetes tests included. But we can also apply the cloud-native computing paradigms to the test tools themselves. So that's a nice and interesting idea that I'd like to show you here as well. Typical, well, initial slide, I guess you know most of this already. The main paradigm of cloud-native computing is that you split up your monolithic app into a microservice. And you get the basic benefits. You can orchestrate this much more easily. You have reuse. You can upgrade and update single components instead of your big monolithic app. But you can also compose those microservices to create higher-level services. And this is now an interesting aspect when looking at the OpenFeed test tools. So we have all those tools that are described. And most of them are containerized already. So it does make sense to run them as a workload in, for instance, a Kubernetes environment. So instead of what we do now, basically, more or less just using Docker to run those tools in an automated fashion, but still it's just Docker, we could use Kubernetes to run those in an orchestrated environment, run them as test tools as a service, for instance, in OPNIV, but also in your private lab. For instance, from an Ericsson perspective, we have a bunch of different test labs. So to say, we have different versions of the cloud product deployed to those labs. If you run the test tools like this in a centralized fashion, you can point them to specific target deployments and then run the tests as a service. You can also do the following. As you can see here on the slide, you could use a single and new entry point to orchestrate all those tools from a user perspective. So instead of, for instance, having to learn all the different CLI commands of all of those tools, you basically get a single API that is able to orchestrate those tools or to trigger the tests inside of those tools correspondingly. And you can even go one step further. Like now, we are running those tools typically sequentially. So you start with your functional tests, and then you have some performance tests. But deploying it like this and orchestrating it like this allows you to create higher level tests in the sense that you run your functional tests and a network performance test and a storage performance test at the same time or with a certain pattern applied. So this is what we'd like to go to in the community. And as part of our transition from more traditional tool chain to applying the cloud-native methodology to the artifacts or basically to the test tools we build and deliver to users. Exactly. So in summary, obviously, I'd like to reach out to you and convince you to join the OpenViv community as such. There are a couple of entry points that might be interesting for you. There is a test working group, the group of people that are active in all of those different projects and who also contributed to this presentation. We have a Viki page that basically provides more information on this. In general, the OpenViv Viki is a good place to look at OpenViv Verified. It's the compliance program that I was talking about if you're interested in that. Check this one out. As I mentioned, especially with regards to the second part of the presentation, we're really looking for input and feedback. What are your requirements and use cases? And if you'd like to reach out to us or to specific projects, there is a mailing list, OpenViv users. There's also a mailing list OpenViv Tech Discuss. You can send requests to. And there are specific IRC channels per project. And there's even a test working group-specific channel. So you can basically use that to reach the community entirely. Yes. So this basically concludes my talk. And I have a couple of minutes left. Are there any questions? My question may be naive because I'm new to OpenNV, but how OpenNV relates to Etsy? That's a good question. Yeah. Some of the test tools, well, OK, to put it simply, OpenNV is how I see it often a good place to implement, for instance, the test specs that Etsy defines on a more theoretical level. So there are some collaborations ongoing between Etsy and OpenNV. For instance, Etsy hosts plug tests. And the last plug test that was in June last year, OpenNV and Etsy came together. OpenNV provided some of our test tools and provided some of the OpenNV platforms to be part of the Etsy intermobility testing that is going on commercial platforms, for instance. Another thing is that some of the yardstick test cases that look at network performance, storage performance, and so on and so forth, they are based on Etsy test descriptions. And I guess that does not only apply to funk tests, but I think also VS Perf is an NFV bench. They have, how to phrase this, they apply test patterns corresponding to what Etsy recommends. One more question. Because the testing tools that you mentioned, they are mostly testing the infrastructure. And what about testing the concrete VNFs or service chains? Are you elaborating on this, or is this not in scope of your initiative? Yeah, so the main scope, the primary scope of OpenNV is indeed the infrastructure. There is ONEP as a project. You might have heard of this. ONEP is an orchestration engine. It's also a project that lives in the Linux Foundation networking umbrella project, basically. And they focus much more specifically on how to test VNFs and requirements that VNF needs to fulfill in order to be considered ONEP compliant and, in general, TECO compliant. So if you have very specific NV requirements, that's maybe where you want to look at. And they will also come up with a compliance program soon. OpenNV uses VNFs to verify that the platform you run those VNFs on is actually capable to run. Those are, of course, sample VNFs, open source VNFs. But still, it should give you a good indication that also commercial VNFs are capable of running commercial VNFs. OK, and thank you.