 Okay, good morning everybody, and welcome to the first session of this morning. My name is Chris Hodge, and I am the interop engineer for the OpenStack Foundation. And with me today is Catherine Diep, who is the solutions architect and performance engineer at IBM. And today we're going to be talking about how to configure your cloud and Tempest for interoperability testing. So why test, right? OpenStack already has a gate, and it's already thoroughly tested. So why should you test your own cloud? Well, when you deploy your OpenStack cloud, you actually want to make sure that you know, you want to know if your cloud works as you expected it to. I've seen this before in my own installations where everything looks fine. I can log into the console and I can boot up machines and I can do all sorts of things. But there are services that say they're enabled, but may or may not actually be enabled. And I don't know, they're actually not working until a user comes up and tells me that they're not using. So testing helps you determine that your cloud works as expected and all the features that you want to be there are actually there. It's also good to test your running cloud to make sure that it's still healthy. You don't know if services have come in and out, if something has crashed. And testing gives you an early warning on when something isn't there anymore. And so it's important to kind of learn how to test your cloud and the sort of tests that you want to run and the expected results. This goes double for an upgrade. If you're going from Icehouse to Juno to Kilo, there are certain things you want to do to make sure that your upgrade worked as expected. Again, just because you have Horizon running or your APRs are running, doesn't necessarily mean that your services are working as you would expect. But one of the reasons, and this is also becoming a more important reason for all sorts of users, especially those who are selling a commercial open stack product is, does your product qualify for a logo? There were recent changes to the foundation bylaws, which now actually require that your clouds pass some sort of interoperability testing if you want to attach an open-stack powered logo to it. This initiative is called Defcore. So Defcore interoperability in you. The committee was formed during the OpenStack Icehouse Summit in Hong Kong by board resolution in November of 2013. The Defcore committee has established principles and artifacts that are required for clouds to be able to qualify for the OpenStack logo. This includes a designated sections of code. And so to be called OpenStack, you actually have to be running OpenStack. Most notably, the designated sections cover Nova and Swift. So you need to be running Nova and you need to be running Swift. It also defines a set of must-pass tests, which are meant to measure different capabilities. And so a test might be testing something like, can I boot a virtual machine? Can I destroy a virtual machine? Can I create a Swift object? These are the sort of things that we expect for clouds to have. But one thing to keep in mind is that these definitions are actually done in a community-defined process. We want to seek a lot of feedback from the users, from the community of developers, and the community of users, from the community of vendors, and make sure we incorporate all of that feedback into how we're defining the process so that how we're defining OpenStack reflects the needs of the community. One of the ways that we do this is RepStack. And I'm going to have Catherine talk a little bit about that. Thank you, Chris. So, now you know that you need to test. And now you know that DevCords already defy a set of criteria that you need to measure upon. One of the things that the RepStack project is for is to realize all the goals that were set forward by DevCord committee. And by providing the tools that will help this journey to be easier. So, basically RepStack implement all the use case that was divided by DevCord. With that, RepStack, there's three parts of RepStack. One of the first part is what we call RepStack client. It's actually a wrapper for the test tool that will be used that you need to test on. Currently, the test tool is we are using Tempest. In the future, it can be evolving with using many other type of test tool. So, therefore, we have a common point, RepStack client, to wrap around all these test tool to give out the uniform result that can be interpreted consistently across all the tools. That is RepStack client. And with that, once you collect all the data, what do you do with that data? So there is a RepStack API server that, in the future, will be probably in Infra, setup in Infra. And all these data will be sent to that server. That server will be a repository for all these data. And then all the analysis tool, and et cetera, can be done there. What is useful with all these data and nowhere, not a good way to look at it? Therefore, we have this RepStack UI. There's different aspects of RepStack UI. At this moment, RepStack UI, or when you send the data, we collect data anonymously at this moment. So that all data, no one knows who that data belongs to. Only the sender knows that data. What we want to do right now is to collect the statistic. Do data analysis of what are the most basic common element or component or implementation that are common for the whole ecosystem of OpenStack ecosystem out there? So with that, later, we'll have a demo at the end of that. We'll have a demo of the RepStack UI. And we can walk you through how all the artifact that was defined by DevCord and how does it imply to your data. And with that, let's get into the first set. So now, installation. As we say at this moment, the test tool behind RepStack is Tempest. Tempest is well known to the technical community. So for installation, if you would like to use Tempest directly today, you can. And therefore, you can install Tempest manually using the Tempest instructions. However, in RepStack client, provide what we call one click tool with a setup tool that it will take care of all your dependency, it will install all the dependencies that Tempest needs, and also install Tempest. And so with that, you will just know that you can, even that you want to test Tempest directly, you can from this point, go into a directory that RepStack already includes Tempest, which is the dot Tempest directory, and operate as running Tempest by itself. Or you can use test R or whatever. The other option is you can also use RepStack client directly. The difference between running Tempest directly and running RepStack clients is running Tempest, then your result will be in subuniforms. And you need to run through certain kind of data processor to format it in a way that you can understand. When you use RepStack client, it will process to a very simple JSON at this moment that only includes all the past tests. That's all it includes. And a few identity. For example, what Keystone ID? We use that as your cloud ID to identify who are this cloud that's sending this data to. For this tool, sending the passing is important because we don't differentiate you are not passing because you fail the test or the test is being skipped. So there's no differentiation. The goal here is know what are the common characteristics of the ecosystem. OK, Chris, cool. So one of the things that we're going to talk about first is the ways that you need to actually configure your cloud to prepare it for testing. There are certain assumptions that you need to have in place in order to have a minimally viable cloud. These include user accounts. And so right now, for testing, you need, at a minimum, two test users from two different tenants. This allows for the testing of a number of different basic operations, like creating virtual machines and making sure that they can attach to networks. But also making sure that a user from one tenant can't actually see the resources that are being created by another tenant. We also need two images that are installed to your cloud. Now, it can be the same image uploaded twice. But you can't use the same image reference twice. A number of tests will actually be skipped if you try to use the same image. And we suggest that you use seros. It's about 13 megabytes in size right now. It's a Linux distribution. It's very tiny. It's fast to boot up. And it gives you all of the tools that you need to be able to test booting, networking, replication, all these things, attaching volumes. And so it's a really nice image to use. And it's open source and publicly available. And that's actually what the gate is using right now, too. You're going to need networks because you can't do anything with a virtual machine if you have no way to get to it. So we have a couple of options here. One is to create a shared network that is visible to all of the tenants. And then all of the machines will be able to connect to that. Another option is to create one network per tenant. And so in this way, you're enforcing an isolation between the networks also. And this is nice if you are, if this is the way that you've configured your cloud to bring up new tenants, is that they automatically get their own network so they can't see one another. You also need to know what your member role is. By default, what is the role that all of the users have that enable them to access the cloud in a non-privileged way? This role needs to be assigned to all of the test users. Optionally, you might need a third user that has an administrator role if you want to take advantage of the more advanced features of Tempest. Right now, there are well over 1,000 tests that are running inside of Tempest. We're interested in a much smaller subset of that, the API Tempest tests. And some of those tests require administrator access. Some don't. If you're just running the Defcore tests, then you actually don't need to have administrator access. That's part of the requirement of those tests. But to encourage everybody to really get a full coverage of their cloud and get a sense of what they're doing, we actually encourage you to try to run as many of the tests as possible. So in this case, it's also nice to have an administrator role. And finally, you need to know the role that your Swift operator has. Oftentimes, this is the member role. Sometimes, it can be configured to be something else. But this is so that, again, you need a user that has the ability to create and destroy Swift objects and containers. So that was kind of the basics that you need behind your cloud to be able to start testing against it. But then once you have your cloud ready, then you need to configure Tempest. So there are four basic things that you need to worry about when you're configuring Tempest. The first is what services are enabled. And so you may be interested in testing a whole wealth of services, from Keystone to Nova and Cinder, all the way out to Trove and Sahara. Or you may be running a more basic cloud. You may just be running Swift, and you're interested in just testing object storage. And so in that case, you would turn off networking and compute. And these are just simple flags inside of your Tempest configuration file. And we'll get to those a little bit later. You also have the endpoints. And so you actually need to know exactly where your cloud is running, and which of those endpoints are publicly available for your test suite to interact with. And so if you have privilege to access to your cloud, that may be a different set of endpoints that you have enabled, versus if you're a user looking at evaluating someone else's cloud, and you want to run the non- administrator tests against that. You need to know the endpoints. And once you have your endpoint, you need to know the credentials for logging in. And so what are the names, the tenants, and the passwords of the users? And finally, you need the assets that we described earlier. You need to know what those are, and often times how they're configured. And so these are network IDs, network names, image IDs, and image names. So these are the basic things that you're going to have to set up in Tempest.conf, and the configuration parameters you're going to have to look for. Tempest.conf can be a little daunting, because there are hundreds of options. But when you narrow it down to these base things that you're looking for, it helps you realize that, oh, actually, a lot of the default settings are OK. And there was a new document that was published just a few, I think a month or two ago, by Matthew Trenish in the OpenStack QA team called the Tempest Configuration Guide. This is publicly available at docs.openstack.org as part of the developer documentation. At a high level, it actually covers kind of the three ways that you can set up the users and assets. You can use storedcredentialsintempest.conf. This is the original way. And this is part of the proliferation of options that are available. You can also use a new mechanism called locking accounts, where you create an account file that kind of contains all this information and the resources that are available to users and their credentials. This is actually a really nice way to run tests, especially if you don't have administrator access. But you do know the account settings. And finally, you can also use storedcredentialsintempest.conf with TennisIsolation. TennisIsolation assumes that you have administrator privileges. One of the benefits of this is you can run your tests in parallel. It's going to create a user and resources for every test that it wants to run. And it's going to run it up to some number of threads. The downside to this is you need administrator access to be able to do this. So for our talk today, there's going to be an assumption that you don't necessarily have administrator access. And so we're not going to cover the third option. We're just going to talk about the storedcredentials intempest.conf and the locking accounts. But the takeaway message is, if you are going to be running Tempest, this is essential reading. And so we strongly recommend that you go look at this. It's very well written, and it gives you a lot of information and insight into how the tests run. So now you have the overview of what you need to configure. I will step you into a few very specific cases. One of the first one is, when you install Tempest, you have a Tempest sample config file. That file is very useful. It has the default setting that's set in there, command it out. But if you don't set it, that is what's being set in there during the test. So I strongly recommend that if you ever need to edit this Tempest config file, you add whatever your configuration parameter, in addition to the default there. Don't just uncomment the comment, and then configure your parameter. Because leaving the default configuration in there will help you tremendously to understand some of the pain that we have had down the road. So the takeaway is, if one of the configuration is so important for you, set it. Even that when you set it, the value is the same like the default value, set it anyway. One of the things that was changing from Ice House to Juneau is the allow tenant isolation flag. It was default to false before, but later was default to true. And that threw us off. Why does the system the same configuration that worked differently? So that is one of the best lessons that we have learned. If the configuration parameter is important to you, set it. Enabling service, Chris talked about that. Some people use neutral networks. Some people use heat. So configure it to the way that your cloud will operate. Again, we can't emphasize enough that if you are able to please test the whole test suite, that will help. The data is so important for us to define the common component for the whole ecosystem. And that's nothing more than data-driven capability definition. Rather than someone define, here's the capability. If we have the data, we will be able to define with confidence, reliable that this will work for the whole communities. User account, Chris also talked about that. Tempers have about 1,600 API tests in the last released. So if you would really want to exercise all the tests, the admin credential is needed. But when Dev Core trying to define the capability, we try to avoid from admin. We try to see the value of interoperability. And not every user will have admin. But for data collection, please do that. And if you do that, then you will need to use the unlocking account-setting mechanism. But that is not needed for what we call certification. But it's highly encouraged. Image, we talk about, you need two images. They can be the same image. But we need two image IDs. OK, network. One of the ones that you can configure very easily is the fixed network. If you use this parameter, then one of the things, the downside is you lose that network isolation. Because all your test users need to be able to have visibility to this network. Again, that is a choice. And it depends on what your network configuration is. Log file, very important. The default is empty. So if you run the test and you was wondering where the log file is, you should set the same as log path. Resizing. Resizing is default to false. And it so happened that some of the tests in the current capability specification involve some of the resizing test cases. So if you do not test this one, you do not set this parameter, all these tests will be skipped. And then from the passing point of view, we do not know whether this is skipped or failed. So if it is important, set it. End point tight. Tempers was designed to test in DevStack environment. And the next level up is a distribution. You can build a cloud using that distribution. You can build it with whatever way that you want to build. But when you go to a public cloud, there is certain way that the public cloud is built. You cannot work around that. One of the things is the end point. Some of the keystone you might not be able to access externally. Sometimes you might need to set it differently. So pay attention to this parameter. Adopt it to your cloud. And something to note here is that when we say a public URL versus internal URL and the options that are found here, these typically refer to the types of URLs that you would find in the keystone catalog. And so if you run the keystone command line and type keystone catalog, you'll see these names, internal, public URL coming up. And you can get a sense of what URLs are going to be available to your testing and which are going to be available. Thank you. So now, volume. People build your VM differently. Not everyone just have one disk. Some have OS disks and also have data disks. System disks, data disks. So if you happen to have the defaults, when you add a disk, the default volume is VDB. And if you already have one in there, your test case will fail. So know your environment, what kind of configuration, what kind of VM you are building, then adopt it to that. These are the lessons that we have learned. And the role is with. People might label a different role. Their policy will be differently. So just pay attention to these parameters. So as an alternative to setting your account values directly, you can also use locking accounts. One of the benefits of locking accounts is where if you set your credentials in the tempest.com directly, then you're forced to run tempest serially, unless you're using tenant isolation. Locking accounts gives you a middle ground. It allows you, if you have some number of accounts created, you don't necessarily have administrator access so that you can generate these accounts programmatically with tempest, then you can use locking accounts. The idea is within tempest you create an accounts.yaml file and you set the test account file parameter to match that file. So by setting that parameter, you are telling tempest, hey, look at this accounts file to find the accounts that we need. In conjunction, you also have to set allow tenant isolation equal to false because if it's set to true, tempest will roll into tenant isolation and start trying to use your administrator credentials for that. You can configure accounts.yaml with a set of pre-created users and the resources that are available to them. And so this is a nice way too that if you have images or networks or anything else that a user might need, but they're going to be isolated within those tenants, you can directly specify what those are. And so if you remember previously, we talked about having one private shared network. Well, if you don't have a single private shared network, you can create individual private networks for every one of your tenants and use that parameter to access them. So and now that you've done this, for every two accounts you create, you can run one tempest thread. And so if you have eight accounts, you can run four threads in tempest and increase the performance of your test run. Right now, if you do a full API tempest run, it takes about an hour and a half, two hours? Two hours? Two hours. If you do a full def core run, it's going to take about 10 minutes. And so if you're doing a smaller def core run, that may not be important. But if you're actually testing in production and iterating and trying to do this quickly, being able to run your tempest test in parallel is actually a very valuable tool. So it's something to keep in mind. Every user needs their own tenant or a tempest will behave in strange ways. There may be a race condition involved where it wants to create virtual machines and then list the virtual machines and help them to make the story properly and they didn't. And if your users are sharing the same tenant, those tests could interfere in ways that are undefined. And again, talking about, you can either use the one shared network if it's available, or you can define your own network names. And one thing that's important too is if you have non-standard Swift roles, you actually need to use the Locking Account so that you can start defining. The tempest configuration guide has a little bit more information about this. But if you're running container tests, you actually are more likely to need to use the Locking Accounts configuration. So that's something to keep in mind if you're having trouble with your container tests, that could be the reason why. I mean, you're not gonna container the Swift object storage. And so here's an example of what the Locking Accounts file looks like. You give it a username, a tenant name, a password, the resources that are available to it, and the roles that might be assigned to the user that tempest should be looking for. So it's a pretty simple file format. See another file format to look at. So now you have tempest configured. And we should keep in mind that this is actually going to be an iterative process for you. The first time you're on tempest, you're going to run it, and you're gonna say, why are things breaking, what's going on? So you're probably gonna have to go back and twiddle some of the configuration parameters to match your cloud, you know, change things you might have overlooked. This is natural, everyone goes through it. But there are a variety of ways to initiate a tempest run. One thing to keep in mind here too is that there's actually an assumption that after you've installed all the tempest tools, and I didn't capture this in the slides, but I'll make sure that the slides we upload for the presentation capture this. You wanna run tempest in a virtual environment, and so there's a .venv inside of the tempest directory. If you're going to use tester run, you're gonna have to activate that virtual environment. And if you're familiar with Python virtual environments, it's you source an activate file that's within the .vn bin directory. So there's something to keep in mind. Tester is the Python testing framework that is used to run the tests. And so that will kind of look at the standard Python unit testing format. There are other test tools that you can use to do this. OpenStack and the OpenStack QA team prefer tester. So you can just do test or run, and it's gonna run all of the tests. There's also a run tempest script that is provided by the tempest team, which lets you run it with a custom configuration file. You can also load a custom list of tests using either run tempest or tester using the minus load list option. This is nice if you wanna say run just the depth core tests. You provide a very exacting list of tests, and it's just gonna run every one of those. If the test is in the list and they can't find it in your test repository, it's gonna skip that test silently. So it's pretty important to build that list off of whatever the current version of tempest is. And you can get that by running tester list tests. And this is part of the tester documentation too, which is worth looking at. You can also restrict your test run to a search string. And so for example, if you wanna run all of the tempest API tests, the easiest way to do it is to just pass dash dash tempest API to run tempest. And that's going to only run tests whose prefix matches tempest.api, and it's going to ignore everything else. So the load list and the pattern matching format are two very powerful tools that keep you from running hours and hours of tests within tempest. And so it's important to keep those in mind as you're running tests. Finally, after you've run the tests, you're going to want to interpret the test results. You'll probably see a lot of output flying by if you use run tempest. It's actually gonna give you a nice view of your tests as they come out and tell you what passed and failed and what the fastest and slowest test was. But actually all of the output from the tests, the kind of the raw output is stored in a .test repository directory. Every time you run tempest, it's gonna create a new file. The files are just numeric increasing. And the format is called subunit, and we have lots of tools for parsing it. And they're all going to be available in your virtual environment that you're running tempest in. Subunit isn't the easiest format to read. It is text, but there's a lot of data, and it's a little bit messy. But you can create easier to read formats like you can use subunit one to two to convert the format to subunit two, and then subunit two to CSV to produce a Kava separated list of tests. So you can kind of look at things. You also want to make sure you look at the logs. If things go wrong, they're going to be logged. You want to check the logs because it's gonna have your Python stack traces and all sorts of information about the tests. But another nice way to run tempest is with the RESTAC client, and Catherine's gonna talk a little bit about that. Okay. So again, we are running short on time. I try to go so fast so that I can show the demo. The important thing is the demo. So the slide, so you can run tempest. You can run tempest with RESTAC. And with, you can run specific tests in a file, like we say here, or run, you just say run RESTAC, it will run all the API tests. Once you run the test, as we say before, the result is in subunit format. You either need to pass it the way that we have shown you a few two earlier, or you can, if you run by tempest by default, we pass it for you to a JSON file that is ready to upload to the RESTAC API. Whoop. So once you upload, then now we can go to the UI and look at the data. So here is the RESTAC.net. You should be able to, from your computer and look at that. And that is the website. So once you go into there, you can click on one of the test results. For example, look at this page. On one of the lines there, you can see that this is the duration of your test. This is what being test about 10 minutes or so, and the total pass of this test run is 88. So this test run probably just run the DevCode, the file test. And then one of the things that you can look into here is the specification. Right now, DevCode have three specifications. 2015.5, .4, .3. So it depending on what release and what specification you want to test around. So you click on that. And then the specification describe or include what are the capabilities that you need to pass or need to pass that they think is the common capability in there. That is where these things, these areas is the capability that currently define in this specification. Not only that, OpenStack give you several ways for, because you might be a vendor that only do object store. It doesn't make sense for you to install all the other projects. So therefore the licensing right now, licensing or program right now, there's a three level of program. There is a OpenStack Power Platform. That's the one that have everything. Whatever capability that. Compute storage. Yeah. And then the other one is currently only compute storage. If you have both compute storage, then it will be platform. But if you only have storage, then it will be power storage, et cetera. So you can then looking at some of the tests. You see some of the tests pass, some of the tests fail. And depending on the specification, if we go to O4, then you see that little flag there. That is the feedback from the community that a lot of people are not passing this test. So there's some flag test there that the deaf core team, people, et cetera, will have to take a closer look on these tests. I want to show you a test run if you go back to this one and go to the last page. Yeah. The second one up. This is a test run that you can see that the number of pass test is 1200. This is what we would encourage you to do. Test as much as you can. Send us the data. Although that capability is the same because we're currently only 100 or so test case. But with this number of test data in there, we will be able to define the next type of capability coming in. We will be able to know what percentage of the community have this one pass, for example. So you can go to revstack.net, play with it and tell us what you think. Your feedback, your involvement in the revstack project is surely welcome by all of us. We need more help. Right, and so why revstack? It's a simple result collection for, you know, it's a tool for collecting and reviewing results. It's an anonymous way to show your capabilities of your cloud and so you can actually report back and nobody is going to be able to look at it and say, oh, product X, you are not working because of this reason. Helps us to discover the most widely used features and define what interoperability means. Like community feedback on interoperability is so important. So run all of the tempest API tests, not just Def Core and report them to revstack. Your results make a difference and the revstack tool wouldn't be here if it weren't for the efforts of Catherine but also David Lenwell was the PTL of the project from the beginning and has put a lot of effort into making revstack a really successful tool that we're going to be able to use for interoperability testing. So we'd like to say thank you to him for that. You know, as well as all the other developers. And thank you. Do you have any questions? Yes, and please use the microphone in the middle of the room so that we can capture your questions for the video. Hey, do you have any support for multiple availability zones for Nova and Cinder? At the moment, the concentrate is on API interoperability. I mean, definitely that will be an area that we have to evolve to, but at this moment, not yet. And to that end, typically what we're looking for is if you have a cloud that, a testing cloud that captures the essential capabilities of your deployment cloud but isn't in some way simplified, like it may only have one availability zone. As far as the DEF course standard goes, we would still accept those test results. Yeah. I think this is still a very early stage and community involvement is very important. Hey guys, for having my question, it's similar to the gentleman before. I'm doing one of my questions in Cinder. I was planning to add some tests into Tempest. For example, I would like to migrate volume from LVM to IBM backends that possible to do the testing Tempest. So the OpenStack QA team is, it's like any other project with an OpenStack. It's open, it's a community of developers and anybody is welcome to join and contribute. And so, I mean, I think the answer to that question is best answered by the OpenStack QA community. The PTL of that project is Matthew Trenish. They meet in the OpenStack QA IRC room and have weekly meetings. So I'm sure you should be able to write a test. How you would be able to do that and get them integrated is better answered by them. Okay, thanks. Okay, thank you very much. Thank you.