 OK. Hello, everyone. Welcome to the RefStack session. I know the keynote is still going on, and people trying to come up, hopefully, and it's kind of very confusing how to get to level four from the keynote session. So today, we'll talk about cloud interoperability and how you can use OpenStack to test your cloud. I mean, RefStack to test our cloud. My name is Catherine Diff. I work for IBM, and I am the RefStack Ptl. And I'm Chris Hodge, and I'm the interop engineer for the OpenStack Foundation. I am Paul Van Neck. I am a Python developer for IBM. I'm also a core developer in RefStack. Hello, I'm Sergei Slipushenko. I am software developer from Marantis. I also co-developer of RefStack. Great. So let's go through the agenda. What we will be talking today. First, Chris will talk about Devcore and interoperability. I will come back and introduce RefStack. The rest of the presentation is about RefStack. Sergei will talk about how do you run tests with RefStack client? And then we'll go, Paul will go through the website. I will come back to some troubleshooting technique and some lesson learned that we have learned through the last years so that it can enable your testing much easier. Chris? OK. So we're going to start off by talking about OpenStack interoperability in Devcore. When you look at the OpenStack mission, and the very first sentence of it, it says that we want to build a ubiquitous open source cloud computing platform. And one of the ways that you gain a ubiquity, if you look at the world around you there, we're surrounded by ubiquitous things, our telephones, our cars we drive. And they share something in common, and that's interoperable interfaces. So one of the ways that you get ubiquity is through these interoperable interfaces. And there's actually a kind of rigorous definition of what interoperable means. It's something that's interoperable is understood. You as someone who is consuming that, you know how it works. You can, no one person or company owns it. So it's widely available for everyone to take advantage of. It's durable, so it doesn't change a lot from underneath you. When you pick up your phone and dial a number, it works the exact same way today as it did 50, 60 years ago. And indeed, you could call old phones from your new phone right now, because the interfaces are so durable. And they're also unrestricted. There is nobody saying that you can't use this interface because I own it and you have to pay horrible licensing fees or have special access to be able to use it. And so OpenStack through DevCorp is trying to define a set of interfaces and policies for managing them that meet these three requirements. So DevCorp is the OpenStack Interoperability Project. And it's actually been running. And we've been testing clouds for a year now, both public clouds, private clouds, and distributions since 2015. The program has been in development for several years. It was launched last year at the Vancouver Summit. And right now, we have more than 30 companies who have obtained the OpenStack-powered trademark through the DevCorp program by testing their cloud against a standard test suite. And so the future of interoperability is now. One of the things that DevCorp does is it publishes guidelines that define what it means to be an interoperable OpenStack cloud. And so if you have an OpenStack product, you must adhere to this guideline if you want to sell it with the OpenStack-powered logo. What the guidelines do is they define components that consist of API capabilities that a product must expose. And these capabilities include a set of must-pass tests. And so this capability might be create a virtual machine or list images on your system or get a Keystone token. These are all capabilities and there's a test that can check to make sure that the capability is there and it's working the way that you expect. It also includes designated sections. And so part of what OpenStack is saying too is that to be called OpenStack, you actually have to be running OpenStack code that was developed within our community. Now this isn't all of the code. OpenStack allows for a lot of different pluggable architectures and things that you can run on the back end. But generally the designated sections are stating that you need to be running the OpenStack API code so that the API calls are consistent. So, but this isn't a DevCorp talk so much as it is a RefStack talk. And RefStack is how you can participate in this process and help define the interoperability standard and also test your product. So, you know, at the highest level, if you want the OpenStack powered logo for your distribution public cloud or private cloud, what you do is you run the RefStack client which itself runs Tempest and uploads those results to a server. You know, in this way, the OpenStack foundation has an easy way to check to see if your product is in compliance with the capabilities that are required for whatever logo you're looking for. But the other thing that is helpful for us too is for you to run RefStack against your installation and just share the data with us. So, RefStack just isn't about running the required tests for gaining the logo. It's also about gathering data so that the OpenStack foundation and more specifically within the community, the DevCorp working group can analyze those results and really see what's being widely deployed, what's working, what isn't working, and we can focus on our efforts on either fixing the testing, modifying the guidelines to match the reality of deployed clouds, and also deciding what are the future capabilities that we want to bring into the guidelines and what are the future components that we would want to create. And so, by running the RefStack client against your cloud and running all of the API tests against that and sharing it, you can help shape the future of interoperability inside with an OpenStack. And so, with that, I'm gonna turn it over to Katherine to introduce RefStack and what it is. Thank you, Chris. Suki. So, Ben, I just want to summarize. DevCorp is a committee that defies the specification and what we call guidelines. What is OpenStack? I mean, that's a community-driven process. And we have a session at 3.30 today. So, please go there to learn how can you participate in there to shape what is the capability that make up an OpenStack, a core layer of the OpenStack, and what must pass test. So, DevCorp defies the specification. There has to be a tool that runs these tests and not only that, the result has to be at a centralized repository that can share to the whole community and that is RefStack. With RefStack, there's two major parts. RefStack Client is where you, the command line tool that we developed so that to make your testing process easier, as easy as it can, we'll go through some of the highlights of that. And more important is the RefStack server, serve at RefStack.OpenStack.org. So that RefStack server is a lens or an eye for the RefStack user to look into the result that have been collected from the community. So, like Chris said, the successful is on the participation of the community, the more data that we have, the more we can understand what is the common, you say what is the common core that being deployed outside so that we know what should be making what the must pass test and capability. With that, I would like to introduce Sergi who is going to talk a little bit about RefStack Clients. Is it working? Yeah, so let's start with a client part and the RefStack Client, it's a pretty much simple CLI tool which actually have two purposes. The first main purpose of RefStack Client it's to provide to user the easy way to run Tempest, install and run Tempest. So in the RefStack Client repository, user can find the kind of script which just when one click install the Tempest with dependencies with so on and then allows user to run the set of tests using this installed Tempest. By default, RefStack Client uses Tempest from some, from some commit in Tempest repository which approved by Devcore, but if you want, you can change it manually and see how your cloud works against some, maybe other version of Tempest. And this is the one purpose of RefStack Client. The other purpose of RefStack Client it's to allow user operate to operate with RefStack server API using CLI tool. So the first thing which you can do with RefStack Client it's actually a plot data which you just get run RefStack Client set of tests against your cloud. And this data can be uploaded to, to get this data, you just should specify Tempest configuration and as a result, you get some subunit test file and you just with one click upload this result into RefStack server and then see how this results meets expectation from Devcore or any other guideline which you can find on RefStack side. Also, you can specify not the whole test list but the part of it to see how maybe the part of your cloud meets expectation from some guidelines. And you can upload your data in two ways. The easiest way it's upload your data anonymously then this data moves into public domain and you can see, and everybody can see your test results on RefStack OpenStack.org. It's easy way how to find out how RefStack work itself. But if you want to manipulate your test results maybe run some test, some set of test results and you want to figure out how it changes from run to run and you want maybe to review your results, delete it and so on. So you should specify your identity to upload your results. And to specify your identity you should use the public key. So you kind of get it review OpenStack.org or GitHub or a lot of tools which also has select lines. You should just specify a public key in your account and then specify this key to upload your data. And then RefStack client use this key to points who was the owner of this test results. And you also applaud not only, you should also specify not only the public key but also there's some kind of digital signature because RefStack should also check that you are the owner not only for the public key but also the private part of this key pair itself. So you should put these two fields and then RefStack check that this public key belongs to you and then you can use this key pair to upload test results in the RefStack website. And then if you go to the RefStack, login into your account with your OpenStack ID you can see the set of your test results and manipulate with it. I think that mainly that's it and I pass the ball to the poll. Yep. Thanks, Serge. So hi, I'm Paul Van Ack. I'll briefly introduce and guide you through the RefStack.OpenStack.org website. If you have a computer you can visit it right now but so the site is basically just where you can view and manage test results. We've been over this previously but it's mainly a way for you to actually see how you stack up against the Devcore guidelines. So I'm gonna navigate to the website right now and just, there it is. Okay, so first we click on the Devcore guidelines tab. First you might see selectable versions. So these are Devcore defined guideline versions. You might notice that each version has a status. You typically wanna stick with the approved statuses as for when you're testing or trying to compare your results. You typically pick the latest approved status guideline or latest approved guideline that corresponds to your OpenStack release. So if you were Mataka, I believe 2016.01 is the only version that corresponds to that. So you'd use 2016.01 for your Mataka OpenStack cloud if you're testing that. But if you were like Ice House, I don't know if any Ice House is approved right now. Oh yeah, we still have 2015.05. And of course we also have the OpenStack or Target program where you can pick the foundation to declare these three OpenStack powered target programs. Platform, encompasses both compute and object storage. So platform will just go through. You might see all the required capabilities like compute, you can expand and see the test. A lot of these are flagged. Hover over, you might see, well no, I guess not. They're flagged, but it's okay. So more importantly, you might see this test list link right here. You click on that. You can use this test list with Rustac Client to test only test cases that you're interested in. For example, if you're trying to certify with 2015.05 and OpenStack powered platform, you'd click on that. This is the test list you would use with Rustac Client. So you're not, I mean, it can be cumbersome to test the entire API test suite in Tempest. So use this test list. You can also use the command line W if you just wanna perhaps make a job that always pulls the latest test list. Also, we have aliases here. Aliases is always included by default. You should probably stick to that setting. Tempest test might be renamed over time throughout the different guidelines. So to be safe should include the aliases. If the, I mean, tests that don't exist will be ignored anyways by Rustac Client. So it's safer that way. Also, you can include flag, but it's okay. That's not necessary for certification. So here you're gonna see community results. These are anonymous, anonymously uploaded results. So we'll just click on the latest one, uploaded the 25th of April. You might see the total number of past tests with all the tests that they passed. So I'm gonna sign in right here. We are synced with OpenStack ID. So this is the same sign in that you'd use for the OpenStack.org website. So why POR is logging in? One of the things is we provide the test list for you so that it is a quicker way to know where your test status is. However, as Chris just say, data is very important for this process. We encourage that when you are ready to send the data to RepStack for certification purpose, for example, it would be better that you run the entire API test so that not only we have the required tests to look at the certification status, we also have many, many other data porn so that we can have some data looking forward to the next guideline, define the next guidelines. Yep, thanks, Catherine. So here after you log in, I can click onto my results. These are test results that I uploaded awhile back using RepStack Client using my private key. So these are all privately owned data. Click on one. You might see I have the option to share and delete, but also if you expand right here, you can share from here and also associate a guideline and target program. So these are what I want to pin my results to. So let's say my, I know that it should belong to the 2016.01 guideline and I'm targeting the OpenStack-powered platform. But hold on, let me just switch it down to something else. So 2015.04 for example. So now when you click on the results report page, you'll see that it will default to that version you selected. So other people who visit this page might show, might don't have to guess which version your test results belong to. So right here, this is only three tests were uploaded so obviously not compliant. You'll see a big red no or a green yes for if you are compliant. Not sure if there's any compliant ones, but you'll see a green yes if you are compliant. That means you pass all the required non-flag tests that DEF course specifies. But again, just after you upload, it's a good way to see where you are in terms of interoperability. And with that, I will pass it over back to Catherine. So I'm going to go to very quick if anything we can talk after that. So one of the things about running tests, one of the most challenging part of running the tempest test outside of DevStack stack is create that tempest config file. In DevStack, you have the default tempest config file. Everything is default, the network name is private or public, et cetera. When you run your production or you run your cloud for certification, your configuration might be different. And tempest config field or parameter is very much associated with the tempest version that you are using. So be very careful if a DevStack allows you to run any tempest version where you can run it with master, but be very careful. Make sure that your tempest config field matched with the version that you run. With the latest version that we install as default in DevStack, we know of this parameter is the parameter that you should configure. About tempest config, there's about 300 parameters in there. A lot of them have default parameter. The one that we call out here are the one that you need to configure so that it is not the default parameter so that you can run your test. I go down the list very quick in the auth session. We have this use dynamic credential. The default is true. So you need to, for us, DevStack, we run cloud with no admin credential. So if you set it as true, there's no way for it to create the resource, the network, et cetera. Your test will fail. So it's very important to set it to false. And also you should use a test account file to set your credential. In the compute section, these are the parameter that you need to set. The most important thing here is a tool image. These tool image, you need to have tool image because it will test one image configuration and use the other to verify, et cetera. Another important session, I put it in the troubleshoot, but I might as well talk about it here, that we do not know during the test which image will be in test. One of the tests will do SSH to the VM and you configure the user ID that used for SSH. If you use the tool image to test and you have different user ID for tool image to SSH in, and we only have one parameter in config, tempest config file for use to configure that, chances are you might have some failed VM that you cannot SSH to. So be very careful of that. The resizing, there's some parameter, some test case that test the resize capability. The default of this is false, so you need to set this to true. Otherwise, you see a lot of skip. From rev stack point of view, skip is equivalent to not pass. So when a test not pass, we don't know whether that test is skipped or that test is failed. For the identity session, the important thing here is if you use HTTPS, which is not by default in DevStack, you might need to run rev stack with the minus k or insecure option so that you don't run into certification validation issue. For network, you need to set public network and these are the ID you need to share. Another important thing is object story operator row. By default, a lot of the member here is a capital M member. Some of the configuration use underscore member underscore. So it all depending on your cloud, you need to set it to whatever the row that defy by your cloud, the policy defy by your cloud. Service available. So right now, in the upcoming release, neutron will be required and by default neutron is false. And by default, neutron is false. So to test the neutron test, you need to set it to true. By default, Swift is set it to true. But if you only care about compute components, then Swift, you don't have to run all these object tests that is going to fail anyway. So you might as well set it to false. And very important, validation test. There's a lot of validation test includes in DevCord, must pass test, by default, run validation is false. So you need to set it to true, otherwise your test will be skipped. Inmate SS user, this is the parameter I talked about earlier that this is the user that you use to SSX into your image. And troubleshooting, I don't know what we have enough time, but some of that I touched about that already. Chris, anything to add? Yeah, so using the configuration settings that Catherine just described, I actually took those configuration settings yesterday and configured RefStack to run against a public cloud. And so when you look at Tempest.com and you see hundreds of options, you can actually almost safely ignore most of them, start with the base template we've provided and then kind of build out from there to bring it to your own cloud. And so one of the hopes of this talk is that you can walk away and have a sense of what you need to do to be able to run RefStack and how that it is a tractable problem for you to solve. Some issues you may run into while you're running the RefStack client is, Tempest may tell you that it doesn't have enough accounts available. This is because when you use the accounts.yaml, Tempest will create a locks directory where it prevents Tempest, if it's running in parallel mode from reusing credentials during tests. So there's a good possibility that you have a stale locks directory. If you delete that, the path is in the Tempest.com file. It should release all of those resources and you should be able to run your tests again. Sometimes you can have failures to SSH to the VMs. You know, make sure that you have, you know, the two images as Katherine mentioned before use the same username. It's going to do key injection so your cloud images have to be able to accept key injection to be able to do that. That's the default way that Tempest logs into the machines. You know, if you are, if you're running into certificate validation issues, use the insecure model for running, you know, that way it'll turn off all of the Python SSL library checks against SSL. You know, and you also, if you're running against a newer version of Tempest, the Tempest is under constant development and it's being refactored, it's being improved. For example, there were two sections of the Tempest.com file that both configured how you logged in with SSH. Those have been reconciled and so some of those settings have been taken away and they've been moved into only one set of settings. You know, so it's worth checking to see, you know, just where things might be changing. Very quickly, right now as part show, you see data being sent up for the public page. You don't see vendor name, et cetera. In the future, we're going to try to enable that so that you can see some association. Data can be identified easier and we are also thinking about central light testing. That means that the test will be initiated by the ref stack, but that is a very ambitious goal that we have. With that, I don't know whether we have time for questions and I hope we'll be here for any questions you have. Right, so the latest guidelines are, as part of the compute program is going to include block storage and networking and so Cinder and Neutron are going to be required. They aren't separate programs and so there's no, right now there are no plans to have say an open stack powered network or an open stack powered block storage. But there is, if there is a demand for that, there's a possibility of opening up DevCord to other programs and so if all of a sudden, Neutron as a networking layer independent of all the other projects just takes off and everybody wants to run it and they want to sell this as a product, then there's a possibility for introducing say an open stack powered networking program, but that's not really on the roadmap. It's more likely that the future is going to have something like open stack powered database or open stack powered big data, encapsulating things like the Trove project and the Sahara project. It's where the community you can come and with all the participation and feedback to for the DevCord committee to identify what are the components that you think should be included and it's not yet included. So, so, listening there is all our sessions. We're going to have to put the slide and share somewhere, but I hope that the session is helpful and if there's any questions, we can entertain a couple more or? Yeah, got it. I got your question. Right now, it's not. Right now, if you'll use self-powered tests, set of tests and the ref stack use tempests. It's a little bit different or even I think it's a significant different set of tests. So right now, self-checking fuel, it's kind of a little bit differencing. So you can just use a ref stack to make sure that your installed fuel cloud is interoperable by a ref stack client. But that brings up an interesting point too. If you already have an existing CI pipeline for your cloud, ref stack includes a capability to take existing tempest results and process those and upload them. So you don't have to use the ref stack client as your tempest front end if you already have an existing CI pipeline. So that's a way for you to easily plug ref stack client into your existing infrastructure. And one last note about ref stack client too is that we advise you that you configure so that you aren't using any administrator credentials. That way you can test against your production cloud against established tenants, you know that it's not going to interfere with your existing processes. Thank you very much for coming to our session. And this afternoon, death course session, and we also have a wiki page. If any questions, just IRC us or we'll be around here for a little bit too. Thank you.