 Hello. Good afternoon, everyone. I hope lunch was good. It's very good time to digest. Meanwhile, we'll be speaking about what's the stack. And we're going to be starting by introducing who we are. So first of all, we both, you don't know our name yet, work for Inovance, a small company that started in 2008 in France that is spreading like a virus, some would say, or like a successful company, like I would say. We've been opening an office in Montreal, Bangalore, and very soon in San Francisco. We have 120 employees. We have lots of customers. We are making a little bit of money that's helpful to pay the employees. And we have been constantly in the top 10 contributed list for OpenStack for the past four releases. So today with me, there is Christian Schwed, Christian. Yeah, thanks, Nick. Yeah, my name is Christian Schwede. I'm a developer working at Inovance. I've been working on OpenStack since the end of 2012. And last year, October 2013, I joined Inovance. I just recently became Swift Core Developer. So I'm mostly focusing on Swift. But I'm also doing a lot of testing and automation stuff in and outside of OpenStack. And if you want to contact me, write me an email, or you can also find me on IRC and sometimes on Twitter. And I'm Nick Barsett. I'm VP of products and pre-sales at Inovance. I've been working on OpenStack since OpenStack had a name. I founded the OpenStack telemetry project a while back with a few other people. I've been traveling around the world helping people deploy OpenStack in various fashions. And if you want to complain about my poor French accent or anything else during this presentation, I invite you to use my Twitter account at Nijaba. So a little while ago, I had a conversation with my customer that told me, hey, what you guys are doing is great, but how do I know we are done? How can I tell whether the deployment you've delivered to me is actually doing what it is supposed to be doing? How are we going to be able to verify that we haven't forgotten something? There is so many pieces in OpenStack. This is a very complex thing. I need a progress bar. That was literally his word. I need some way to measure how far we are into the deployment. And of course, I immediately jumped to the whiteboard and wrote his progress bar. Look, we are at 80% of the project, but he didn't trust me. So we had to figure out something else. That's when I started talking with Christian about, hey, isn't there something we could do with Tempest to verify on a regular basis that everything is going well? But Tempest at this trouble, you need to be an experienced admin in order to run it. And you even need, in order to run some of the tests, quite a bunch, actually, admin rights on the cloud. And Tempest leaves resources a little bit everywhere after running. So we couldn't really use Tempest directly. So the idea of making a web app came to mind. And Christian, by the time we had the idea of a web app, Christian had already developed a proof of concept. And when I saw the web app, I say, hey, this is way too cool. We shouldn't keep that for ourselves. Why don't we actually make a website so that other people can use it? I'm sure other people will find it useful to be able to measure from an external resource how the cloud they are going to be using behaves. Is it compliant with Tempest? Does it expose the services correctly? Provide them with a clear outlook of what they were going to be dealing with. So why do you want to do that? Well, first of all, you want to make sure that, functionally, your cloud works correctly. That means looking into as many API calls, verifying that these API calls are working. And by doing so, you'll suddenly detect whether you've forgotten to configure Cinder, because you can have a very well-functioning NOVA with a very poorly working Cinder. Or you can have a very well-working glance without any NOVA to deploy it to. Well, that's a little bit hard to miss, but you see the picture. And also, you can know if your goal is to deploy all of OpenSack, that you are done once all of the Tempest report passes. Well, that's in an ideal world. Of course, the coverage of Tempest is not 100%. We all know that. But we've been slowly working at improving that, jointly at the community. Another thing that came to mind is, lots of people are very excited about the idea of doing hybrid deployment. I want to move my workload from Cloud A to Cloud B. How am I going to make sure this is possible? Because even though A and B uses OpenSack, well, we all know that all OpenSack may not be exactly the same. So taking the point of view of an external user of the cloud seemed to be the best approach to solve this user of the cloud question. One may or may not trust a vendor to report on this feature directly. A vendor may care more about what's happening underneath than what's happening in front. The user experience is what we will be focusing on, giving non-developer, non-admin the possibility to verify what's going on. And of course, if the person that is executing the test is independent, that also gives a little bit more credibility. Eventually, you may even want to be able to share the results of the test you've asked to be executed. And this is also a feature that we built in. So on this, because I was only the product owner on this great adventure, I let Christian walk you through the exact process, the architecture of what the stack. OK, thanks, Nick. So let's talk a little bit about using Tempest for testing. There are some challenges if you want to use Tempest for testing. Nick already said it. Maybe you want to test your deployment as a non-administrator only with her user permissions. And in that case, you might not be allowed to upload any images. So for example, your Sarah's image is missing. And this is a little bit problematic because normally you use a Sarah's image, which is a very small image to test, for example, Nova. So what should we do now? Well, we decided to select simply the smallest available image. And in many ways, it's most likely it's a Debian image because Debian is the second-next smallest image. There are also some problems with API changes between different open stack releases. So for example, there was a change in a transaction ID of some logging inside Swift a while ago. And the stranger action ID is no different than before. So all the Tempest tests for Swift were failing. But Swift itself run fine without any problems. It was just this log line entry that made some problems. Actually, there's a discussion within the Tempest community. And there's a session also right after this about branchless Tempest. So the problem was that Tempest had a branch which was tied to another branch, for example, to Nova. So with branchless Tempest, I think we are going to solve a lot of these issues. And there might be some customized back ends. Your back end might look like a Keystone back end, but it's not a real Keystone back end. Or you might have some, for example, SSL termination in front of Keystone, removing some heaters or adding some other heaters. And your test might fail because of this. So we had to think about that. I already said it, your image service might make problems. So maybe it's announced as public. And if you do a Keystone discovery, you see the image service, but you're not able to access it. Then the next step is you need to select some tests. So we were focusing first on user executable tests that gives you the chance as a non-administrator to run tests. And this is a special useful if you're a developer and want to try a different cloud and a different vendor and see if this vendor supports all the things that you need. We are also skipping some tests. So there are a lot of tests that are executed as JSON and as an XML test. But behind the scenes, they're doing the same. And we are skipping negative tests for now. So we are focusing on the core functionalities. And what does it mean? For example, for Keystone, we need some authentications. This is obvious, I think. For Novel, it maybe means that you want to create a booted server, assign floating IPs to it, or they're using different flavors. And for Swift, it might mean that you want to create a container, upload some objects, download some objects, and similar things. Or in glance, that you are able to select different images that are available to you. So for this, we first started with a command line interface. And we called it TempestReport. And TempestReport simply executes a Keystone discovery with your credentials, selects the smallest available Nova image, the smallest available Nova flavor, and the network ID, and creates a simple Tempest configuration file. And after that, it executes a subset of the API tests within Tempest itself. And because the results might be a little bit overwhelming to you, we try to summarize these, so that you don't only have all these log lines, but that you have a summary by a service, what was successful and what not. So TempestReport is part of Tempest? No, TempestReport is not part of Tempest yet. There were some discussions in the community with it. And I think it's going to be very interesting in the next few months, because there are different ways how to do this and what people want to do with Tempest. And, well, it's going to be interesting. It's available on our GitHub account. So it's open to the public. And if you can help out in the community, well, we're open to it. So by the way, I'm asking a question, but you can ask a question at any time. Feel free to get up and go behind this microphone. And if you ask a good question, I'll give you candy. So as I said, we have a part that is discovering your settings. And it's included in TempestReport. So if you run it, for example, on a DevStack installation, you simply set your Keystone credentials to the environment. There's a small file within DevStack called OpenRC that does exactly this. And you simply run Tempest Discover afterwards. So that creates a Tempest configuration file. And now you can set some other environment variables to tell Tempest, okay, we want to use that configuration file. And if you have done this, you can use your favorite tester. You can use NOS test or a tester or whatever else you want to use and simply call the Tempest test itself. And in that case, I just use the test for Swift. And as you can see, it succeeded. Okay, it's only a example, but you get it, I think. So TempestReport uses this discovery of the Tempest configuration file and gives afterwards, after you run all these tests, you have, or you get a nice summary about the failed tests first, then the successful tests and finally a summary grouped by each of the services that you tested. Because the slide is a little bit limited, I simply select it Swift for this case and for the slide. But I think, well, you have, for example, then object store Swift and Nova and similar things. What we also try to do, but this is more like an educated guess that we are doing is to detect the version you're running. So let's take Swift as an example because I'm most familiar with it. We have some services and some middlewares that might be introduced in Grizzly or in Havana or now in Icehouse. And if we see a test that passed, for example, a middleware that was introduced in Grizzly, then we can assume that we have at least a Grizzly release here and later on. And well, it's an educated guess, but it might be useful for some of you and I think it's a nice way. Swift itself got lately an addition called slash info that exposes a lot of your Swift environment Swift deployment information. And I think there are some discussions with another projects to do the same. So that would be a nice way to get a, well, better estimation of what is running. So the command line interface was only a first step for us, but it's not for everyone. As Nick already said, you might be someone, a manager who wants to verify your deployment and you're not familiar with the command line at all. So what are we doing now? We created a simple web application and we're doing the same with the app web application now. We add, you enter your Keystone credentials, just click on start test, wait until all these tests are finished and at the end you can use or you can view the summary of these tests. And then Nick came to me and said, well, let's open it to the public. Doesn't make sense to keep it for ourselves. And that became what's-a-stack. So what is what's-a-stack? What's-a-stack is basically a tempest as a service. Let's call it like this. And now we have another acronym, AAS acronym. It's a website that runs tempest report and tempest in the background. So it's open to the public since a few days and you can just create an account for your own and enter some of your information about your deployments and start using these and execute tests on it. So how does it look like? We have a configuration editor where you just put your username and your password and your Keystone URL into it. In many cases you have only a single tenant within your, for your username. So we simply select that tenant, but in case you have more than one tenant and want to use a different one you can also specify the tenant that you want to use for these tests. And after you save this configuration you can start a new test. And hopefully, well, if you use a DevStack you should see something like that where all the tests passed. Well, this is no surprise. We're doing this or the people from the QA are doing this a thousand times or even more every day on the gate. But this is happening on your deployment now. As you can see, the tests are summarized and grouped by services. And you can have a look at the detail for each of these services. So let's take Swift again. And we have some basic services running here, basic tests, for example, object and container actions. And we have some middlewares that are tested. Quota support for containers, versioning or static web. But unfortunately, not every time you see all only green lights. You will see some red lights probably. So in this case, we also have a Swift installation but now only the basic tests have passed and we are verified successfully. There's a second tab that shows a small red fall with the errors. And if you look into the errors, you see that these tests failed on your deployment. So you might want to be, you might be interested why these tests failed. So in the background, because we were running Tempest we have all the logging information from Tempest itself that we store for you. And you can just simply click on test logs and see what happened there. That might be not for everyone. So it's more focused, this one is more focused to the developer or administrator. But it's very useful because for example, if you're a manager and we go one step behind, we have some links here on the upper right. Share a public link or download and export it as in PDF. So if you run these tests, you can make it, well, let's call it semi-public because it's only known or only accessible if you know the link. But you have a link that you can share with some of your developers or operators or admins. And these admins can then have a look, oh, what happened there within the Tempest tests? And of course you can also download it as in PDF. So behind? Oh, sorry, we have a question. I noticed that you added some text right at the page before where it seemed that the title of this test are coming from. Oh, the title of the test. Well, we added some titles for the test on our own. So because our idea was if you have something like Tempest API object test, test container static, what does it mean? Nobody knows, well, of course the operators and developers know this. But if you're someone else, you might not know about this. So we try to add some useful title for these tests or test classes that people that are not really aware of what is happening behind the scenes can get an idea what happened there and what is tested there. That's the idea behind this. So you might wonder why should I give you my credentials for my deployment? And well, it's a good question. We try to secure this a little bit. I think you're all aware of the security breaches that are happening to even the biggest services around with database dumps going around the world and broken passwords credentials and things like that. And of course we don't want to have this negative press for us. So behind the scenes we have a web server with a web application itself. It's a Django application. Of course we have a database where we want to store all the configurations and of course also the results from the tests. We have a message queue. So each time you click on start test we send a message into the queue. And finally we have some worker nodes and the worker nodes are running Tempest and executes these tests. So we try to separate it. Of course one thing is to distribute the load over more than one worker node. And we don't want to have a crashing sign because there are a hundred people want to test at the same time. But also for security reasons. So what we are doing is if you submit your password, we use a public key on the web server encrypt that password and store it in the database. And the private key part of this key is only available on the worker. And the worker is not accessible from the outside. So there's no as it's H running or HTTP web server, whatever else. It's just behind our firewall and it's just going to the outside but not coming from the outside again. So what is happening here then is that the worker decrypts your password and creates a configuration file for Tempest and executes a Tempest test. And finally we store the results in the database. And because it might happen that for some logging reasons Tempest stores also your authentication token or your password inside the logs. There might be some lines. We remove these lines. So there's no, should be no security credentials in the log file at all. And if you're really paranoid I would recommend that you change your password after the test is run. Yeah, that's a good idea. And because we were focusing only our first step for unknown user tests that are not executable as admin. If you're just using a regular user you should be well, I think it's a good idea to start like this. If you want to run these tests please make sure that you have a quota that supports testing. So what I've seen lately is that people trying to test their deployments. They have, well let's say they have a limit of 10 compute instances. They already have 10 instances running. And you see a lot of failures of course because Tempest is not able to start a new instance. So what are we going to do next Nick? Well that's really a good question. I think first of all we want to see if that's useful for anyone. I mean it's useful for our customer but if we start maintaining it as a public tool as we decided to do, there is no point if nobody uses it. So we are going to be monitoring the use age from now because we opened the website today, correct? Yes. And if there is some update, well we'll keep on maintaining it and if there are people proposing new features on Git, we'll be happy to merge them if they make sense. And if there is a lot of update then I think it could be a good idea. They become a little small project on Stack Forge. But really we want first to see what's the use you could make of it. And yeah. There is also the Defcore REST stack effort. Maybe what we've done could be useful to that. I mean we don't want to pretend that we have revolutionized the world. We're just making a useful, something we can make useful available, right? Yeah, exactly. Oh, I had one, another question. Yeah, of course. Running the test, does it cost money? Well, it depends. So if you're using a public cloud offering and you have, you put somewhere your credit card data into it, it will probably cost you something because we're running the test, we're starting instances, we are creating objects and Swift for example. So it will consume some resources. And it's just like if you start a, for example, an over instance on your own, it will also cost you something. And we try to ensure that the created instances are stopped after these tests. But still it's an ongoing effort for us and also for the templates community. Just have a look afterwards. Is there no instance still running? Would be a good idea for you. So we don't build anything for this. So this is operated by Inovance and we don't charge you for anything. Yeah. Use the microphone in the middle and you'll get candy. And I'm not kidding, I brought them. You're not forced to meet them. Have you pointed at any of the public clouds? You can see whether it works with them. Yes. Yes. I won't say which one, but we've done it for some public clouds. And that was one of the reasons we tried to modify a little bit of the code and have some small patches on our side because we just saw what I was talking earlier on that for example, you're not able to upload an image or there are some customized authentication backend which is normally failing. So. Yeah. We run it against them and I would say a good amount of tests passed for them. But some of the features are not available maybe yet to everyone or on every open cloud. So yeah, what do we test of that? And there are some things that you need to take into account. For example, there is a vendor that has a tenant name that is made of a single empty space I think. So yeah. So yeah, if you want to test on your own, just register yourself on our website. There's another question I think. Any chance you make available an offline version that can be installed on-premises and used to test private deployments? I think we could recommend to use Tempest report that the command line for now. For now we could recommend that one, yes. Setting up a Django plus a worker just to do one test may take a lot longer than just using Tempest report. Yeah, that might be one reason. There's also a project you thought in the community to something, well, it's a little different but it's called T-Cup I think. And it's a different approach than our approach but yeah, we could think about this also. Making a container, something like a container. Yeah, a container, a Docker container. Yeah, the Docker container would be nice. Hi, right. Yeah, yeah, that would be cool. How did you say that you handle different versions of OpenStack is the point more to follow Trump or go for different tests? Yeah, so for now we try to use the latest Tempest release because you have some API channels or whatever else between recent OpenStack releases. We try to patch these locally on the environment to avoid failing tests because just a lock line changed or something like that. But for the future, there's the branchless Tempest effort. So Tempest is getting branchless and there's, well, there's this talk right after this on this level, I think 301. So that you're able to use Tempest with at least the latest stable releases of OpenStack. So if you have OpenStack in a release, for example, then you might be still able to use a recent Tempest version just for testing. We didn't test Ceph yet. And I need to think about it a moment. I'm not sure if Ceph is yet included in Tempest. No, it's not. But I guess in that case, we generally people using Ceph as the object store, we'll use the compatibility layer through the Rital's gateway to make it look like it is a Swift. So in theory, it should pass. But yeah, we must say that we haven't tested it. No more questions? How about automatic test job scheduling? Job scheduling? Yeah. If you want to, you want to test that? So would you want to have the test operate in a recurring fashion so that you can verify that over time you're... Well, in fact, we were thinking about that a while ago. The feature is at least not public yet. But we were thinking about that. We need to figure out how we want to do this because this is at the moment a completely open service from us so you don't need to pay us for the service. Let's assume you have 1,000 users on now. Let's assume you have a hundred users that want to test their deployments regularly. We're only just serving these users. So we need to balance this a little bit. So what we are doing now is that in the case you start a new test, this test is queued and the next free available worker will execute your tests. Of course you're open to just retest your deployment well, a week later or a month later or something like that. But automated scheduled tests are not yet available at least, yeah. But of course if you want to submit a patch you're very welcome. Yeah. Do the user have ability to choose the test cases or they are predefined? No, not yet. We try to, for the web interface we try to simplify this a lot to focus only first on the first step to make a simple web interface. Of course if you're running Tempest report the command line interface. You're free to edit the list of existing tests and maybe just the tests only were swift on over or what you want to test. So there's no, well, you're just open to it. Christian, have you seen who is going to ask the next question? I think we should run. Yeah, we should skip it. I am a colleague of those guys. But that's a proper question. Did you think about having an API? So you can have an API so you can specify that you want to be able to test. So basically you can automate those things or whenever you did some change in your cloud you can send an API call to here. So you can test? Well, we don't have an API yet. But, well, that comes to Tempest as a service then I think something like that. So to define. So you have a restful interface for it. It's a nice idea, I think. We need to think about that, I think. Might be a good addition to our web service. Thank you. Yeah, thanks. What's the stack has? Sorry? What's the stack as a service? Yeah, what's the stack as a service? Yeah. Yes, sorry. How do you select which test to run? Well, the test we run we were first focusing on, well, it's not the official don't take me wrong. It's not the official core definition but test that we thought of our core for functionality. So to create servers for example and boot servers and stop them and assign different flavors. So we selected a list of tests that can be run as a normal user and tried that on our own deployments. And, but for the future I think we have a close look at what is happening within the community and especially in Dev Core because there are a lot of people thinking about exactly this topic and which tests we need and which tests should be run. So, well, for the moment we selected them on our own but I think we will closely follow the Dev Core definition. We are. Yeah, we are, we are already. I guess that's it. Thank you very much. Yeah.