I'm Masayuki Igawa from NEC Solution Innovators.Thank you for coming to this session.This session is Tempest Scenario Test.Let's get started.Here is the agenda.First of all, I'd like to introduce myself.I'm an OpenStack Active Technical ContributorI'm a member of TempestScore.The next is the overview of OpenStack.Maybe you know the OpenStack.This is the overview of OpenStack.OpenStack is the most popular OSS Cloud Infrastructure.It consists of several loosely coupled components and many features being developed with six months release cycle.It's a very high multiple component.It's one of the greatest concerns from developers and operators' viewpoint of OpenStack.Operators' viewpoint of OpenStack is one of the greatest concerns from developers and operators' viewpoint of OpenStack.This wish is they want to verify the cloud as well as in the short time.Because to avoid any regression and to compare the stability of different software versions.And in some many situations.But how to define?We need a test suite for it.But do you know Tempest?Maybe you know the machine session.But I'd like to talk about Tempest's overview.Tempest is one of the projects of OpenStack projects.It's a set of integration tests to be against live OpenStack cluster.Tempest runs in every patch code.So we need to have Tempest code for integrate the project to ensure the validity.Next is the purpose and use case of Tempest.For developers, new codes can be tested to check the expected behavior.And the second is to verify whether new codes introduce any regressions.And the third is on each new patch, the Tempest runs in through to make sure it qualifies the expected quality and stability.This is a diagram of code review diagram.And the red square is Tempest runs.And the second is operator's viewpoint.Taking their cloud works correctly.That is a test suite for production environments.And the second is to check and avoid regressions while software upgrades and so on.And the third is can compare the stability of different software versions.And another is RefStack and Defcore.RefStack uses Tempest as a verifying tool set.RefStack is you can check the URL.So next, I'd like to show you some key points of Tempest.The document says the gate job should install the project using the DevStack and run Tempest tests.So implementing of Tempest tests is one of the minimal graduation requirements from the incubation project.And there are 12 services in Tempest now.These are all, yes.And next is the growth of the Tempest code.The code is growth in the significant country.That is 10 times more than the year below.The next is Tempest source code directory.Tempest source code directory is separated by its category.So the next is a system test for verifying between multiple OpenStack components. Other categories are not suitable to be implemented in complicated scenarios between multiple componentsbecause they are aimed to validate a single component.So I'd like to talk about overview of Tempest scenario tests.I'd like to give you some key points of scenario tests.Scenario tests are through-part tests of OpenStack function.It's a complicated setup where one part might depend on completion of the previous part.And they ideally involve the integration between the multiple OpenStack services,Opposite components to exercise the touch point between them.So the key point is, sorry,across the multiple components and sequential testing is a scenario test.The next is a scenario test should be used by the official Python clients' library.Tests should be tagged with services they exercise.So for example, if the volume service is not available by Tempest config,this test will be skipped because of the volume is present.And the goal of scenario tests, from operator's viewpoint,operators can use the scenario tests for validating their crowdswith simple process such as one command or one click.Of course, the validating the process should be automatic in the production environment.But anyway, this should be the simple process.From the developer's viewpoint,they can check whether new codes will cause any integration on the other components easily.For example, the grenade is using them now.So as a result,we have been able to verify the open-stack upgrade process in a short time.Full test is most of 45 minutes, but the grenade is 10 minutes.And this is the overview.Here is the overview of the scenario test, test cases.There are 24 scenario tests now.But there are some missing services such as telemetry and database.That's a thermometer and a trove.Like this.I talked about overview of scenario tests.Next is how to use scenario tests.I'll explain how to use the scenario tests with using the source code.First is a cloned source code like this.Next is copying and customizing the configuration file.That is a Tempest Conf beam and Emacs and so on.Next is a run Tempest.That is two ways to test our commands with test our commands or run Tempest script.Next is the first is to check the result.There are about 300 configuration options.So the second thing is very difficult to modify the configuration for that.Here is the result.The above test case is failed, unfortunately.In this case, I need to investigate the cause of the program.Another is okay.These tests are finished normally in these seconds.Another result, this is a failure case.In this case, these are details of the failure.We need to check these logs and server logs.Demonstration.Demonstration on DevStack code.DevStack is the best tool for starting to use Tempest.I'm going to execute number 7.Number 7 is here.This one.This one is a basic image and instance volume network and CRUD test.The next is to check the result.I played the demo.These are my kids.Thank you.This demo is interesting but not fun.Because only CLI and flow the log.But I will do.This is not live demo.Only automatic demo.Sorry.This is not here.This is the start but running the test.This may take only this code, this code will take two minutes.So I move forward.So this is a Tempest log.So I need to move forward first.It took time two minutes.In this case, success succeeded.After the test, cleaning up process is proceeded.Yes.Not so fun.OK.That is demo.OK.Next.Next is issue and future of the scenario test. Issues.We need more scenario test.That is more services such as telemetry and database.Neutron.Yes.Of course, we need Neutron and maybe Horizon and Nova.All components needed.More complicated but popular scenarios.I have no ideas at this time.But maybe more complicated scenario is required.Next is difficult to write scenarios.Because it needs Python development skills.And another difficult thing is to prepare setting for existing clouds.Tempest Conf has 300 options.Next is difficult to analyze the cause of failures.Elastic Recheck is one of the solution for that.But it's only for developers, I think.So we need more easy to analyze way is needed, I think.The future thing is more easier.Operators want to verify the cloud without command line skills, I think.So the Tempest GUI is one of the solution, I think.And more useful.We can specify nodes and zones by setting.We should specify nodes and zones setting.So I propose this patch and under the reviewing.Yes.The summary.The summary is OpenStack has official test suite that is Tempest.This is not only for developers but also the operators.Also operators of OpenStack cloud.Especially the scenario test, test cases across multiple components.It can be useful system testing for OpenStack cloud.By using the scenario test, it's possible to reduce the evaluation cost of your cloud environment.So in comparison with making test suite from scratch.So increasing the test scenarios and more useful, please join us to enhance scenario test.This is communication plus path.OpenStack development mailing list.This is not for usage questions and IRC.Okay.That's all for me.Any questions?Yes.Can you use the microphone?Hi.In your last slide, you mentioned that you can also use it for system testing.So it would be possible for you to elaborate on what your definition of system testing is?Levelage.How would you categorize system testing?What would you define as system?System.System is...We need integration tests such as Nova and Keystone, Grants, Cinder.But except scenario tests, they are individual component tests.API test is only one component for testing.So scenario testing will be in the scenario system test for that, I think.So are you talking more at a point of component integration testing where various components talking to each other?API test is component integration test.Only one component.Individual component tests, I think.But scenario test is throughput test.So such as boot instance and attach volume and delete instance and reboot instance and attach SSH.So we can do that with scenario test.So that is a system test, I think.Thank you.Hi.Quick question.So with the scenario test, it almost seems like you might have the same problem that exists with the negative test a while back where people came up with their own individual,Oh, I think it's important if I do A, B, C and somebody else says no, I want to do A, C, D where you have different...You might have a bunch of different permutations.Is there a set of criteria used right now to decide what becomes a scenario test or what is maybe something too specific or something that is too off the wall?That maybe depends on the scenario.So probably the answer is pull request accepted and take a look at it.There is no answer to the solution for that, I think.Thank you.Thank you.Hi.For myself and perhaps for many here, I'm interested in this from an operator's perspective.When we're setting up a open stack installation, you want to be able to verify that it's working the way you're expected to.This project looks like it originally came from the dev world and now it's being applied more from an operator's perspective.You touched earlier on a slide about configuring open stack can be configured in a lot of different ways.What kind of setup work needs to go into using this tool because how easy and how quick it is set up affects how useful it is for an operator.You mean the component configuration?You talked about more scenario based testing.If I roll out an open stack installation, I want to be able to verify that it works the way we expect to before we put it in production or start using it.How much work do I need to put into configuring Tempest for a given setup?Configuration is not only scenario test but also the whole Tempest issue.I don't have the answer for that right now.We need to resolve with other developers for the solution.Sorry.My friend said earlier, a lot of the Tempest tests right now, the scenario tests, they have to be written in Python.Is there anything on the roadmap that makes it easier for operators to write test cases in a language that is not a programming language?So, I don't have enough roadmap right now.But one of the Tempest GUI is this Juno Cycles design summit topic.I have the session for this topic.Anything else?Scenario test experience that there tend to be more flaky or more time consuming than API test of course because they go on more complex scenarios.If we start adding more and more of this test into the gate,is there an impact on something that is affecting the developers?So, how do we cope with this?So, how do we keep a decent number of scenario tests?How do we keep them tested in a way that is good for operators to run them but good for developers because the gate doesn't take three hours to run?You mean, more scenario tests take a long time to in the gate?Exactly.They take a lot of time to run?Yes, I think so.But there is only 24 scenarios now right now.So, maybe not so impact in the gate right now.So, as a first, we need to more scenario and increasing the execution time.So, if that's all, we need to create a more scenario and then we should think reducing the execution time.So, I don't think, not so concerned about execution time right now.Okay, so the gate is still the right place at this moment to put this kind of test?Yes, right now, I think so.Hey, Derek Anderson from Verges Stream.I've got another operations related question.Obviously, people are really interested in validating.They've got their install working correctly using Tempest.But in an ongoing basis, imagine I've got a situation where I want to run a test or run all the tests actually against a production environment.I know the tests clean up after themselves, but do we consider them to be safe against the environment and that they're going to back out the things they do and leave everything in the state they found it?For example, if I get a customer complaining that they've had some kind of flaky behavior and we don't see anything that's easy to diagnose, can we run these tests and know that when they're done, I haven't done more damage or should we not expect that?How safe is your feeling about that?I know it's a hard thing to say for sure.Yeah, that's right.Maybe we need to completely clean up all Tempest-created things resources, but there are some bugs in the Tempest.So the resources are remaining right now, but we need to fix the bug.Anything else?OK, thank you very much.