 Hello, everyone, and thanks for joining my session today. Today I will talk about the tools for better testing for open source projects. In this talk, I will use OpenStack as an example to introduce the approach and tools used for OpenStack community for testing and to explore how an open source project can do QA in upstream and how open source projects can use tools to improve their QA process and the potential benefit it can provide. First introduce myself. I'm Dong Ma, currently living in Beijing, China. I'm working for Holy Packer Enterprise as software engineer mainly focused on the free and open source software development. I have been an active technical contributor to OpenStack project since the OpenStack library release circle. I'm also the co-reviewer of the OpenStack subunit to circle project. Currently, I'm working for Holy Packer Enterprise OpenStack team, mainly focused on the Jenkins OpenStack CI and the QA part, the upstream contribution. I also work for the Fosogy project as a co-contributor before. It's an open source lessons analysis tools. Quick review of what I want to talk today. First, I will quickly introduce what's the QA and why are they important for open source projects. And second, I will introduce what's the OpenStack QA and next to introduce some tools for better testing in OpenStack. At the last, I will introduce how to leverage the OpenStack way in your open source projects. OK, what's QA? I think most of the software developer or engineer is very familiar with this world. The QA is short for the quality assurance. It's a way of preventing mistakes or defects in the manufacturer products and to avoiding some problems when deliver some solutions or service to the customers, which the ISO 9000 defines as part of the quality management focus on providing confidence that quality requirements will be fulfilled. And for the software development, most of the QA is mainly for the testing things, like all kind of tests, unit tests, integrated tests, functional tests. Also for the tools and the process to automated testing and the tools to analysis and report the test results. So why the QA is important for open source projects? So for the traditional QA software development, it makes the commercial software in stable and rapid successful. Under the traditional software development model, software quality assurance is to constitute the set of systematic activities, providing evidence of the ability of software to generate the software product that is fit to use. I think the traditional QA is dedicated with one team or group to work on that. And for example, it to investigate the effectiveness of the reviews and the efficiency of the test the technicals. However, for the quality assurance in the open source project, it's not abundant or not well supported. As for the open source project, all the source code is available for all the contributors to developing, writing, or debugging for the rapid evolution. So in most of the open source projects, they didn't do much about the QA in upstream. Sometimes the upstream code base may not be very stable or easy for user to use them. So actually, the good QA process will make the open source project more stable and rapid development. So what's the open stack QA? The open stack QA defined as develop, maintain, and initiate tools and plans to ensure the upstream stability and the quality of the open stack. And its readiness at any point during the release circle. The open stack QA project team is made of multiple projects, which each run independently towards the goal of the project team to make the open stack more stable. So to understand what's the open stack QA, I first introduced the open stack CI, a continuous integration workflow to how to support the QA process. And what happened when you do code changes in open stack projects? In open stack, it used a score code review system. Instead, it directly submit code to the washing control system. The developer first submit the code into the code review system. And then it will trigger the check test. And the check test pipeline is do some like unit test and functional test to score the code changes. If you get the field feedback, they will get back to the developer as soon as possible to check the log what have wrong in the code changes. If it passed, it will go next for the call code reviewer to review the code. Only get the check test pass with plus one. And more than two call code reviewer gets the plus two score the code also ready to be merged. Then it will trigger the gate pipeline test. The gate test refer to the process of running tests before the developer's patch set is merged. The intent of running the test is to validate that the new changes will not introduce new bugs. Also do some regression testing in the gate which to make sure that the changes merging will not change the behavior since the test was last running in the check test. Also some projects also do some additional tests in the gate queue. After the gate test passed, the new code changes will be merged to the washing control system. After that, there will be also some process in the post queue. I didn't show it in this diagram. Once the code landed to the washing control system, the post queue will do something like to publish the documentation to the live website. Also to implement that workflow, the OpenStack infrastructure consistent with the workflow. Showing the picture, first, the developer you will submit the changes where the gate review command line to the code review system. We call it Garrett. The Garrett is connected with the Git repository. In OpenStack, the Git repository is mirrored on the GitHub and Git.OpenStack.org. When the changes submit to the Garrett, there is also a trigger, the check and the Git pipeline is managed by the Zoo. Zoo is also a tool to manage the Git check pipeline in OpenStack. And the Zoo is communication with the Jenkins master through the Gilman server to start the job. And the job is running on the Jenkins lever. It's managed by another tool called the NodePool. NodePool is a service used by the OpenStack to manage and deploy a pool of DevStack images on the real cloud server to use for the OpenStack testing. So based on the OpenStack CI workflow, the QA testing is automated triggered by the check and the Git pipeline when you submit a patch into the code review system. So what kind of tests running in the Git pipeline? There is a bunch of testing jobs running in the Git testing. First is Python unit test. They separate the unit test with many Python environment working like the Python 2.7 and the Python 3.5. We also have some style rules checker to verify the code styles. And also the biggest part is called the DevStack and the Tempest, which these two types of tests is running on the real OpenStack cloud environment to verify the OpenStack function and API works. To implement the OpenStack QA testing process, there's a lot of QA projects maintained under the OpenStack QA repository. The Tempest is OpenStack integration testing and DevStack is a series of extensible scripts used to quick bring up a complete OpenStack environment based on the latest version of everything in the Git master. Also hacking is for OpenStack style guidelines. Granite is a testing harness to execute the OpenStack upgrade process between different releases. It uses DevStack to perform the initial OpenStack install and then work through the upgrade for each project. Also the OpenStack health and the stack width are the virtualized tools to show the test result and test status. Okay, next I will give some... quickly introduce of the most important tools we use in OpenStack QA and CI for the better testing. I will introduce following test tools, the DevStack Tempest subunit to circle, of stack health, elastic recheck and stack width. For the DevStack, DevStack is a series of extensible scripts used to quickly bring up a complete OpenStack environment based on the latest version of everything from Git master. It is used actively as a development environment also as much of the OpenStack projects functional testing. The DevStack's mission is to provide and maintain tools used for the installation of the OpenStack from the source. It's suitable for the development and for the QA testing. DevStack has a bunch of comfortable scripts. You can just install it with a simple configuration for some sanity testing. Also you can with a large configuration to deploy like OpenStack cluster. In OpenStack QA, the DevStack is a base to create a real OpenStack environment for the testing. I think most of the OpenStack QA testing is based on the DevStack. Next is Tempest. Tempest is a set of integration test tools against a live OpenStack cluster. Tempest has a lot of tests for the OpenStack API, validation, and scenario testing. Also it has other specific tests useful to validating OpenStack deployment. Like I mentioned, the DevStack is to deploy an OpenStack environment and the Tempest is running tests against on the DevStack. The OpenStack QA team deployment and maintain the core Tempest library and plugins, also some core OpenStack service test cases like NOAA, Neutral. Our other OpenStack service projects are maintained by their own test cases in their own test repository. Next one is OpenStack Health. OpenStack Health is a dashboard for actualizing the test result of the OpenStack CI jobs. There are currently about 12,000 jobs running the check-in pipeline daily. So it's a big number of jobs for the developer just to check the log. So it's very important for developers to use virtualized tools to check the job status. OpenStack has two parts. First one is some Python DB API to query the test result from the database. And another one is a front-end JavaScript to generate some chart and diagram. The OpenStack Health is the resource of OpenStack Health is from the subunit to circle and the elastic research. Subunit to circle is a tool for storing the test result data in a circle database. It is used for the OpenStack Health to generate the virtualized result. Also, subunit to circle provides a DB schema and a Python API for interacting with the database. Currently, the OpenStack infrastructure team maintain a MySQL database server and all the test results are stored in the MySQL server and it's also public to query the result. It only maintains a six-month test result. This chart shows how the subunit to circle works. As I mentioned in the OpenStack CI workflow, the individual test is running on the Jenkins lever node. And the Jenkins master shows the SSH to get the test result and store it in the logged server. When the test is finished, it will trigger the Gilman server and the Gilman subunit worker will query the subunit test result and convert it into the circle to store into the MySQL database for OpenStack Health to use. Next is Elastic Recheck. Elastic Recheck is designed to answer the question, have you seen these bugs or errors recently? It leveraged Elastic Search to identify the failures with known fingerprints. It contains a repository of Elastic Search queries with known failures. It has two parts. One is a bot which can watch the changes and reports to identify the failures by the fingerprints and reports to the gallery and RSC. Another one has a dashboard which shows the failures by the category session. The dashboard of the Elastic Recheck, you will see the frequency of the hitting of those bugs. For example, this bug has eight failures in the last 24 hours and 22 fails in the last 10 days. Another one is a tool called StackVis. We have this tool to virtualize the individual CR test results. The high level is show the test jobs running time and how many test cases is run and how many failed, how many skipped. Also, you can go into each test to check each test case running time and which one is failed. I have a quick demo to show how OpenStack QA to monitor the test status. It can be accessed by anyone. You can check this website link. This is a website of the status.openstack.org. There is a lot of things. First one is the tool. The tool is the management about the check and git and the post pipeline. You can see all the check test status and the git test status from this page. Also, this one is the elastic retakes I mentioned in the slides. This is the live status. Also, see this bug is showing frequently in this time. Next I mentioned is the OpenStack Health. OpenStack Health is a dashboard that shows the test result of the OpenStack Theatre jobs. It's implemented by the JavaScript so you can check some status on the chart. Also, there's a bunch of job failures by the rate. That is most of the OpenStack QA team to monitor the test result. Next part I will give some tips about how to level the OpenStack way in your own OpenStack project. First tip is try to understand the importance of the QA in the OpenSouth project. I think some of the OpenSouth project didn't care too much about testing in the upstream. It's very important for the project. It can make the code base very stable. Also, the next one is all the CI and QA process in the OpenStack is also OpenSouth. This kind of tools and process you can all get from the OpenStack website. You can use it for your own OpenSouth projects. Next is a tool I mentioned earlier, all is OpenSouth. I think the DevStack and the Tempest is most specific for the OpenStack scenes. But the other tools are not specific to the OpenStack. It can be reused to other projects. Yes, that is all my talk today. All the things you can get from the website about the QA OpenStack. And this is my contact information, the email and the RSC FreeNode nickname. Okay, thank you. Questions?