 Okay. Hello, everyone. Welcome. Thanks for joining us today. My name is Andrea Frittoli. I work for IBM and I'm the current PTL for the QA program. Hi, my name is Ken Ichi. I'm working for NEC and I am previous QA PTL for Newton and Ogata Cycles. Okay, so just before we start I wanted to get an idea of how familiar you are with QA program and our tools. So maybe if I can ask you a favor if you know about DeskTec or if you used it before, if you could raise your hand. DeskTec. Oh, DeskTec. Yeah, okay. And if you know about Tempest or you used it before, okay, that's good. And if you know about PayPaid and you hate it or love it, sorry. Okay, okay. I will explain something as a first. So as a first, FAT is a QA project. So QA project is quality assurance project. And here is mission statement of QA project. Develop, maintain, and initiate tools and plans to ensure the upstream stability and quality of OpenStack and its readiness, its release needs at any point in release cycles. Most important thing is that we ensure this readiness always. On upstream development, developer posts patches to get it. And these patches need to be passed on the gate testing. These tests verify each patch does not break existing tests. To do that, we ensure each patches, the repository is always released readiness. That is very important thing for us. And QA project is horizon mental team like doc team or something. And users are OpenStack developer, package and operators, not normal or end users. So here is the project background. QA project founded during the fourth summer release of OpenStack. And the first IRC meeting was August 2012, almost five years ago. And at the early stage, we had very few contributors. But now, we have almost 200 contributors for the previous orgata release. That is a great thing. And thank you for contributing so much. And user adoption, as I said, QA component are used on most OpenStack project for CI, which means getting tests. And upper graph shows how much jobs runs on the gate. Each jobs contain 30,000 tests. That means, so graph shows 1,000 jobs per hour on average. That means we are testing 30 million tests per hour on the gate testing. That is a great thing. And QA team is concentrating on six core projects like Cinder, Nova or something. But we are on big tent policy and we have many projects except core project. So we needed to provide some mechanism for verifying stability for this project. And we provide plug-in mechanism for some component that is grenade, tempest and DevStack. Most lower graph shows how many projects are used each component, each plug-in mechanism. Biggest number is DevStack because big tent project want to deploy their own component for testing on the gate. And we are here, many companies used tempest as internal integration tests, sweet internally. But now QA project is out of scope from user survey. But we'd like to get some feedback from users and we have QA forum tomorrow morning. So please join if you're interested or you have some complaint or some idea. Please join this forum. Oh, sorry, this is my slide. So we have many components for QA. And this slide shows part of QA component, not all. And most famous component is tempest, the integration test suites and grenade is upgrade test suites. And grenade verifies upgrade does not remove existing resource or we can use old configuration on new component or something. And patrol is relatively new young projects that we started in this year. And part of the role verify with Policies Jason, Policies Jason works fine. And DevStack is development tools for developers. Please don't use DevStack for production environment. And OpenStack health, StackVis shows some gate status for developers and hacking bash hate. S-Brain is code-style checking and OS-Testal is testal wrapper for OpenStack specific project. That is all for me and please explain Pyke's theme. Yeah, so we kind of have a common themes across the QA project for Pyke. So we are a horizontal team as you said. And so we serve the community. And we want to provide developers with good tools because they use them in their daily development work or testing. We run a lot of this in the gate, 30 million per hour perhaps. So we really wanted to focus this cycle on stability and polishing this tool up and make sure that they are nice and usable. So in Tempest specifically, for instance, we have a set of APIs that are exposed to the OpenStack communities. They want to implement tests using Tempest plug-in framework or just using Tempest runners or just the library. The fact is that a lot of plug-ins or tests outside of Tempest, they use today modules within Tempest that are not marked as stable APIs by the QA team because they were not ready, but the project needed them so they just start using them. So we are going through a kind of slow process in this cycle to clean those up, trying to have the least possible impact on our consumers with the end target to have them marked as stable and maintained so we can ensure a backward compatibility in the future on every interface which is needed by the community. Unfortunately, having those stable APIs is not enough to have a stable gate. So we saw in the beginning of the pike cycle, we had the gate in rather bad conditions so the number of false negative was piking which gives a pretty poor developer experience. So if you submit a patch and it fails and then you go and look and there is an error which is totally unrelated to what you are doing. It's like okay, you spend time investigating and effort in there and so it turns out that it was a combination of things that evolved over time. So some of the services that we run in the gate, they had a memory footprint which grew over time and we added more services. We added more tests which generated more load on the system and the test. So just to make the combination of effects even worse, we had a version of leap-beard which under the specific stress condition that was in the gate was starting to crash. So then we had kind of untuned my SQL as well so there were quite a number of factors that all together created a situation so by solving them one by one we managed to get the gate back into stable place. It was an effort done not only by the QA team at all but with many people from all the projects they jumped in and helped. So it was thanks to everyone who helped here and hopefully we can have ideas in the future to avoid this from happening again because it was kind of a showstopper. Everyone had to look into these fixed things that we couldn't do anything and that's why I put it as a theme because it was really like taking a chunk of the beginning of a pike. So the other thing is I was mentioning briefly usability is it's all good to have stable interfaces but if it's totally unclear how to use them they're not useful at all. So we are focusing a lot in creating better documentation with plenty of examples so that you can get codes to see how basically how to use the interface, what is recommended way to write code to use our tools. Something that is kind of always a theme for us is on interoperability we are not directly responsible for interoperability that's another project and a team that looks into that but they use our tools. So there are changes that go through Tempest or some of our tools for which we kind of have to raise a flag, a red flag and say okay well this looks like a change that if it goes through it means that there could be a backward incompatible. Change in an API of some service so we kind of have this role of we cannot really get on that but we can at least raise a flag so that the project or whoever is involved is aware about that. More specifically so what did we do in Pyke? So for Tempest we increased the surface of stable interfaces we are still doing it we still have some Pyke time ahead of us and we plan to work on increasing the number of schema that we have. So we have for NOVA we have JSON schemas that we use for validating the responses on the APIs and we want to have these for all the services. This is a kind of midterm target that we it's going probably to go beyond Pyke and we're working a lot on documentation. For destec point of view we did some performance tuning also destec is mostly bash code but some of the function that used to be written in bash they've now been moved into destec tools. It's an external project which is Python based so they are easier to maintain they have nice unit tests and everything. We switched to the even to cloud archive packages for Libvert so we could get the latest or a later version of Libvert which sold some of the instability issues in the gate. Also we started running the services under system D as opposed to screen which brings a usability benefit in terms of reading filtering log when you're developing there are some utilities you can use in combination with system D to filter log files. So yeah a core new feature of course is patrol is the new service that we have. One of the main things of course is to build the test coverage for the core project. So Kyrsten Glansnova and Cinder Neutron Swift. And because it's a new project you have to set up the repository build set up the CI make sure you have docs releases jobs everything generated so all the kind of administrative things around creating a new project. And that's pretty much in place now so shout out to the patrol team but the great job during pike to get this in place. And of course still during pike and then beyond the idea is to get other projects beyond the six projects involved in this because one of the things that patrol can do apart from testing. The policy is actually enforcing a common handling of authorization failures and policy across projects so it's interesting for a project also beyond the six core ones that we handle in QA to test for policy. And so we'll need some kind of plugin mechanism similar to what we have in the other projects. Open stack health and stack bits they are really I find them really useful at least personally I use them when I have to debug the gate because with open stack health you can see trends. So if you see a failure in a test you can have more context you can know oh this is the first time it fails or it fails every now and then it's really flaky test why do we still have it there or you can find this kind of context and it helps a lot. So it's in a state where now where it's there it's running it's fine it gives you some information it could do more but we don't have many contributors really we don't have bandwidth to contribute much to these projects. So if you're a JavaScript developers developer and you want to contribute please reach out to us we really need help with these two projects. And upgrade testing so we had some discussions in the beginning of the bike cycle at PTG about doing things for rolling upgrade zero downtime upgrade which are very interesting for operators to have. With grenade what we do now it's very useful we make sure like you said that we can use the old configuration that we don't break things that way and we don't break existing resources. But to do that we shut the entire cloud off then we do the upgrade and then we turn everything on again it would be more interesting scenario for operators to know that we can actually do rolling upgrade. And this is something that Nova started to do in their own gate for instance using grenade upgrading only one of it to compute. But yeah so we had some volunteers from Ozick that plan to work on these but because of the news you have heard with Ozick not being there anymore so they're not working on these anymore. So yeah this is an area where we could use contributions again. So beyond pike so what are the themes that we have in mind. So ideally we would be able to increase our scope do more things do rolling upgrades do new cool things new type of broad new type of testing do more things. And also scale. Yeah yeah so do do more things maintain the things that we're doing but in reality. For QA and other horizontal project it's kind of not so easy to get. Permanent new contributors so we get a lot of occasional contributors but it's kind of hard for companies to say OK well I will invest. 10 of my very good engineers and doing QA and infra. Because you don't get a direct or obvious return out it's not like OK you put these engineers in QA and then you get a new shiny feature in Nova coming out of it. You get benefits but they are kind of less directly measurable. So it's kind of hard to get more contributors which are 100 percent committed to this project. So there are other things that we can do so we can make sure that we can benefit as much as possible from occasional contributions which is something that we are trying to do but we need to do better. We can streamline our processes make sure that it's easier to follow through to use the QA tools to maintain them over time so we spend less time in that. But there are the things that we can do for instance if we want to do more projects like we want to have rolling upgrade testing we want to have scale testing. These are things that a lot of companies they're maybe already doing in downstream and they're like parallel effort happening in different companies. So if we manage to bring some of this effort together in the open then it could be beneficial for all the companies. There are a lot of subtleties and difficulties in doing this because maybe there are different type of scenarios different type of deployments. But I think the type of deployments of OpenStack they're kind of getting more aligned and we're getting to a state where it could be possible to do these more and then get new contributions in this way because then it becomes more like for a company. Well rather than investing all this effort in doing this kind of QA locally I could put some of the engineers in maintaining this process in upstream together with other engineers and then we all get a benefit similar to what was done for instance for the stable branches or for OpenStack in general anyways. Another theme that we could have for Queens is adjacent communities sorry I don't know how to pronounce that in English like Kubernetes or Ansible so that's critical for OpenStack to start working with this kind of communities. And even from a QA point of view it makes sense probably to share at least best practices or experience we have a lot of a few years of experience behind us in running CI and running tests for OpenStack. And it probably makes sense to for some of us to go out and try to and talk to people that they're doing QA in these other communities and see if there are things that could benefit from contributing there in these communities. And hopefully that will also come back and contribute into OpenStack. So this is also a way we can create more interest in our tools and more get more contributions. Interoperability of course stays there as a thing that we always have in our mind. In terms of more detailed features that we could think of for Queens, well as I mentioned rolling up rate testing is always under the radar. We could think of fault injection, a CHA type of testing is something which is difficult to have in the gate but it could be done in different kind of scenarios like periodic or third party testing. There will be some work to be done again on tempest on schema, adjacent schemas that we will continue over to Queens. One other thing that was mentioned earlier, the problems with the gate in the beginning of Pyke, I think we could do and it would not be too difficult to build some kind of automation to monitor this. So we already collect a lot of data, we collect these stat logs from the gate runs. It would be interesting to collect this data and aggregated over time and monitor it so that we can see if we start having deviations. A single run is not statistically relevant here but if we collect and aggregate the data we should be able to see that at some point in time the footprint, the memory footprint for a certain service is significantly increasing. Sorry, like 10% more or so so then we can take action before the actual entire gate collapses because of some of the effects. So this is something that we could work on maybe together with the infra team and try to get this sorted out so that we don't have to invest all the time in this again. Other things we could work on, I was mentioning non-functional testing on third party premises could be a first step to get results from scale test for instance on downstream premises. But get this result publicly available to be shared. Across community testing there is a feature in Zoo Free which hopefully is coming soon which will allow to do something quite interesting to use if you know it depends on a feature in the gate. So basically you can write a patch in a project and you can say this patch depends on a patch which is another project in a different project within OpenStack. And then Zoo will take the different pieces and put them together and run the test with a combination of patches. So now Zoo Free will support doing these across boundaries. So it should be possible in future to say something, I want to test this change in Tempest based on a specific change which is in test tools for instance or in a repository which is not maintained as part of the OpenStack ecosystem. Kubernetes or something? Sorry? Kubernetes or something? Oh yeah, Kubernetes, GitHub or different kind of backends. Great. And maybe there is space for some of the QA tools to be used by other communities. I don't know, there is another community using Python that could want to use Pep8 because everyone loves Pep8 so much. So maybe. So beyond Queens and I'm almost finished. So rather than real plan, kind of vision. Vision. Because it's kind of away in the future. So you may have seen the effort that you see is doing in terms of creating a vision for the TC. They're talking about constellations. It's not fully clear to me what a constellation is yet. But it's basically the idea is to have a combination of services or a collection of services that OpenStack says to users you can use this combination because it makes sense for certain scenarios. And this would have an impact. It's kind of in the direction of maybe reference architecture slightly but not necessarily. Anyways, this kind of combination would have an impact in terms of QA because it would make sense from my point of view if we propose certain combination of services to have a tighter integration between those services in the gate. So like extend the concept that we have integrated gate for six core project. If we say, okay, well, it actually makes sense to have a combination scenario where you only deploy a Cinder with Manila and Keystone. And then we need to test these three services together in the gate. We could continue working on the area of non-functional test and bring them kind of in the open and have common frameworks to be used there. It would be interesting, I think, to have a common framework for doing kind of scalability or performance testing that can be used by different vendors using different drivers against different type of hardware so that we could then compare results having a common tool which makes things easier. We could still work with contributors from adjacent communities and, yeah, of course, keep an eye on interoperability as usual. Some references, the priorities that we have identified for Pike, the PTG is the first link, then documentation for the various projects we have there. And if you want to contact us, we are in OpenStack QA in IOC or you can write to the mailing list, OpenStack Dev. You can quote QA in the subject for people who use filters. We have weekly IOC meetings because we are kind of diverse team geographically, so we have different nations across the world. So we have two different times. We alternate for QA meetings so hopefully you can find one that fits your time zone. Or you can talk to us face-to-face here at the forum on the PTG. And, yeah, thank you. Thanks for bearing with us so late in the day on Wednesday. And, yeah, if you have any questions, we should have some more time in the session, so feel free to ask. If you want to ask questions, you may want to use the microphone. Yes, I have a quick question. One use case at AT&T is we had different sender driver backends. And so the way we have been running Tempest in the past is we would have to change the sender.conf for that. Would you guys be open to possibly, like within Tempest, like within the Tempest.conf, specify the driver? Is that something that we could do in Tempest? Yeah, we have a similar discussion. So, specify the driver directly. Probably it's not something that we can do. But we do have, we can have more feature flags where it makes sense. So if certain features can be enabled and disabled, then you need to have the logic in your gate to turn on and off features based on the driver that you're using, basically. Okay, thank you. Just wondering how far you usually extend backward compatibility. My team's still on kilo. Okay. So, we have a deprecation mechanism for APIs. Okay, so in terms of stable APIs, unless we deprecate something, it will be supported. If we want to change something, we start a deprecation cycle that normally takes a cycle. So you get a warning for six months or so. And then if you're still using that API in the deprecated way, you will have a problem. So you can pin the version of Tempest that you use. It's on PyPy, so you can pin that version. You still use the old version of the API, or you can upgrade to the new version of Tempest. In terms of which cloud, which version of the cloud you want to, you can target, it's similar to the support we have for the releases in general. So when a release goes end of life, we stop running Tempest against that version in the gate, which means we don't do anything proactively to break it, but we don't verify anymore that it works, basically. And then we will start deprecating configuration flags that were specific to all the versions. So then you can probably still run most of the tests. You may have to filter out with regular expression a few tests or so. Okay, any more questions? Okay, thanks everyone. Thank you.