 Okay, so today I want to show you a new version of testing in QA, very interesting to us and I hope that we can get as much as possible feedback. First question here, how few is not a member of the community? So I want to give then a quick overview about what is related to our Firefox what we have to do for release testing, how automation has us here, who is in QA? There is only one point which is related to cloud. So testing by this, this is the biggest part here, which is about automotive cloud and in the end how it gives the demonstration how it works. So what does it mean for us QA for other leaders? So Firefox 4.0 beta we have lately switched to release cycles of two weeks. That means a lot of testing for us and there are mostly hundreds of bugs which have been fixed and new improvements have been done. So the owners of this component have to watch all the bugs that have been verified and fixed and that's a lot of work. Besides Firefox 4.0 we have two security branches. We support build with 3.5 and 3.6. We have security leaders about four to six weeks, which have normally 50 to 80 impact states and those two leaders normally have these at the same time. So we have two of the twice of work for those leaders. And lately when we have two weeks we have 4.0, we had three leaders at the same time to handle. So what is necessary for us for testing for the reasons? It's more for the 4.0, we have to do some exploratory testing. We have our scripting testing on the move. We have to run for all the areas which have been changed. That's the second point here. So we have approximately 40 different smoke tests we have to run on each platform. And about 300, there will be a test. We run normally only on one single platform because there's a lot of work for us. One person will need about one day to run for all of those tests and that happened three years ago with the automation in place. We are much faster now. I will give you some numbers on the next slide. The time for shipment for both 4.0 and 3.6 and 5.0 The feed audience is mostly in the time range of a couple of days. So we have three or four days of testing to make sure that we deliver a good product to our feed audience. And then we have for the older grandchildren, the security leaders, about one week until we release to the public. We like normal tests. We also have to run a couple of update tests. So we have to make sure that the order we use for Firefox gets correctly updated to the most recent version. So what we are doing is we are testing the previous version. For example, we have released 3.6.14, so we have to test 3.6.13. We have to test all the releases, so we have different updates. So if we have the most related release, we get a partial update, which is normally only 3.4 megabyte in size. And the complete update really downloads the world, which is depending on the OS 10.0 or OS 30 megabyte. For testing, we have different channels. So what I told before, we have a couple of days until we release to the feed audience. So for that, we have the data test in the data channel. So data test is only for us. So we can make sure that the updates gets correctly delivered to the users. And the data channel finally puts everything out to our data users. And the same happens then for these testing and for the release channel itself. So we are running there with a couple of builds. And in the former time, everything has been done many early, and it was a lot of work for us. Now we have automation in place. And we call it MOSFET. It's a function testing framework you can use. And so we have actually automated about 110 tests out of the 230 tests from our Rhythmus test suite. And it's about 30%. So when we run those tests on the platforms, those are done in seven minutes. And we can run everything parallel and all the support platforms. So we're already done in 10 minutes with everything prepared from one day to one person to run all those tests. It would be amazing. And the remaining 200 tests, we have people who are running those tests manually in the remaining two days. And make sure they have the same process expected. What I've started lately is the execution of localization tests for all localized builds. So we've got a great feedback here. So I will come back with a later for the demonstration so I can explain it to you a bit better. And what is newly done is the fully automated update test. So we can really scrape the FTP server, get all the builds and run everything automated. So we don't have to do anything. We only take out the update test and get the results and report that to the engineering and then the update. And what I've already told, we run all those tests on all platforms in parallel. So who's into it? We already said that it's not so large. So it's a core team. It's only about serving people. So we're working in different areas. That's for mobile services and optimization. And we have a couple of other contractors who help us for leases. So that's early for the smart team and we already need all the automation in place. So we can manage all those many different leases we have at the moment. Additionally, we have a lot of committee members here who run through our manual tests. We have a litmus mostly on a daily basis. I think most of the users are currently nightly users. So of course, like yesterday, we had about 80,000 nightly users for Firefox for all builds. And about 15,000 for all the Firefox branches. And that's really a pretty good number for us. And we want to really get those people to help us here in testing. And that's really something we want to do with the crowdsourcing. So I come over now through the content of this presentation. So we want to get these users to run our tests, to report everything to us that we can do for testing us through Firefox. We have a great community around the world. That's probably the best thing we can have in our company. So we have people in each country around the world who can run our tests and sell localized versions of Firefox and with extensions in store and more than that. So that's really helpful for us. We only have to outreach that we have to the extension right now. So what do we expect from crowdsourcing? What I mentioned before, we want to have a higher quality of localized loads. So we can add more and more tests depending on the feedback from localizers what they want to have tested. And we don't know anything about the system that those tests run on. Only on our side, we have some systems running some data tests. We know the consideration, you know how much memory on the CPU, how long it takes. On the user system, we don't know nothing about that. So that's really a pretty infomation. It can give us really pretty good information. So we want to target this institution. The same what I skipped here for different locations on the globe. So depending on the redirects, you can see you're in France, you're open Google and you can read the French page or you are in Australia. You have the Australian page so that we can also find out if there are differences depending on where you are located on the globe. And that's what I mentioned before already, performance-related information. This can give us good information about the four-year start-up times and all that stuff that we can interact with. So what is necessary for us? In the past, we have shown that installing more than one is quite difficult for the users. So we wanted to have something which makes it really easy to install more than one on the system. And any installation we want to do should not affect the global system of the user. So we only have a test environment located on the system. We install software inside this test environment, but nothing else gets polluted. And that was one requirement that we had here. And all the needed tools which are necessary to run our MOSFET test are in the MOSFET clean environment. We have pick-on-pick-out which is downloaded, is extracted and prepared everything. And that's really everything you really need for that. And then additionally, as more tests we have, as more feedback we can get from our users, localizers, and then outdoors. And that's one reason why we have to add more and more tests. And we're outlining for feedback so we can work on to add more tests to scenarios and to get more feedback. We like that. When users are running our tests, it doesn't tell when they have the results locally on their own box. So it's really important for us to get source results. So we have to make sure that we can report all the results and we can use this for analysation so we can identify the problematic areas. And, as part of the list, we have to promote it even more so more and more people know about this example. And it helps a lot, we have already seen. So how it works, we have implemented it as an extension itself. So it's compatible with Firefox 4.0 and Firefox 3.6. It gives 3.5 because the brand doesn't need it any longer. So we don't have to care about that. And I think most people will have to run it in Firefox 4.0. So that's completely fine. We have automated completely the setup of the environment. So you only have to click one single button so everything gets prepared to run the test. And that's all we have to do here. And the larger part is installing the extension. So you have to resource the browser so we don't have a shit-pick for that. Yes, we'll see what's getting done in the future. For the UI, we have a really simplified UI where we can say which application you want to test, which test run you want to execute. And I will add some more stuff in the future for the moment it's important for us because we can run our tests, we can send the reports and we can analyse those reports. Although we have to make sure that when we run those tests, users are always operating on the latest version of the test because probably it'll be a fixed test on a daily basis. So it gets checked in and sort of depository and users really have to use those. Otherwise it wouldn't make sense for us. And we can this part execute and report to our database. Here is a quick chart how it works itself. So you're running Firefox in your normal daily user profile. You have installed the multiple cloud extension in this profile. One thing we have to take care about is running those tests as I can't pay a lot or whatever else. So we cannot use the user's profile. So we always have to execute our tests in another profile, in another Firefox instance. So all that stuff is only possible in our test environment. So Mosmule is running as a command interface. It's a Python test environment. So from the extension because it's a Python process and it executes Firefox with a testing profile. Mosmule has extension installed. So these two processes can communicate with each other. We get the latest test and script from the repositories executes the test and finally can report to our database. What I mentioned before here are some results that we got from our Elton N tests. This is one test for, I think, the geolocation. So we are testing here, in this specific case, access keys which are used multiple times in the same scope, which is normally not allowed. And at the same time we are also testing elements which are cropped. It can happen that when you translate Firefox to your own language, most of the time the string gets longer and if you don't adjust the width of the window or the test element, it gets cut off. So this good test we have in place at the moment. We have some bugs in there, so most of them we rely now on the access keys. So I don't know if you can read this stuff from the tech. So you get a list of elements which are affected by some multiple usage. It's really hard to read. That's why we have the feature in place that we can create screenshots from the specific cases. So it's easier to see here. All the names here are underlined. These are the access keys, so they are used multiple times. So you really can see these elements. We have to update all the entities and then it's ready to go for the access. And we got a lot of check-ins in the last stage as what you have recorded. So I think we will continue to run all of this against the next beta and standard users. So what's left for us, only some points here, we don't know how reliable our tests are. We run our tests in our system so we know what happens but we don't know how reliable it is in other locations or on other systems. So that's something we have to figure out here and how we can be able to improve our tests in the case. We have big problems with the focus. So the test itself, or Firefox instance, under test has to run in the front, always in the front so the user cannot switch the application. Otherwise any test which relies on text input and the auto power will fail. So that's something we could have a solution. So we are working together with the Selenium guys who have implemented native events and that can help us here. I'm not sure yet if we can implement it in our which way we want to choose here but it's really interesting and truly a big problem we have to solve. What I mentioned before, for now we only allow to run our tests completely. It takes 10 minutes and in the future we have implemented all the tests and it can take about 30 minutes. So we only have to find ways to allow the user to select types of tests or he can select tests if he want to run and he can or to help us in fixing program tests. All the stuff will be included in the future but for now it's really only for the test execution. Important for us here, we have started with an end test which was a great success in the next quarter. We will continue with add-ons. So we will ask add-on authors to create most of the test for the add-ons. So we can run those tests on our side and from the crowd. So that's a great feedback to make sure that the add-on is going correctly. And then something else, our endurance test we have done already, we have started with so we can measure the performance of kind of over time. We are doing the same stuff again, like opening test or open panorama, go panorama all the time for 10 minutes, 15 minutes and we have already seen some great interesting tests out of those tests. And finally we really need feedback from you. Okay, so sure about time Brian. Yeah, five more minutes. Okay, so I have it prepared here already. So this is a link profile. So as you can see, cloud is not installed yet. Obviously it's installed, we restart the cloud one. And it appears in the tools menu. So we have cloud interface here. By default it has been selected to application. So you are currently running and then you have a set of different tests you can run. The general test run, the first one is what we are running for our release test which are one-to-one copy of the litmus test. As an end test, our axis keys and the crop elements and add-ons test, we only have two add-ons here. We are testing only one or two tests which is only for self, no, we will figure out what we can do here. As an end test run, I want to run it with my new tools and the test start. In this case, I have already prepared the test environment. So I have the light on the network here. So normally it takes up to five, ten seconds before everything has been set up. Here is another instance of firehose test started with another profile and in this case we are running through the preferences each sub-element, each sub-window to have a check for multiple axis keys. We haven't run it twice. Now we check for the crop elements. We collect all the information. We create screenshots for those test trailers which appear and we should have been then the information to check in the preferences we send it to the cloud database. So I can... So I should appear here at the top. That is correct. So we run this, we have two parts but also two test trailers. When we click on here, we see that there is a big number of things which we have identified here. These are for crop elements which is not going to be working correctly. That is important here for the axis keys. Those elements, those endless eyes. Screenshots. At the same time, we have saved and there is a screenshot on this so you can... There is another look in here. But you already have it present on your disk so you can work on that. You can fix your problems in your locale. You can run the test again and everything appears correctly. So if you want to get more information, Mossman Cloud can be found on AMO. If you are interested to help us with great tests or have any ideas on how we can improve our testing join us on qualitymuscle.org. Mossman Coach on Shithub, Mossman Test. If you are interested to write some tests and we have even more areas you can help us. Any questions? At the end. You are creating a new profile before running the test. Yeah. Maybe by duplicating the user profile to give you more tests to the user profile. We have some problems with some extensions so for the normal DFT test or general test one, we can't do this in this way but it should be easy enough to set an option where you can edit a checkbox or check it if you want to have an existing profile so we can clone it and can use this profile for testing. It's good. But not at the moment. This is the problem with false positives. Is there any handling of false positives? Not right now. So we don't have any idea the question was how we handled false positives as the test results. So we don't have any idea yet. We have updated the add-on to AMO two weeks ago so it's really fresh on AMO so I'm just waiting for the feedback we haven't gotten so far anything so once we know what we have to do I can give you some help for now. Sorry. One more time. Brian or for one question? We have time for a couple more questions. Okay, excellent. How does multimil decide when to actually restart the browser? We have two different types of tests. We have non-resort tests and we have restart tests. So it's in moments that are two different applications and we call it separately. In the future for Mosmit 2.0 which will be released by end of March we will combine it to one thing with an extension. So we have many test files so we can indicate that tests in this folder are normal tests and this folder contains restart tests. So we are training a fresh profile before we execute the test in a folder and for the restart we are using the same profile again and again. So we only can test the installation of add-ons, installation or theme installation switching of themes I'm asking because for the other end test run the start-up and tear-down is about as long as the actual tests that we run. So we could cut off time by running both the croft test and the exoskeleton test. We have to find out how we can reset the UI. That's the main problem we had because we are story-ins so we somehow have to reset everything if you have an idea I would be up for that. So we can remove it because it's a normal test. So it's there for purpose. So if we can find a way we can be following those tests to a normal croft test both to get an exo-gold trigger. Okay. If no one has questions you can find me around here and talk to you then. Thanks.