 I'm Yang from Chrome Operations Team. So Chrome Option Team is help make Chrome easy to change, hard to break, fast to release. So I'm here next 30 minutes to tell you about the generic testing and Chrome's specific things. So I hope in the next 30 minutes, you can learn something. I know there's lots of information in the last two days, great talks. I hope this is a relax. You'll learn something, and I will give the talk links. And you can follow me, follow up with me, or follow up with the teams. So the agenda would be, what's the test? When do we test? And how do we run tests? And what make a good test? And why should we test? And the last one is that I wrap up, see the test-to-contact maps. So what's our test? So before I start, can you developer and tick the airplane, right? The airplane you built, you tested. Do you have confidence to take that? So from here, the test is important. Test is a part of the product. So what's a test? Next is a test is something we want to ensure products is behaving as we intend. So here we're using Chrome. This is a Chrome University. So a test is something we want to ensure Chrome is behaving as we intend. So the key word is we. We is everybody. It's not like a testing team tested product for you. It's you are the developer. You make sure your features, your product, as behavior as an intent. So next is a test is something we want to ensure Chrome is correct. So this is something like a functional test. So you make sure your products, your features, is correct behavior is correct like what you design. So that could be have a functional testing. It's have an ad hoc testing. It's a unit testing. And this is all like a functional. So next slide I like that is, oh, I forget to mention. This whole slide is based on previous contributors. So this one I like is a performance. So a test is something we want to ensure Chrome is performance. I like this word is performance is a word group of the all performance things like a speed testing. I think speed is my coworkers will present later. And the other thing is like memory, binary size, power. So as a Chrome running on the, let's say running on the mobile. So size is really important though. You cannot build the product is correct, but you're using lots of memory or probably consuming lots of power. So in the mobile user will suffer this kind of pain, even though you have very good products like function well, UI well, you get the things, but consuming power quickly. So we need to make sure Chrome is performing. Next test is something we want to ensure Chrome is secure and stable. Just a half an hour ago, an hour ago, yeah, people just build like the security security related things, right? They did a very good job as a cluster first. So finding the hundreds of bugs every year. So we need to make sure the Chrome is secure and stable. So this is what tests to do. So next, so this is a pyramid. It's coming from Google testing broad. I think I raise a hand, you already see this. This is in lots of area. So this is like a 70, 20, 10 percent. So the bottom is unit test. The middle is integration test. The upper is E2E test. The reality is like lots of, we need to provide lots of unit tests and the last end to end tests. So the reason is that you can see the right side. So the unit test is a fast reliable and the debugable. So unit test is a small, is a modularized, basically is a no dependencies on hardware. And this test is a testing like a function, is a testing like a class method. Integration test is a combination of the like a UI test or some of the special automation tests. E2E test is the most like a manual test or look at the, you compare the two UIs. So in Chrome, we have a two kind of unit tests. So one is a G test. I think a desktop, Chrome desktop is using G test and iOS. And the G unit test is a Java unit test. Mostly it's an Android. Integration test. So you can see the list of them is like a browser test, auto test, it's just a change in name called test. It's a test running on Chrome OS. Ergory is running on iOS. This is some black box UI testing on iOS. Telemetry is kind of performance related. And the instrumentation test is Android scenario test. And the layout test is a blink and check the rendering. So these are all different kind of tests. I don't believe there are somebody know everything. So there's a list in the top is a manual. Oh, sorry, the layout is a little bit tricky. So manual test and the live page test. So think about it. Don't rely on manual test. This manual test is the last things you are doing. So you're supposed to thinking about more like a unit test and the integrated test, more automation. When we do a running test, as soon as possible. So when you're developer, you design, thinking about the test. Thinking about the test code. I have several slides later thinking about how to write in testable code. As a developer, you need to think about test first, design, and then your development. So test is earlier in the developer process. It's better for your code structure. In Chrome, I'm using here the four steps to describe what's the Chromium running like a test when we're running tests. So first is pre-summit. Second is a common queue and then waffle and a branch. So it's a little bit different to the generic. We look at a pre-summit. So in Chromium pre-summit is kind of when you have a C already, when you upload a time. So it's check the configuration, check the format. These are various shots and a simple and a quick. And then common queue is the major Chromium to check the pre-summit. The developer have a code ready and there are a bunch of the tests running. And we won't running everything. We were running like some tests that related to your change. So there's a lots of tests already there in the common queue. So if the common queue like fails, you're not able to check in. So, oh, sorry. So this is the sample from Gary. So you can see when you have a sale upload, so you can see different kind of the tests running. So green means pass. If red means fail, if it fail, you cannot check in the code. So after you submit the code, so that's called waffle. We call continuous integration. So there are some tests, it's unlike a unit test. It's not very stable. For example, performance test. Oh, some test is taking a long time, two or three hours. This is not easy put in the common queue. This is running continuously. So we have this test running. So don't be surprised if your code be reversed because your code is passed some pre-summit, but sometimes it's failed on the CI. It's a continuous around. So when you get this, you're thinking about how to make your code better or probably how to make your test better can catch that earlier. Last one is a branch. So when we finish everything, there's a branch cut. So at this level, there's a manual test team jumping. Look at the integration test. They run in scenarios. There's other testing as well. So next is how do we run automated tests? I want to specify focus on two of the face, right? CQ and CI just mentioned before. So in common CQ is a kind of pre-summit is common queue. CI is kind of like continuous integration test. So here we look at this, right? People probably want to ask why we call this waffle. So I think a waffle is kind of like a UI of the build chromium.org. It's a, you can see the bottom. There's lots of builder configurations. You can see each column. So there are different configurations. So green means these builder parts and the yellow is still running. Fail is red. And this is a build.chromium.org. Everybody can look at this. So when we run this, before we run in this, we have a three step to configure this test. First is a build configuration. So for example, are you going to build like a debug or release build? Which OS is this Windows, Linux or iOS or Android? And architectures that could be like x86 or ARM and library linking is a dynamic or static. So this is the first is a build a configuration. Next is a test configuration. So you need to make sure what kind of a test type. Is that the functional test or the performance test? Or make sure this test suite, what kind of test suite you want to run. And also you can set up like a runtime flag. So you can put a different type of flag to run the test. Last one is an execution environment. So this is the OS version. So different to the first one, OS OS is Windows Android OS. OS versions, you know, Android has lots of OS versions. It could be KitKat, right, L, M and OP, something like that. So Windows 7, we don't stand. Hardware is, do you have a specific hardware, especially GPU? You need to run a test, especially on the GPU. This is the workflow, how the workflow is running, the whole system is running. So I just quickly go through this. It's probably in the backside, it's very, very small. So the machines or the system is like waiting. If a test request comes, so that wake up. And if not, it just keeps sleeping. It's a certain time. I don't know exactly how much seconds we're just waiting for a request. So first, starting with the compiler. So compiler is, we have a distributed compiler system called GOMA. So when we give the task to compiler, GOMA finish and give back all the informations. So we collect those informations, along with all the test configuration we just mentioned and the test flag. And put everything and the test resource into the system called isolate. This is a content addressable storage system. And then give a hashback and this is a trigger to test. So before that, so all the above, if everything fail, so we've got to build a fail. So if this part, so then we're running a test. So then get to the test running and this system will send the user back all the test log with the pass or fail. Then you can check the build system, see what kind of test to fail, why this is failed, all the logs. So this is a typical, I guess, our system. So this is running parallel every day. We have lots of tests running. So one day of testing in Chromium, the data is all the data. It's not like the latest data, but still you can see there are 250k plus unique tests running every one day. And there are 650 million test excursions. So interpreted to the machine time will be like a grade and three years of swarming tasks. Why do we run so many tests on CI? So you may thinking why we're running so many tests. So let's do this. We have a seven plus operating system, Android, Windows, iOS, Mac, Linux, but one. And a four plus architecture. So 32-bit, 64-bit, X86. And a two plus models as a debug, build, or release build. And a three plus operating system, versions, like I mentioned before. Each OS have a different kind of version. So total combination will be 100 plus configurations. And unfortunately it's getting bigger and bigger, right? So like Android, iOS is keep adding new versions in. So every day Chrom developers is put to the cell in and we run it all the tests. Next is what makes a good test. This is an important one. As a developer, writing codes to make the future ready. This is our probably high priority. Hey, let's get it done. But thinking about, hey, we need to make a good test. A good test is consistent. So when we value the test, if the test, the good test like a pass and a fail, something behavior bad is a fail, behavior correct is a pass, right? So there are something like we call flaky, right? This is a nightmare for us. A flaky test is always a test bug. So why is that? So usually flaky test as a, I put the three category. One is it could be product bug. It could be infra issues. It could be test bug, right? So thinking about a sample as a product bug. So flaky test when a product bugs, I believe there are lots of people as a C and a C++ developers, right? So let's assume like if you're using shadow copy instead of deep copy. So sometimes it's very hard to find the issues there. You may keep like its rates and simulate a different situation get there. So as a result, the test is keep pass, pass, pass by. Sometimes fail. So it's very hard to catch that kind of issues. But this is a product bug, but it's a flaky test. So second, it's a infra. Let's say if you're running server client test, you are now using Mokito. You're just using server client. So there are so many dependencies. It's a Wi-Fi, Wi-Fi adapter, network server. So everything could make your test sometimes fail, sometimes pass. It could be like a flaky on the infra. And also poor test is causing a flaky test. So for example, you're adding like notification. You are waiting, some UI happens, and it's not happening. Oh, easy. What? Just add a timeout, like waiting. Now, not coming back. Keep adding timeout. So then the test is a really bad test because sometimes it pass, sometimes it fail. Chromium have the good tools. This is like we check the frakiness. The left side is you look at, this is a browser test, I think. And the right side is a huge different color. So green means pass. So gray means fail, we retry. If we try pass, it's pass. So you can see black, small black part. This is a retry fail. So we call this a test fail, all right. This side, I think this dashboard is going to be deprecated soon. We have a new tools ready and probably in the next couple of months, stay tuned. So the reason is this dashboard is extremely slow to load. So there are lots of information there. The tools team has built a new tool, like Analysis Chromium Orb. So everybody can look at this. And if you check the flaky part, as you can see the dashboard. So in this dashboard is your test cases have a score. So this score is some algorithm to check how flaky it is. I give you one of a sample. This sample is, the tool is finding out. So you can see the top in the very left, there are 100% pass, all of a sudden is a drop. The tested pass rate is dropped. And the tool is finding out, this is because of the CL 563370 causing the following test, like CL is keep failing pass, skipping pass. So then we find a bug. So these are tools find a bug. Hey, starting from this sale, we thinking this sale has some problem causing this flaky. So next is, this is kind of the bug. It's from the tools called find it. So this is a tool like we find out this bug because of several flaky tests. So we find a bug to the owner. The other UI is telling alerts on the Garrett code reviews. So you find it, identify this sale is flaky. So that's the reason you look at the log. And the next sale is, sometimes it will be auto-revert. Don't be surprised, all right? If it's auto-reverse, that's some flaky test. So when you hit that in the future, please. Fix that as soon as possible. Your flaky test may causing other developers lots of trouble and taking lots of time running all different the build, right? So please fix your flaky part test, right? So based on what I'm mentioning, right? Shouldn't test just not be flaky. A good test, right? When we write a good test, it should be fast. Running fast is not only the local, developer running local. Also like running on the waterfall, right? You can see, if you're running your test longer, which means everybody like waiting for you, for example. We are running lots of tests every day. And a good test have high coverage. So everybody know code coverage is important, right? I'm using this way is a good coverage test doesn't mean you have a good test. But the other way around, a good test must have a high code coverage, a good coverage, right? So coverage is, we build the coverage tools. Thanks to the people working on that. So currently in Chrome and we support Linux, C and C++ and the Java Android. So this is a sample, you can take a look. Basically, we put the line coverage, function coverage, region coverage. I think a line and the function is pretty straightforward. And the region is kind of like if else, and some switch case, you look at this detail, what's the coverage data it says. And the other thing, the tool is not just to tell you, hey, how good you are, you can dig into the link and see every code, like a code line. You can see the code is covered or not covered. So this is a pretty convenient for developer knows what kind of codes are not covered, the test. Next is, a test is small. So I would encourage, as your developer thinks, right? Thinking about your code is more like a modularized, right? So that's easy for testing. So every test is the small test will be less dependencies and less flaky. A test is a hermetic. So like I mentioned before, right? If you have a non-hermetic test, like a network client at a server, so what's the best way? So first you need thinking about, hey, my code, the test is supposed to have a Mocha server. So everything put in the one machine and your Mocha server. Then you like reduce some flaky possibility from the network, so. Right testable code. So I give one sample here. There's lots of sample you can find out in the web. And for this sample is a non-deterministic sample. So this is a very simple function. It's just a check, this is even or odd, right? So everybody can read this. So this is very hard to test. Why? It's because that's a line as a random red, you call it your random. So how can you test? Test means you give an argument, you see the pass and the fail, right? So supposed to be like this, right? So this is like a function. You put the arguments there. So there you can have a couple of tests, like give the number is odd number or give the number is even number. So backside to look at this. This is really hard. So I give a sample is, hey, when you write a code, you're supposed to write this kind of code. Or you probably have another API to calling this even or odd. So that's easy to write test. And also the other thing is that there are some code that is using money like a system timeout. So please try to avoid that. You can use it in the other ways. If you're using most like a lot of timeout, it's very hard to test. Why should we test? A test is an investment. So test is not just the, oh, I write easily. It could write some infrastructure. You could write a test framework to support your idea to write in the test to qualify a code. And the test is a part of your future. Don't undervalue the test. So as a developer, test is a part of your features. And the other thing is, I would say the test is a part of your future's future. What if you extend your future? What if you change your future? If you have a good test, it will be easy for your future development. So we're almost done. What if I don't know how to test? So first, thanks, Lele. Check this document. So this is a public testing in Chromium.md. You can look at how to run a test in Chromium. Next, look around the code base. I know you are all good at looking at all the other people's testing cases. Next is ask someone who had been around a while. So ask me. Oh, by the way, I'm just one year old in Chromium. So you can ask your peers. And the next is ask Chromium and your protein. And your protein, we will be very happy to support you how to write a test. If not, we can point it to people find the good mental some sample to you. Last is the test contact map. I'm just wrapping up what I mentioned last 20 minutes. So I look at the test several dimensions. So first is what? Where are you going to running? It could be functional or non-functional. So I'm just using the functional, non-functional. So I don't have a big room to extend all the points. So next is where to running. It could be a virtual machine or running a mobile or desktop. I'm just using three samples. It could be a lot of samples. When we're running, it could be pre-summit running or post-summit or some integration or regression. How? It could be automation, semi-automation, and the manual. So anytime when your partner or whatever your clients, your peers ask you, hey, can you run your test? You need thinking about not just the, oh, I have a test. You need thinking about the combination of this, the point. So give a sample point. This is automated security test run on mobile for post-summit. So there are lots of dimensions you can thinking about. I just put the four dimensions here. So anytime you're running a test, you need thinking about, hey, is this test, oh, is this code work behavior intended on the GPU specific hardware?