 Morning, everyone. So as has been announced, it might be very boring if you already committed some code to Chromium or Chrome OS. If you actually did commit Chrome, I mean, your own code. Can you raise your hand just so I get an idea? OK, great. So I can go fast. Obviously, you can interrupt me. And there will be Q&A time at the end. I try to give a bit more because there are some Dory submissions, too. There are, well, set to this goal. So I work on Lucy. You don't know what it is, and it's fine. It lives at the very base of Chromium. It's kind of layered. Most of the developer tooling is built on top of Lucy. And actually, Chrome itself is not really the only project which runs on top of my stuff. There are a few others, and these are just public ones, because this talk is public. I can only tell you public stuff. So please don't ask me questions which are internal, like how many machines do we use for tests. You can do so after the talk when we are being recorded. All right. So this talk is focused, as has been announced, on Chromium, especially, but most of the things are applicable to other projects. For example, Chromium has Fuchsia. I don't know how many of you heard of that. They also use similar infrastructure. Chromium is transitioning to use more of what I discuss. So I hope eventually it will be mostly the same. So still, the way you contribute with the different projects is very specific to those projects. Chromium itself depends on a bunch of other sub-projects, like V8 and so on. If you want to commit code to those projects, you have to really look at the docs. So for example, here, if you look at the docs for Chromium, specifically, this is the Chromium docs, which says how to contribute. I presume most of you have seen this, or at least did something similar. So typically, this shouldn't be new to you. The way a bunch of code makes it to the Git log in the end is with these eight steps, assuming eight isn't happening. So you don't have to read them all. I'll go through them one by one. So first of all, you need to know what you work on, right? Typically, it will be either a bug or you work in a feature, in which case, we strongly encourage you to file a bug, because usually it's not just a single CL, and you want to group them by bug. There are usually started bugs. If there are very new people here, I think that 800 now, they keep growing every year. I recommend to look at the latest ones. They're more likely to be up to date. But anyway, suppose you pick some bug, which looks like this. There's some struct trace over there, which means you're lucky. But the struct face is nice. That said, reading the whole code is better. Seeing just the line number isn't good enough. You want to see more lines around it. We have awesome tool to code search. Raise your hand if you've heard about it. Oh, that's great. So one focus much here. You can search here. Now, we have two code searches. Actually, we have public one, which is what my team maintains. And that's what is available to all the contributors to Chrome who are not Googlers. We also have internal code search that I hope you've heard about. If you work with internal Chrome code, you can use that to search it. However, unfortunately, not all repos are available there. So it is possible something is missing. All right, with that, let's go one extra bit. I don't know why, but many people are not aware that code search isn't just showing code. It also shows you history and other metadata, like who changed those lines. It's really useful if you want to figure out whom to ask questions about the code, particularly if you're new. If you figure out, you need to modify some file for your work. But that is outside the expertise of your team. That's how you find who the expert is, apart from owner files. Now, once you found a bug that you want to fix, don't forget to assign yourself there. It's a common problem. Don't work unless you assign it to yourself. So you don't waste work with somebody else from your same team. All right. So how many people have to check out Chromium code? All right, great. So you're familiar with how long it takes, right? So let's skip this. This is useful for many people who haven't done it yet. I hope everybody has deeper tools. Without it, it's a bit hard, very hard. So you probably set up your Git cookies. So that's great. Now, I mentioned here that there are many different ways to check out the source code. These things actually keep evolving. We have historically been on G-Client. So probably all of you have heard of G-Client. But Chromium still uses repo, which is Android original tool. And I think some of our projects are migrating to vanilla Git with submodules. So I hope, maybe in three years, you will mostly see Git submodules, and you won't have to know what G-Client is. But that's probably a silly wish. With that, you have to wait. So every time you wait to code check out, I have a recommendation for you, even if you're on a corp network. Invest into learning Git and tooling, particularly if you're new to this. You use Git and Git-Cl stuff every day, probably a couple of times. Learn how to rebase stuff, really understand what's going on during rebase. Raise your hand if you have to rebase your code. Good. Do you understand what's going on doing rebase while, particularly if you have multiple branches? If not, I recommend these two links. I think they're really helpful. I've seen people improving their productivity because they can have stack changes in their local environment, which allows you to work on a new CL while previously I was in review. That allows you, essentially, to pipeline your work, just like CPU does it. So if you don't know this, basically if you remember anything from my talk, I hope would be this, that you look at the docs and improve your understanding of the tooling. So you yourself get more productive. Please slow latency for our typical operations. All right, let's go next. No flame wars in editors. Choose whatever you want. But I strongly recommend you use Git New Branch if you work in Chromium. You may use Git Checkout-B, probably. If you do so, it's fine, but you may hit some problems when you use multiple branches. So I recommend the switch to this. How many of you have Git Config file in their home OK, for those who don't, I recommend you look at somebody's senior on your team and ask what they have inside. Usually they have some shortcuts that save their time. I have probably 20 inside. Admittedly, I'm an infrastructure person. I wrote many of those tools. I think normally it's five or six. It's just shortcuts for commands you use. So you don't have to type by hand Git New Branch. There should be a shortcut and so on. So use your shells and tooling to minimize your latency so you can focus and work. All right, let's go next. This is not fun. You just fix a bug. I don't tell you how to do that. I actually have no idea. I don't work in Chromium. I work in different source code. So it's up to you how you fix it. But obviously, we have multiple different languages in Chrome, so it really varies. There are also different ways to run tests. Some tests you actually have to compile to just Ninja, probably. Some tests you just run some weird local shell script. Well, you have all of that. In this case, it was, I think, some Python test. But usually, if you're lucky, you can just run tests locally. And it gives you some kind of feedback. Sometimes you need to run tests on other platforms that you don't have under your desktop, something like iPhone. And you maybe have on the Linux and Windows. So in this case, you probably use SecureDry Run. If you use SecureDry Run, raise your hand. OK, that's great. I'm happy because I added it. Let me do work. So before you can use that, you have to commit to your CL. You have to make a local commit. Most of you did that. However, please remember to attach a bug. Some projects actually require a bug reference. I don't think Chromium does. But please attach it. One monotone bug numbers. As mentioned before, we have lots of different projects. Chromium is not the only one. Admittedly, it was the one in the first place, so it gets the numbers without prefix. But if you work with people outside of Chrome, I recommend you add a Chromium prefix in front. The left and right are equivalent. And you can have multiple bug lines if you have a lot of them. All right. So when the upload is changed, the lower one is for Chromium. Actually, if you work in their repos, sometimes you can also use Git CL upload, the same way you're familiar with, I presume. But you can also use repo upload. You can also use vanilla Git if you understand what you're doing. Actually, can you raise your hand if you use Git push reps, foreheads, master, and so on? Actually, more than I expected. So basically, that stuff also works. I just use it a lot. Git CL upload and repo upload simply provide you more goodies around it. But you can also use vanilla Git push with Git notation. All right. I'm sure all of you have seen this. If you upload a code, there's a giant blob of code. Someday, we'll improve it. But for now, we have to scan, train your machine learning in here, and identify a URL so you get your URL after you upload it. I'm sure all of you have seen Garrett, even if you haven't landed any code. But one thing I would note here, this thing's called TryJobs. I'm hoping next year, the talk will be very different and it will not show this screen because it will be more test-based. But for now, we are build-based, so you get to see all the different builds that we run in CQ. How many of you had to debug a red build? All right, good. So let's skip this, slide, somewhat. So if you haven't read one, typically, you'll end up something like this. We try to improve this UI, but I think it's still pretty bad. To build Chromium, I think it takes about 170 steps for some builders. So you might need to scroll up and down on this page. We added this small selection. Hopefully it will be somewhat more useful. But if you haven't seen GNRX in the output, I recommend you look at it now. It helps you reproduce. Well, if you have the similar machine, the same failure locally. Different projects like V8, and I think WebRTC and a few others, they also started emitting GNRX the same way. So if you see a failure, at least you can hopefully locally build the same thing. But that's the way for the repro. Sometimes it's difficult to repro and sometimes the failure is in some weird machine that you don't have locally. So you can't reproduce it locally. In this case, and this is really only for Googlers, sorry, I went too fast. We have links to so-called shards. So you're probably aware that we don't run all tasks in the same machine. So we build them by machine and then we run tests in multiple different part of machines. We call them shards. When shards fail, we insert to this URLs. I admit they're kind of hidden, but I hope now you know. If you click on them, you get a new interface with the debug button. Now, since this is recorded and it might be used for public people, outside of Google, you will not get to this page. That's only for Googlers. But Googlers can click on this and essentially you get, sorry, you get a machine for yourself. And you can access the machine, I think even Windows now, and you can run tests on it. This is kind of extreme case. I tried to avoid it, but sometimes it's inevitable. Actually, if you don't mind, can you raise your hand if you use this thing before? Okay, only two hands. Well, partially it's great. We hope you don't have to use it, but sometimes you have to. So keep it in mind, this is your possibility if there's some weird spec or some GPU that you don't have. This is useful. All right, speaking of failed tests, sometimes tests fails, you have no idea why. You didn't even touch related code. Also known as flaky tests. We have some tooling that I showed here. Anybody seen this before? Oh, I'm sorry. Really sorry. Well, we're hoping to improve it, but this is so-called flakiness dashboard. For each test, each row is a test and it kind of shows how tests behaved previously. So if you see like a lot of green and reds that basically means test is flaky. I'll talk about flakiness a bit later. So I assume it was your bug. So you fixed up all the new patch set. That's easy. Eventually, hopefully you get all GTM. I don't discuss this since it appears everybody knows what the GTM is. One thing to be aware of in Chromium, we use code review plus one. This is like actual GTM. Other great projects use different scoring system. So for example, Chromius is common to use code review plus one as in, it's okay, but somebody else has the GTM. We don't have this option in Chromium, the Chromius has and there's some other projects which use even more viewer or schemes. So just be aware of that. Hope someday we'll improve so it's more intuitive. All right, submit the CQ. I'm sure those who committed code have seen this button. It's essentially the same thing as before except now you test and lend. Sorry, if you think your CL is really great and you think it may lend as is, you may use auto submit. How many of you used auto submit? Okay, then I'll talk more about it. So we have a label called to submit. If you vote in this label, what it means is whenever a reviewer comes along, suppose you are based in Europe or Asia and you expect Mountain to review your CL and you think your CL is like maybe a couple liners. So you hope to have been like to lend it immediately as is without no changes. So what you're basically hoping is a reviewer would do LGTM and also CQ it for you. It has upsides and downsides, but that's what this button does. It mostly works. It's in Jalice Camp and I don't understand how it works. So it's too hard. So let's go next. Once you're lucky and CQ lends it and I know it may take a long time, but for those of you who haven't touched Chrome OS, think of it this way, you wait an hour, two hours in Chromium, they wait about a day in Chrome OS if they're lucky. So your life isn't so bad, but I know it's a horrible excuse. We're actually trying to make Chrome OS better so the bar isn't as low. So once the CL lends, you get these weird hashes. How many of you know what the CL commit position is? That's great. How many of you actually use the commit position like the integer number? Okay, that's great. I hope we can get rid of it at some point. So we actually, from SVN days, so I have this T-shirt. I got it for migrating Chromium from, I'm not the only one, just to be clear. We migrate the Chromium from SVN to Git. And SVN has these numbers which keep on increasing. So if you have revision 22 and 23, you know kind of how they stack up. Git has hashes, it's hard to compare. So we preserve the numbers. We have very weird contraption, which is anti-Git from my perspective, but it's still used in releases and so on. But I'm happy that nobody uses it. So almost nobody uses it. So let's go next. This is Git holes. This is how it looks once you CL lends. And at this point, many people kind of stop because CL lended, what else do you have to do? I hope you don't behave the same way, because what mostly happens, your CL gets rewarded at some point. And the reason is we also test your CL after it lends. I hope it's not a surprise. One thing to note here. So first of all, how many of you have seen this Chromium main console, this giant blob of stuff? Okay, thank you. So each dot here is essentially a configuration at the top phase. You may wonder what's the configuration. It's basically like a combination of things, which affect how we run tests and where we run tests, like which platform or Windows and so on. So actually we have, I think, way more than a hundred and many more internal ones, which run outside, like not visible for the public, so I'm not presenting them today. But the interesting thing here is a hundred plus is a lot more than what you see here when you run it in CQ. That creates often bug requests coming to my team saying, hey, my stuff got rewarded because it failed in this weird configuration. Why don't you test it before I lend my CL? And the answer is we just don't have capacity to test all hundreds of configurations. So there is a careful selection process for what tests, what configuration gets to run on CQ. It is possible that you would notice something that really should run on CQ because it frequently fails. In this case, you should escalate to the, to the infrastructure teams and tell them that you think it should run every time as opposed to only post commit. All right, this is Sharifmatic. I presume most of your Googlers here, that means you're probably Sharifing or will be Sharifing at some point, which essentially means keeping the tree green or in other words, keeping the head of the tree, of the chromium repository good enough so developers can locally build and test the code. That's what green tree means. Sharifmatic kind of tries to surface some of this information automatically for you. So life for Sharif is a bit, I hope a little bit easier than five years ago. It used to be just merely watching all those red and green rectangles and acting on them. But we have motivation now. For example, if you land a new test and we detected it's flaky, like say after 10 runs, after you landed your CL, it succeeded and failed five times each. We would try to over-revert the CL for you. Well, maybe you don't like it, but we will do so automatically. So this is the example of a commit which got reverted for this reason. And it's, you know, I presume you can't read it, but it shows you the reason and links to how you can see why exactly we thought it was flaky. We have other automation like code coverage. I think that's like last year, fairly new thing. We try to make it available on more platforms. I think for now it's still in the Linux and it's a bit slow, but please use that and add tests which cover more uncovered areas. It should also help us to release this later on.