 Okay. So, hi everyone. So, thanks for staying so late for the talk. Yeah. My name is Leandro, and I work at R as a DevOps engineer. One of the main things we do is basically support the compiler teams and two chains in general on building CI testing and benchmark with actual hardware. I'm also a contributor for LNT, which is a performance tracking tool for the LLVM. The idea for today is to give you one more tool for your toolbox when investigating issues with the compiler or the tool chain in general. So, basically, when we are investigating some issue or the tool chain is playing up, clung is crashing or something or any tool is crashing. One key information when investigating it, if you have the repositories or you are a contributor to that, you are really interested in finding which commit cause that changing behavior. So, that information, even if you are not an expert on that part of the tool, you can understand a little bit more about what's happening. Then, the process in which you can use like a methodology to do that, and one of the ways you can do is try to investigate like the previous versions and try to find out this exact version in which the thing was working and then stop working. One problem with that is that in projects like LLVM clung, in which we can have more than 100 commits a day, it's like very time consuming. For that reason, many tools, as it's a very routine and repetitive process, there are many tools that can help us to achieve this. A little bit of theory then. So, the code by section is this process of looking backwards on a given repository trying to find a specific changing behavior, can be crashes, can be performance regressions, and you might as well be looking for when something was fixed. So, it was not working and now it is working, and I'm interested to know who fixed it, or if the fix was a side effect of another change. Then, in order to run bisect or any bisecting process, you need basically two things. The first one is to have a repository in which you have a relationship of the relationship with changes. So, there was this original change, then there was this subsequent change, and then this one and this one on a timeline fashion. Also, we need some test case or some way to label these given versions as this is a bad version or this is a good version. So, as you have changes happening over time, you basically are interested to see so on the latest one. So, basically something that's not working, then you go backwards and you go backwards until you find somebody or some version that in which the thing was working, then you can restrict your search space to that region and then you rerun the process until you will find the very specific version in which you have that thing working. Usually, when talking about source code, we are looking for a specific commit and commit is tightly connected to source control. So, it's natural that source control offers some bisecting. Then, GIT, SVN, Mercurio, and others, they offer a bisecting option. It's good because you have all the information that you are looking for on that repository so you can run bisection and look for very specific changes. So, the source control, it doesn't know what is the expected behavior for that code is just a repository. But, this gives you full flexibility to build that code if needed in a very customized way to fit your needs. On the flip side, you need to rebuild it and you need to tell it. So, this is the specific way I want to build and this can make things a little bit difficult when you want to share this information with other people because you have all your local environment and the influences of your local environment when doing this. Also, not all versions are able to build. So, if you look for some, let's say out of a hundred changes on the compiler, you will have some of these versions that are not possible to build because they are broken versions. Then, as I said, you need to set up the whole thing, you need to build it, you need to run your things. If you want to have a clean build each time, you need to set it up and clean up things as you need. In big projects, it can be very time consuming. When you have a big code base that takes a little bit more than a few minutes to build and a lot of changes. So, your search space is actually very big. Then, as we have a big code base and lots of commits, there was a need to have a specialized tool in order to run bisection on Klang and LRVM. This was created three years ago and this tool is LRVM lab bisect. So, using this tool, you have a specific environments to bisect bugs or regressions in general on LRVM and Klang code base. So, as I said, so this tool was contributed by Chris Matthews and Daniel. Three years ago, they are both at Apple. This tool is written in Python and is specialized tool to bisect Klang and LRVM. There is documentation on GitHub. You can find documentation is very comprehensive. In order to install it is a Python package. I recommend you use a virtual environment. Then, it's basically those steps in blue. So, you clone, you install, and you have it. So, I didn't say it before, but I'm running a few demos during the talk and my demos are running on a Raspberry Pi. So, we can actually run this tool. This tool is completely supported by any Python two or three environment that you may have. So, let me find the first thing. Yeah, this one. So, I already installed this on a virtual environment following that steps. So, I have one virtual environment is a directory B. So, I will source that thing, then activate and then I will have LRVM lab, and then I will have the whole tool already running. So, as I said, this is my Raspberry Pi, it's an RV7 running Linux and that's as simple as that. So, yeah. So, installation. Basic usage for LRVM lab, you need basically one thing. You need one test case to reproduce the failure you are investigating. So, when you call this, you will call LRVM bisect and then you will provide the test case. I will show you how to do that. But when you run any test case, so no matter what test case you are running or options that you provide, you will basically do four things. One, it will download builds from build cache. I'll tell you more about that in a moment. It will run, it will create a sandbox with that version. It will run the test case and then it will navigate through the repository doing that process that I showed before with the timeline labeling versions and finding the specific commit we want. A few concepts that are involved. The first one and one of the most important for the sake of LRVM lab bisect and is the main argument for using that is saving time by using it. So, the build cache is a cache full of pre-built packages with the toolchain. So, basically CI systems or continuous integration systems are Jenkins and Buildbot generate these packages. There are many types of packages that you can use out of the box like x86 packages on v7 and AR64. They are stored on Google Cloud Storage, so you can download them. Then, this is something that we did recently. We added support for on v7 and AR64 like three months ago, and then these packages are being populated very often. So, I will show you how we do that. So, for every commit that you do that goes to the main repository, we will have a Buildbot server. So, the upstream Buildbot will keep pulling the SVN repository two minutes. Then, if it sees some changes on either Clang, LRVM libraries and a few repositories, we will trigger builds on native ARM builders. So, there are other x86 builders that are not maintained by us. These are the ones that we as infrastructure team at ARM maintain. So, there is one native AR32 and one AR64 builder that will upload the binaries and packages for Google Cloud Storage that will be available. So, as I said, this works very often. So, it's two minutes, it pulls the repository, and it's important for us to have these packages being created quite quickly. So, we got two very beefy AR machines or ARM machines, and then we can complete this process and have, from a given commit to the build cache, you have them in 16 minutes more or less. This is the Buildbot job that is live and upstream. So, you have also the configurations for what this job does on Zorg, which is a repository for infrastructure code and builds and automation within LRVM. Then, you have all these builds, you'll see many of these builds every day, to have as many versions as possible to narrow down the resolution of the results of the bisection process. So, you have very specific commits as much as possible. So, when running bisection processes using LRVM lab, you need to pick a repository in which the tool we will use to download builds. So, there are many. We are, if you don't see any repository, it will use one x86 repository and download things to run locally. There are two of them that we are interested in this talk, which are these two ones. So, Clang AR64 and RMB7, which are the ones that we are publishing things. In order to pick a specific repository, we want to tell an option dash B and point to the specific repository with these ones. I will show you that. Are those the old ones that we were doing before? Probably, yes. Yes, so we only have rights to publish things, that's why we didn't remove anything or we didn't criticize anything. But we say LLVM lab LS, so it's pretty fast and you can have the production ones. There are no restrictions if you install the command line tools, you will have access to the whole thing. So, there is nothing difficult or that you need to ask permission for. Then there is the other important thing is the sandbox. So, the sandbox is the temporary location in which for every time you download a given version, you have this directory with the whole tool chain. So, you'll be able to just with the time that you need to download and extract this package, that's the time you need setting up your test environment. This is important to say, but if all sandboxes are deleted after the end of the process, but you can save them by providing dash S and then providing a directory. By default, they go to a slash temp and they get deleted. Which sometimes if you are trying out your bug reproducer, it can be like a waste of time if it takes like 10 minutes to run the whole thing, you can save time by providing a custom sandbox directory. The other important bit that I will go quickly is the predicates. So, these are basically the command lines that you run to reproduce the behavior you are looking for. So, you can buy the simplest usage, you have LLVM lab bisect and then you'll say, this line fails, find me where this was working before. If any, right? So, you can have a very simple command line that will trigger different versions and navigate through this repository and then it will give you the revisions that you are looking for. So, as you can see, there is something that's quite different here, which is this bit, the percent path string. And this is basically, it will be substituted by or replaced by any version that you are trying out. So, if you get that specific version path, then it will run the clang from that sandbox. So, it will get replaced by the tool so you can test many versions. And also, there's not only the variable path, so there are many other variables that you can use to implement your test cases. You can also use all your previous knowledge from shell and pipes and outputting to files, graphing and all your previous knowledge can be used to help you. And this is very useful when you are trying to troubleshoot something. So, you can use it in two ways. One is via the command line that you use the Python printf syntax, which is that one. You can also have it run as a shell script and in that way, when calling the shell script, I will show it, you will get these variables injected. So, these for the whole list that I showed before. So, this is how you call it on command lines, just inline or you can say run this shell script and the shell script will have the variables in that format. So, that variables are when you are running the test, you'll get those variables. After you finish the test, you can also use test filters. So, you can see for that specific test case, whether the test passed or failed, you'll get a result on a variable called result. And you can also have the time measurements as in user time, system time, and wall time. So, this is the way you can use it. So, you can say for that given test case, if it passed and user time was less than half a second, then you can use that expression to evaluate whether this test case was passing or not. This is useful when you are running benchmarks. So, if you are trying to see why a given benchmark progressed or something, you can compile that benchmark with that version of the tool chain and then run it. So, you can just do the command, semi-colon, run it, and then you will get these results as an output. There are other useful command lines if you like. So, there is variable reuse sandbox, which saves time by keeping the sandboxes, the temporary sandboxes, so you can reuse them. So, this is something that we are upstreaming as part of the preparation of this talk. So, I was trying so many bisections and they were taking long, so I externalized this option as part of this. So, you can restrict versions, so you can make the bisection process to run faster. So, I have a couple of demonstrations. I think I'll show one of them, and then you have these slides and you can reproduce the other. So, the first one is an actual bug open on Clang. So, Clang crashes when calling a function while both omitting a parameter and misspelling a parameter. So, I can show you that one. So, control BW. Demo one, cut test.c. So, when you provide this, you have the parameter as is unused and another thing which is non-existent, toolchain will crash. Then, the way to reproduce it, let me show you, cut stardust.sh. So, this is basically the way I am running it. Oh, I have this command line in a better format. So, this is the way I'm running this bisection process. So, I'm calling LVM bisect. I'm calling reuse sandbox. So, it won't download things and extract things on my local machine. It's just to speed up the process for demonstration purposes. I set a given max revision that I have previously downloaded, custom sandbox, and this is the reproducer. So, I'm calling shell, work link is this thing. I got all this information on the bug report. And then, what I want to find to say not a given version is working, I want to get an undeclared identifier rather than a crash. So, let me show you this thing. So, this is the bug report. So, what it does, it basically will crash the whole tool chain. And what we are looking for is a undeclared identifier. So, I will just run this thing. And you will see that it will start, well, it's not very useful, but you will see many fails. And you will see a few passes. So, it is limiting the search space. So, it is failing, failing, failing until it passes. Then, okay, it rebrands the process on that space as well. So, in the end, it will say that client revision, this number, this timestamp is the first working build. And this one, this timestamp is the next failing build. So, it was working in this revision and it was not working anymore on this revision. So, that's the output you get for anything that you want to troubleshoot. So. It should be the last working build yet. Yeah. Yeah, not the first working build. But that's just a message. Oh, okay, yeah, but I think the first working build it's seen from latest to oldest. So, it's the first working build in the opposite direction. And not the last one from the source, I guess. That's the meaning of the message. So, the version was three less. Well, yeah, but this is because it pulls the repository each two minutes so we can get a smaller resolution than one single commit. It pulls from the wrong side. It's two of the last, yeah, okay. Yeah, okay. So, yeah, I just need one more minute just to a few discussions. So, there is just to summarize the gains that we will have on time spent on bisection. If we, that demonstration I did, I have already cached the download the versions and I have already extracted that versions. So, it runs very, really artificially quick because I have everything pre-cached. In real world on Raspberry Pi, it will take around three minutes to uncompress that's the most time spent on that thing. I have a few suggestions on how to fix that in the future, but still, three minutes is not so bad. And if we do that in real world, we will have one hour and 10 minutes to test 23 tool chains times three minutes each, more or less. But on the other side, comparing this with the time we usually spend on creating the build cache items, which is more or less 10 minutes. We will, to do the same process, we will have almost four hours to run the whole thing considering we get that specific versions that we know that are possible to build and we run the same like jumping through versions to optimize the process. So, this is basically two hours and 40 minutes that you get more or less in this comparison. Also, that we are always having versions that we'll build and we'll work. Yes. I guess more than 24 hours. Oh, yeah. Yeah, that's probably more. I'm just not considering that we are building on a Raspberry Pi because that would be like crazy. Yeah. So, yeah, so that's the thing. We are building on a very high end machine. I will tell you a little bit about that machine. And then, still, if we compare Raspberry Pi versus Raspberry Pi, then it will be like days versus hours, right, to have all those versions. And I'm talking about 10 minutes because we are building from scratch. This is not an incremental build. So, this is a very high end machine. So, this is the demonstration number two. I'll just skip that. So, a final remark. So, automated bisecting is a valuable tool to have on your toolbox to troubleshoot things on the tool chain. We can save time by using LRVM lab bisect as we just saw and recently added on the 7NAR64. So, there is also this move within the LRVM community to move from SVN to Git. And then, these will impact tools like that that rely on revision versions. So, I've sent a message to Chris Matthews a few weeks ago and we are discussing about that, getting ideas and... With the monorep, it would be a huge deal if he was a multi repo, whatever that was called before. Yeah. Yeah, monorep is actually probably simpler because you can use Git bisect. Yes, yeah, and also we don't really need to track the revision number versions to trust that in this scenario. We can just get the continuous integration to build ID or something that is shared and known. So, these machines that we've got for the LRVM project, these machines are actually free and each and every open source project can request some of them. This is an initiative called Works On Arm, which offers machines or remote machines that you can use. So, very powerful machines to try your continuous integration jobs and see they're working and supported on Arm in general. This is the URL. You can request resources, opening basically a GitHub issue. All instructions are on that website. And we use it and these machines are very reliable. We have them running for months without any problem. Yeah. So, we use them two machines. One is a Thunder X2 and one is a Thunder X1. And the other one is another high-end machine. High-sixty-sixty. Yeah, this one, yeah. Yeah, and that's it for me. Thank you. And if you have any questions, I'm happy to answer them. Usually what I have to do is I have a test case so I'm guessing with this it looks like you need to manually extract the test case out of the end of the test case and put it in a shell script and then pass it to the... But it has the run line, right? It has the run line. I think you can. The way I guess I would do it is that when LLVM lit runs, if you have it set to the external shell mode, it actually spits out a Vash script and then you would just need to change the hard code path to what? Yeah. So the question is about the test cases in which you have tools like FileCheck and not those things, right? Yeah, I didn't test with any of them. But yeah, I think you can find a way. If you have a Vash script out of it, you can reproduce the thing very quickly. Yeah. And then you probably just need to change the places in which you need Clang to have that path variable on it. And that's it. That's Clang. Yeah? So one possible feature enhancement here is that I guess if you want to, at the moment, do x86, an emulator where you kind of say run this test, but run it under an emulator and then you could affect anyone from the x86 machine could debug an ARM program. Yeah. So the first thing, on this way, so from the ARM machine, from the x86 machine run something on a QMU, this is kind of already supported because you can have a Vash script. And if you can do that on a Vash script, you can do that on this. Probably. Yeah. And there is also a curiosity on this, is that we build also the back end for x86 so you can use, on your ARM machine, you can generate the x86 binars and even if you have a way to run them, you can do that as well. Yeah? A lot of time. A lot of time. Thank you again for your time. Yeah. Thanks very much.