 Okay, who wants to use the clink-set analyzer? Okay, excellent. So let's start it. Let's start with some motivation. Why do we need to validate software, right? Why do we need to test, to simulate, to analyze it anyway? And the answer is simple, right? We have bugs, and we want to get rid of bugs, unless in those rare situations they become features, we want to get rid of bugs, right? This is a really nice image. I'm not really sure it's real or not, but it's supposed to be the first bug report ever created by Grace Hopper. So that's the bug that was in the first computer. And we still have bugs, right? Since the first computer, we still have bugs. So let me just present you some bugs. They're around for a long time, and sometimes we just learn how to live with them, and sometimes it takes a while just to find how to fix it. So there's an 11-year-old in Firefox, a four-year-old bug in Wine, a 70-year-old in ReactOS. And so the question is, these bugs are there. They are really hard to spot. So we need tools to actually find them, right? So automatic bug-finding tools come to the rescue, kind of, not really yet, but we have a lot of good results recently. So this one is from 14. It's a tool called CPP Check that found a bug that was 13 years old in Xorg. So tools are improving, and we want them to find bugs and actually report them in a nice way so we can fix them, right? So what's the difference between doing some kind of analysis and testing? Testing, let's suppose we have a state space of the program over there. So testing usually checks one path, right? We're going to give some inputs, some concrete inputs. We're going to run the tool, and usually it checks one path. It might check some other paths. We have run-down in branches, et cetera. It may miss errors, but it's fast, so we can do it a lot of times, a lot of tests. And we either get an okay, okay, there's no bug here in this path you checked, or there is an error. On the other hand, when we are talking about static analysis, we have a specification. For instance, we have it at LCL, and we actually check in the whole state space of the program. And after the analysis, we have either, in this case, model checking is one kind of static analysis, so we have either okay, there's no bug, or there's a bug, and here's a trace. So it actually shows the path where, how the error was triggered. It's nice because it explores every single path in your program, but it doesn't scale, because, well, we explore everything. So these are the advantages. This advantage is one is fast. It doesn't explore all the paths. One is low and explore everything. So this, we have to come up with ways to try to avoid this explosion, right? This path explosion, exploring all the paths. So there are two main techniques there, and how to explore the state space, the BMC, and symbolic execution. I'm going to focus in symbolic execution, because it's how the Clang Stack Analyzer works. So the symbolic execution, we have a state space, right? And then we explore the paths individually, trying, looking for proper violations. So let's say we have one path there, and then, and look for the third node from the end to the back. It's a, sorry, the fourth one, the fifth one. So it's a branch there. So what it does is follow this path, then go back to that branch, and follow the other path, and you go back to the previous one, follows one path, and so it keeps doing that. And it does it to all paths. It will explore all the state space, maybe in a few cases. But we have some small queries, right? Instead of analyzing everything as a whole, we analyze everything in small chunks. So this will make the analysis a little bit faster. And this is how the Clang Stack Analyzer works. So the Clang Stack Analyzer, it's built on top of Clang. It supports all the language that Clang supports. It performs this context sensitive in the procedure analysis. And this is something that we have to stress, is just look at translation unit, one translation unit per time, which might trigger some false positives bugs. So you have a function defined in another file. It might report some false positive bugs, something that is being worked on by another groups. And it offers a wide range of checkers. So you have even for pattern matching checkers, so using of unsafe as they are called, a CPI for instance, and path sensitive checks, which are the checkers that are worked in this project. So it looks for the paths and see if one proper violation is triggered in that path. One more point that the reference is one example I'm going to show you in a bit. So the idea is we're going to sacrifice precision for speech. The Clang Stack Analyzer, the idea behind this, it's going to analyze your project. It might report some wrong results, some false bugs. It might miss some bugs, but will give you something. So that's a more industrial aspect of the Stack Analyzer. We want to provide something, right? It might not be the best results, but it's something that the developer can have a look and understand. So let's start with the next example here. So it's a very simple function. We have a pointer to zero. We have a branch and at the reference, otherwise it returns zero. Is this program safe? Surprisingly enough, this program is safe because this is never true. So we are just selecting the first bit of A and selecting the first thing. So it's always false. This is always false. It's fragile. I agree with you guys. It's something that you don't want to have in your program, but it's safe. So this is a very small example that Clang, when analyzing it, Clang doesn't support bitwise operations. So it just assumed that this condition might be true. And then it repossed the bug. So if you run Clang, then the second one is that you have the reference no pointer in line for column 12. And it also generates this nice HTML report field. So if you have a look there, you have the steps that it took to trigger the no pointer, the reference. So it started with that, then assuming the first part of the condition, the second part of the condition, taking the two branch of if and then the reference. And this is really nice. It's one of the best things about the Clang Center Analyzer, the bug report. Because it generates to you the HTML, you open your browser, and you can use the errors just to walk on the path. So it's quite nice to follow and try to understand the bug report itself. Yeah, so this is the wrong assumption that the Static Analyzer did. I was going to do a demo, but my laptop didn't work. So we're going to try again by the end. Hopefully we can get it to work. Okay, so we have an imprecise analysis in the Clang Center Analyzer. So why don't we replace it with a more precise, yeah, why don't we replace it with a precise solver, right? It's wrong then. So that was something that was done in 2017 by Dominic Chang. So he used something called SMT solvers. The one he implemented was Z3 from Microsoft. It's an open source SMT solver. And it's full precision. You encode everything, you have a constraint using this theory of SMT, and you analyze your project. The problem is it was up to 20 times slower than the constraint solver. So it kind of defeats our initial idea of producing something. So we are so slow that in the end the user just don't want to use it, right? So we still want to be fast and we still want to produce some results, some bug reports. So the user can use it, right? So we propose something else. Instead of replacing this imprecise solver in the Clang Center Analyzer, we're just going to add this precise analysis as a post-processing step. So we're going to generate all the bug reports with the imprecise solver and then we're going to post-processing them encoding full precision and check for satisfiability. We check if those bugs are actually feasible. And surprisingly enough, we are adding an extra step and it makes the analysis faster. Yeah, I know it sounds weird, but I'm going to show you the, and explain why briefly. So we are only interested in the path sensitive checkers because we're not looking for pattern matching. So we are looking for the constraints in a path and we're going to call these constraints and see if they are feasible. So we are using SMT solvers. We encode, as I said, encode and check for satisfiability. And so currently we have five different solvers. Zetri, Bolecto, Matsat, EISIS and CVC-4. Only Zetri is in the claim main repo, main repository. The reason why is that we implemented the five of them and basically we had the same results. So right now there's no clear advantages of having them all. It will be just more cool to maintain. So right now there's no advantages of having them. If someone finds a need for them in the future, it's there already. The patches are up. It's just a matter of merging them. So this is the brief overview of the static analyzer and the extra step. So on the left you have the symbolic execution that I said. So basically what it works is the symbolic execution will look at the claim ST. It will give this ST this path to these checkers. The checkers will generate some constraints. So values of branches and et cetera. And just give to this constraint solver, which is the one that is not precise. The solver will try to solve these constraints and will say, okay, this is satisfiable. This bug is real. And it will generate a bug report. So this is basically how the static analyzer works. And now this next step called SMT refutation. We have, for each bug report, we're just going to look at the constraints. We're going to encode in the solvers and check for satisfiability again. And then we're just going to report the ones that the solver said, okay, this is satisfiable. This is feasible. This might happen. So this is the SMT formula for that program there. The first one that I showed you. So just real quick. There was a variable called a, right? So this is converted to style zero. And basically what this formula is checking is get the bit from zero to zero of variable style, sorry, dollar zero. And this should be one. And get zero, the first bit from dollar zero and then should be zero. So basically what this is doing is give me a bit vector, right? A vector of bits where the first one is zero and the first one is one. So this cannot happen because we are dealing with binary variables. So this is not satisfiable. And for their reason, when we do this more precise analysis in that first project, the first program, it will not report the bug. So we evaluated this analysis in 12 projects. So you see some big projects there. I think the biggest one is XNU, the MecoX kernel. So we have those five different solvers that I told you. And so all the instructions to reproduce these results that we had in there. And if you guys, you can check the other results, all the versions of the tools that we used and everything. So these are the results. I know a lot of numbers. Let's go through them quickly. Just heads up. These are the average time of all the solvers. They basically perform the same. So I just used the average there. So out of the 12 innate projects, we removed bugs. So this more precise analysis was worth it and removed the bugs. And on average, about 10% of the bugs were removed. All of them were false bugs. We are not removing any real bug. And on average, this is 6% faster, which is confusing, I know. But the reason is, that's a funny reason. So the kind of technology that generates the HTML reports and they are big, right? So for that small project there that I showed you, it's about 100K. So for every project, for every bug report, you just have to generate these bug reports. So the reason why we are faster is that we are saving on the I.O. So there's no right to the disk of those HTML. Yeah. So that's why it's not a huge gain. It's about 6% faster. All right. So on the projects that we didn't remove any bug, on average, it's a 1% slowdown. So these are really good results because when we can do something, when we remove bad results, we are actually faster. And when we don't, the overhead is minimal. It's 1% slowdown. So on total were 89 bugs refuted. On average, there was a 35% speedup per project. It's around 6.5%. So 35 seconds. And when there are no bugs, there's one second slowdown that's usually 1% slow. Sorry, one second slowdown is 1.24%. That was going to be my second demo. These, we're going to try to make it work later. But all the instructions I was going to give here, they are available online. Hopefully, it's going to show you how to, if you have a small program and you just want to check your small program, there's a nice command line that you can use. If you want to use some kind of automated, every night, run the Clink Stack Analyzer. In the repository, there are some scripts to do that. So you just set the path to your project, and then create a Chrome job. And every night, we just run everything. Every night, analyze, generate bug reports for your tool. Hopefully, we can get it to work later. OK. So we have this SMT support in Clink, right? And it was released as part of Clink 7. So a while ago, during this project, we had to refactor a lot of code from the Z3 backend. And now we have a proper SMT API in LLVM. And as I said, we have patches for all the servers, but since they are not a clear advantage of having them, we just haven't submitted. So what's the future now? So the future starts with that patch. It's called the D54978. So it was too late for Clink 8 and 4th play, but basically it's moving the SMT API to LLVM. And what does it bring to us? So the first one is there are people already using it to validate a transformation pass in the LLVM backend. So in this case, its color evolution is for the original big code and the optimized one and just trying to validate if the transformations are actually real, if they hold. In the future, who knows? We might have an SMT backend in LLVM. There are some problems that need to be fixed, solved there, so memory handling, how we're going to handle loops there. So with quantifiers, you know. Some acknowledgments. This project was done part of the GSOC and all of these people helped me and Enrico to achieve that. And that's it. Well, thank you very much. So this is me, the experiments, if you guys want to take a picture. There are all the shrunkions to run the Clink Statinalizer that was going to show here there. So you don't actually need to... I don't need to actually explain it, but let's try to do it. The Clink Statinalizer web page, and this is a five-minute video just showing you how to run the Statinalizer on a terminal so you can follow and apply to your project. Yeah. Thank you. Yeah, let's try to... Yeah, let's try to... Let's see if it works. Yeah, it's not working. Yeah. Sorry, guys. It's not working. If you go to display settings, do you want to try it? There's no external monitor. No, there's nothing. Not even any video thing. Yeah. Any more questions? Yeah. Yes, so the question was why we didn't integrate the SMT in the symbolic execution. So that was the thing that I said it was up to 20 times lower. And the reason was because how the Clink Statinalizer was created is every time we generate a new state, so every time we are doing the symbolic execution and generate a new state, we query the solver. And this is extremely expensive to query the SMT solver. And to change that in the Clink Statinalizer, we require a lot of work. So that's why we moved to this faster solution which was just using it as a post-processing step. Let me go back. This one? Oh, sorry. Oh, OK. Yeah, that's a good question. That's a good question. So he asked, I said that I removed on average 10%, right? So how many more false positives are still there? So without the refutation, the numbers are high. The false positives are high mostly because of the lack of cross-translation unit support. So it's around 40%, 50%. So this removes one part of the problem. So there is another approach to try to remove those false positives left. So it's still under development. I didn't even put it here, but the idea is when we generate the formula, it will generate for me a contract sample. So every assignment to the inputs of the functions and everything that will trigger this proper evaluation, this bug. So one idea is we can use this information to generate tests, and then we just run the test. The test that don't crash, don't give any error, just remove those. But this is not implemented yet. We haven't started to do it. This is one approach because right now we don't have a good idea how to implement the cross-translation unit support effectively. So I think the test generation, it will be a little bit better. It will give better results for now. Okay. Yes, so going back to the question about solvers. So part of the issue is probably that you're giving the solver more precise constraints. So have you looked, so A, have you looked at just giving the existing constraints and what is the performance difference in that case? And then B, is it possible to restart the solver like D3 so it doesn't evaluate everything from the beginning to the end? Right. So the first question was, the second one was, is there any way of restarting Z3 to make it faster? The first one, sorry, can you repeat the first one? Well, it's about which part of the... Oh, the precision. Right. It's about the precision. Right. And the first one is about how can we control the precision in Z3. So the first one, I'd say we don't want to be less precise because we are already running a less precise analysis and we want to actually remove the false bugs. So we want to be as precise as you can be. The second question is about trying to restart solver to improve the performance. We've tried that. We also tried to do something called incremental solving but the results were not great because the solver, as it is implemented, it's stateless. So we did it. There was a lot of hack involved in the code. It was not pretty and we didn't get any performance improvement. Okay. That's it.