 Well, the title was already set, it might seem a bit complicated, so what we'll do first is Let's see what actually is this tool or this project trying to achieve The goal is to check if some part of the kernel, the Linux kernel, behaves the same between two different kernel versions This part can be either a function or it can be some setting, some runtime parameter, we want to check if it affects the kernel the same etc. Generally, and I will stick to this, we will compare the change in behavior of a function between two different kernel versions As for, for example, parameters, these are represented as global variables, for this we will check whether, well, we will check all functions that use this variable and we will again check for equality, so generally we will check whether two functions behave the same between two different kernel versions This is very difficult to test since you would need a test for each such function, for each function that you want to compare, you would need a test and moreover the test would have to cover all possible parts that through the function. This is impossible to write, so what we will do, we will check the source code So this was the what is going on. The next step is why do we need this? Why bother comparing such things? Well software evolves, it evolves fast and evolves quite a lot and especially in such a project as kernel is which is community-developed there is a lot of changes and you really cannot control or you really don't know what's going on after each change but in the same time if you want to use such project in your product, it is often very nice to have some stability and compatibility For instance, it is if you have some setting in kernel, for example if you set some runtime parameters to some value, you expect it to behave somehow and after that, if you have a new version of the kernel, you expect that if you set the same value to the same parameter it will behave the same, but is this really the case? Can you be sure that it really does behave the same? Second thing or second use case is in Red Hat we guarantee stability of so-called kernel application by their interface, which is basically a list of functions which are guaranteed to have the same semantics in different releases of our kernel. Also here, we want to check whether this really holds So as I said, we'll use some checking orders code and the first obvious thing is let's use diff Can we do it? Let's see an example. Here I have two functions It's called biopage from REL75 and REL76 for kernels in these rows The question is, are these functions the same? Well, from the first site, yes, they are. If you pass them to diff the diff would be empty, so yes, they are. But the function calls another function, for example, actually two of them, but let's stick to this one, maxSizeOffset which actually it can be different, right? And truly it is. If we check definition or implementation of a function, we can see that the return value changed If the return value changed, then the value of this maxSector variable could have changed and it could have changed also the result of the original function, but we would not see it by using diff only to the top-line function. So, actually, at least our problem requires checking of the call functions This could still be doable with diff. So let's see another example Are these two functions the same? We have a function, the TCP propinit, from two different kernel versions and well, according to diff and according to the image, obviously they are not We have the second function has some macro which checks if some condition holds and if it doesn't, it probably creates some bug diff would say No, these two functions are not equal but what if we wanted to check effect of the boothSize variable, it's a global variable, which can be set by the user by setting runtime parameter So we do want to check whether the effect of this particular variable on the code is the same between these two versions. If we look into it, the value of boothSize cannot affect these two lines and here these four lines So basically what we need to check here, if we want to check with respect to the value of the boothSize, is that we only check these blue parts and they are actually even syntactically equal. So here we see that the problem requires at least some understanding of the code. Our tool has to somehow understand what's going on to to be able to separate these cases because otherwise it would yield just too many false positives. By false positives, I mean a situation when we say yes, there is a difference and in fact, there is no difference or not at least from the point of view of what we are interested in and we can go even further. This is not from kernel. These are two different implementations of C standard function, so therapy, BRK and the question is are these two functions equal? If we give the same two pointers to both of the functions, will the effect always be the same? Yes, it will and this can be proven. So for every possible argument, the effect on the functions is always the same. This requires quite a deep understanding of the code. So the question stays the same. How do we do it? Using this, not sufficient. We have already seen it. We have to use something what's called a static analysis, analysis of the source code. We can analyze the C code directly. Yes, we can, but the C code is quite complex, quite structured. We will need to write our compiler, etc. It's just to complicate. We can analyze assembly. So the code that is generated by compiler. This obviously is not a good idea as well. It would be machine architecture dependent. Again, we would need to parse it somehow, etc. So what do we do? We use a compiler. We use its internal representation and we run the analysis over it. If you were on a talk by Ulis Repriestere about the compiler optimizations, you saw that compilers actually do this. They translate the code into some internal representation and then run transformation analysis over this code. Advantages are clear. It's simple and simple, but still it contains much more information than assembly and we can use this information to compare better. In the project, we will use the clank or the LLBM infrastructure, compiler infrastructure, and why we will do this? There are multiple reasons. The first one, it has quite well structured and human readable internal representation. It can be represented by such nice, so-called counterflow graphs. So it's easy to debug. Well, easy. It's easier to debug and so on. But what is more important is that it has very good infrastructure. It already contains a lot of built-in analysis and code transformations, which we can actually use. It has a nice API and can be used as a library, which is also an advantage. And last but not least, there's already a lot of static analyzers built upon LLBM and they usually do their analysis on the level of the intermediate representation. And so we can use these analyzers to get some useful information from the code and we will actually do it. So let's see how the overall system or how the overall tool works. At the beginning, we have either a parameter or a function. Generally, as I said, we have a function. First, we need to find a source code or source file where it is defined. Then we compile it and we get the LLBM internal representation. Next, we do, I think, this slicing and simplifying, which essentially takes the representation that went out of the compiler and produces a much simpler, but semantically equivalent version of this representation so that it's much easier to analyze. And now the diff comes into play. First, we do a syntax diff. So we compare what is left of the functions syntactically. If they are syntactically the same, then one, we can say that these two functions are equal. If they are syntactically not the same, we're not done. We need to perform a semantic diff and, based on the result of the semantic diff, we will see or tell if functions are equal or not. In case they are not equal, we will also yield or produce an output of diff so that the user can see the actual change. All right, let's go to, let's see individual parts of this. For source finding, we use Cisco, which is a tool for finding definitions and declarations and usages of functions, variables, et cetera, in projects written in C. It works pretty well, has some bugs, but we were able to overcome them. As for the compilation, this is more difficult because Clang is officially not supported for kernel building. Basically, what we do is that we use kbuild. It's an internal build system of kernel. We check which command would be used by kbuild to build some module function, file, and so on. We take this command, remove all optimization because for analysis it's better to have no optimizations, and then run it with Clang instead of GCC. This gets us the internal representation. As I said, Clang is not supported for kernel building, so we need a bunch of hex that we implemented to overcome limitations. A good example is that Clang doesn't support assembly go-to, which is now required by kernel, or kernel requires that the compiler compiling it has to support this feature. That's why Clang is not supported. The outcome of this is that the produce code is not compilable into an actual executable or an actual object file. It's usable for analysis only, which is sufficient for us, obviously. The next part is simplification of the code. First part is so-called slicing. It's a technique of removing information from the code that is not relevant for the current context or for the current analysis. This is the example with the boof-size variable that we have seen in the beginning. You can see that if we want to analyze this function with respect to the value of boof-size, then we can remove these first two lines because these two lines can never be affected by the value of the boof-size, or of Google. Next, we run a lot of code simplifications, either those that are built into Clang, or we have some custom, such as, for example, removing content of printed functions because we are not interested in, well, if some module prints has some different warning message, it's not a semantic change, then that code elimination, which is built into the compiler and a lot of stuff, which simplifies the code so that it's as simple as possible to analyze it, but still keeping the original semantics. And now the diff comes into play. The first part is syntactic diff, which we based on LLVM's component called function comparator. Basically what it does, it takes codes and goes instruction by instruction and compares them if they are equal. This is basically a syntactic diff, but with some features, so it can discover or it can handle some syntactic changes that do not affect semantics. For example, variable renaming. Since we are using the internal representation, variable names do not have any meaning, or we don't care about the name, so if you rename the variable, this will already handle it. Another example is it can handle changes in structure layout, bit casts, etc. In case that syntactic diff says that functions are syntactically equal, we are done, as I said. In case it doesn't, it returns a list of functions that are different and that need further analysis. And that's where semantic diff comes. And that's one of the reasons why this talk is in the academic section, because that's where science comes. Our semantic diff is based on an academic tool, LRF, it's developed in a university in Germany, and the tool takes the programs and translates them into logical formula. The logical formula expresses the effect of the program. Then we use a tool for solving logical formula, a so-called SMT solver, maybe some of you heard about it, some of you may be used it, and we ask the solver the following question. Is there an input? Is there some input such that the first program run with this input yields a different result than running the second program with the same input? In case the solver says yes, it is, there is such input, then we have proven that the programs are not equal, because we have found an input such that both functions have different effect. If the solver says no, there is no such input, then we actually proved that the programs are for sure equal. All right, I think we have some time, so I would like to show you a demo of how or what the output looks like. Can you see it? Okay, make it larger. So we'll run difference in two functions in KBI. We get versions, these are the REL 7576 versions. We specify the function and we say that we want the syntactic diff. Let's see, hopefully it works. Take some time to compute and we can see the result here. I have my mouse, so it says that it is actually, this is the function from the very first example that I showed. We can see that this is the name of the wireless symbol, this is the name of some code function, and we can see the code stack, so how we can get from the original function to this code function, which is semantically different, and we can see the diff, which is actually, you can see, if you recall, my first example, there is change in the return value, and I did some of my statistics. Okay, let's see another kind of or more verbose output. We will check difference in a set, in a parameter, in a run time parameter that you can set by ctl. We'll use a different, well, front end versions, and now we will give it name of the parameter, which user actually can set up. This is a run time parameter that user can set. So, let's go, and now, since it's a parameter, which is actually represented by global error, there are a lot more things checked. I will maybe make it slower, yeah, not slower, smaller. Here we can see a more verbose output, and we can see that it compared quite a lot of things. First, it compared a so-called cross-handle function, which is a function that is called when the parameter is set by the user. It found cross-handle function of the first one, the second one, it's called this, and we can see that it has n equals index. After that, after the user sets this parameter, it reflects in setting a value of some global variable that can affect various parts of the kernel, and actually we can see that it can, it is used in one, two, three, six functions overall, and the tool check that all of them are syntactically equal with respect to that particular global variable. Again, some statistics on the end, at the end. All right, it's near the end. So, that's basically it from my talk. I would be very happy if you want to try this camp. We have a prepared docker image or container image, as I found yesterday. You have to call it. So, we have a prepared container image on Docker Hub. You can download it. Everything is prepared there. It's an open source tool, so you can, we have it on GitHub. You can also find a readme there, where there are instructions on how to run it, et cetera. And if you have any feedback, if you find some bug, you can file an issue. If you fix the bug, that's even better. You can send a PR. Everything is welcome. And that's it. Thank you for your attention. If you have any questions. Currently, it is usable only for kernel. It's designed for kernel. But generally, the method or the approach can be used for anything. And I was trying, this is one of the reasons why I used LLVM, because there's quite a lot of languages that can be translated into LLVM internal representation. So, basically, you just, well, you just change the part of compiling, et cetera, and then the comparison stays the same. But currently, we are developing this for kernel. Yes? Sorry. No. We actually know there was no semantic analyzer run. The semantic analyzer itself, it's quite a difficult procedure, which is quite expensive. And actually, in kernel, there are only, well, there are no such things that I showed you for the Lipsy implementations. There are only very few examples where it actually helps, because usually, when there is a change, the change is real. At least in kernel, I was working on. Yeah, we analyzed the whole, well, we dipped kernels between Rails 7.5 and 7.6, and out of, I think, there is like 750 wireless functions, and 100 of them are syntactically different. Or 100 of them have caused some function that is syntactically different. This one? Yes. That is true. That is true. Sorry, I can't hear you. I cannot tell, because I'm not a kernel developer. So, I mean, yeah, yeah. No, no, no. It wasn't enough for a question. This is actually what, this is the reason why we're doing this, is that, of course, these things must be checked by developers of that particular kernel module. But he doesn't want to go through thousands of lines of code that is equal. So with this, we can give him the particle or concrete changes, syntactic changes, and he can check and say, okay, this is fine. This is actually an optimization. Let's keep it. Yeah, but it's not up to me or up to the tool to say this. You were first, yeah? That's why I'm using, we are using it without any optimizations. We could have a difference that is, yeah, we could have a, or to repeat the question, the question is, the statement is that we actually can have a difference which is not seen in the syntax, but it appears once you compile the same code with minus O2 or minus O3 in GCC, and there is a semantic change. We cannot detect this because this is a compiler change. It's not a source code change. So, unfortunately, we are not able to detect this. I think we are out of time. So thank you for your attention for your questions.