 Okay. Welcome everybody. Thank you for showing up. So my name is Doji, and so this is Sini. Well, I work for Red Hat in the Platform Toolchains group, and I mostly work on static analysis framework to perform analysis on binaries so that we can analyze, ABI's and so on and so forth. So Sini, what are you working on? Hello. Good morning everyone. So I work in Fedora team. Before that I was involved in Libabigil directly, but now I mostly do some Fedoratomic host and Fedoracora stuff and also contribute sometime to the Libabigil project and how we do ABI verification in Fedora. So that's what today's talk is about. And one thing I'd like to tell you that Doji has been my mentor for a long time, and he helped a lot. So yeah. Okay, let's start. So today we will be talking about these stuff. So you will see how exactly we are doing ABI verification in Fedora, how exactly the design looks like and what are the internal tools which we use for the ABI verification and analysis exactly. And then we will look into how we are doing in Bodhi, the gating of the packages update during the Bodhi update run and then the future directions. So our ABI verification design in Fedora is based on the Taskatron. So how many of you know about the Taskatron? Okay, cool. So if you don't know, Taskatron is a framework which helps you to run the automated tests. And we use it in Fedora to run some tests like RPM lint, RPM grill, depth check, and also for the ABI checks. So how exactly we do? So whenever you do a package build, so how many of you have done ever package building in Fedora? Okay, that's cool. That's really nice. So whenever you do a package build in Fedora, so not scratch build, it's a non-scratch build. So once it succeeds, there is a Taskatron task gets triggered. Among them, it's a Task ABI check. And what exactly it does, it takes the package which you just updated and then it takes the last table update, updated package available in the Fedora and then it will perform the ABI comparison between them. And during ABI comparison, it takes the shared libraries which is available inside the packages and it will do the comparison from the previous version of the package library and from the latest one which you have just made the build. So once ABI checks runs, you will see the results like failed, passed, or need inspection. We'll talk more about how exactly we categorize this failed, passed, or need inspection in the later slides. So for now, the package maintainers can see the results of this test, how exactly it is like passed, failed, or need inspection and the logs of the test results, what failed or what's there. So for that, you really need to go and add yourself to this notification and you will see get an email for that, the package test run and you can check how to fix it, the log contains all the details also. If not, if you are going to push that update into Bodhi, the package some set of test runs if you recently have done or maybe in late before that, you will see there is a tab inside the update package, Bodhi updates any package, there is a tab called automated test. So that's where you will see ABI check test as well. For instance, we have this link at the down Bodhi.FedoraProject.org slash update. So you have this tab here which has all the automated tests and somewhere you can find, I don't know, the ABI check tests here. So it passed. So it says that there was no ABI change. So it depends how exactly the test run. If it has some ABI changes, it will show it here otherwise you will see no change. So that's good. Now we will talk about how exactly the design of the two links look like which we use for actual ABI analysis using the lib Abigail and I will hand over to Dochi for that. Okay, thank you Sine. So a few years ago when people wanted to know if a library update contains ABI changes, people would use tools like NM, I don't know if you guys are familiar with that, tools to go look inside the health banners and see what symbols, health symbols are in there. And the kind of changes we could see back then were changes like a symbol disappeared. For instance, you had one symbol which was the symbol corresponding to a function name and boom, that symbol disappeared. And that's an ABI change if the symbol was obviously used by some applications. But then we wanted to have more precise information related to the meaning of those symbols, right? Actually programmers don't even want to talk about symbols. Programmers talk about things like functions, variables and their types. Symbols, that's too much of kind of plumber stuff, right? When you change the type of a parameter of a function as a programmer and if that impacts the ABI, then you want to have reports in terms of what you changed like in terms of types, structures, structured data members and so on and so forth. So to be able to give that kind of information by looking at the binaries themselves, we came up with a framework which name is ABI Generic Analysis and Instrumentation Library, ABGEL. And so that's basically is a library that knows how to, yeah, incidentally we were to starting this project and ABGEL was also the name of the wife of the other guy, but anyway. So it just stuck, no one came up with a better name, so we just kept that one. So that library, which is a C++ library today, knows how to load an ELF binary and of course it's debug information because we need the debug information to be able to know which kind of things we're looking at inside the binary. Debug information will tell us that, oh, from here to here what you're looking at, a variable. That variable has a type. The type is a structure, blah, blah, blah. So we need to analyze debug info just like any debugger or, I don't know, things like Valgrind or and so on and so forth would do. So we load that and what do we do with that? We build an internal representation of the exported interface, right? Just like what a compiler would do, but the compiler will do that from source code and we do that from the binary. So we build an internal representation which is a graph basically and some nodes of the graphs are types and interfaces. Interfaces meaning either public functions like exported functions that are not static, for instance, in C or global variables, you know, things that applications can use, you know, from the binary. So we build an internal representation of that. And then the library also has facilities to compare two graphs, two internal representation. And then once we've compared that, I mean, like think about it as what GNU-DIF does, right, on text files. But one of the big differences of what we do compared to GNU-DIF is that once we have the result of the comparison, we just don't spit it out. We also build an internal representation of the changes as well. You know, not every single change needs to be, like, emitted, reported as is. Let me give you an example. You have, for instance, I don't know, two functions. The first one is name is foo, let's say, yeah, all the functions are named foo, someone. And the other one is, you know my example, right? Bar. So, and the two functions have, take a parameter, which is, I don't know, in C, a pointer to a structure, for instance, I don't know. The structure's name is s. And suppose that we run an ABI comparison between the two versions. So this is a super fancy library we have, right? And we come up with a new version of it and we modify s, the structure s. So suppose that the report says that, okay, in function foo we have a change and that change is, well, it's s that changed, basically. And because of that, it did impact foo, right? Do you want to know by default, for instance, that bar has also an ABI change and that change is due to s? You can say yes, but usually most developers who came up with ABI, ABI div result at the beginning told me, ah, this is noise. You told me about the change of s on foo. I don't want to see the change of s on bar either, you know, like by default. The second one is redundant, okay? So whenever we have the internal representation of the ABI change, what we do is that we analyze that as well and then we will flag stuff, like redundant stuff or, you know, changes that are not meaningful. For instance, let me give you another example. If you're analyzing a C++ library, for instance, and you change some data members, moves from being public to private, I don't know, or the other way around, okay? This is a change and we see that in the debug info and we detect that. But is that really an ABI change that is meaningful at first sight? It doesn't really, you know, break things, basically. Some programmers would want to see it, right? So we will put that in another category, which is a category, you know, which says, okay, it's a change but not very important. So we do all these things on the graph of ABI changes as well. So this is where we're different from something like new div, you know? We also work on the changes. And at this point, we give a chance to users to say what they want to see as changes and what they don't want to see. For instance, you can have a second function, a third function named baz, I don't know, so full bar and the third one is baz always. And for a reason, baz is exported, you know, as an elf artifact. Programming model in your project, baz is private to you. It's only meant to be used by other libraries that are part of your project. So baz is public from an elf perspective, but exactly. You don't put baz in header files, for instance. So you don't want to see changes on baz, even though it's public. Yeah, programmers are weird sometimes. So for that, you can write what the Valgrant people called a suppression file, right? Saying, okay, this function, I don't want to see any changes related to it. We call that suppressions. And so suppressions will be taken into account when we are analyzing the graph of ABI changes. Okay, so this will be at that point, basically. And then we have passes. It's basically compiler stuff. We have other passes that will walk through this graph of changes and will report those changes. Okay, so you see the pipeline a little bit? So this is what we do. Sometimes it can take a lot of space and time, because if you look at it from a tool chain perspective, what we do here is just like what linkers do. Okay, we look at the library, the .so file. And from the .so file, we break it apart and we look at every single translation unit. You get that translation unit that made up this .so file. And from those translation units, we build a common model, right? So just like what linkers do, except that linkers don't look at types, right? They just look at relocations, symbols, and so on and so forth. When you start looking at types, things can become really messy and super big, okay? So that's the main difference between us and linkers, but they're similarities. So based on that library, we came up with different tools directed by the feedback we got from you guys, from users. So we came up with a tool named ABI package diff, which takes two RPMs today. Someone added DBN support, so it works on DBN packages as well, anyway. And for those RPMs, we will look at, you know, the binaries or only the shared libraries. It depends. I can look at both that are in those RPMs and we will perform comparisons, ABI comparisons on them. Of course, for that, we also need the debug info RPMs, okay? And these are, so it looks, things always look simple when you put them on slides. But think about it a bit. The debug info is usually stripped when we use our, you know, RPM pipeline. The debug info is stripped out. So we, that's one thing. And secondly, it is compressed. You know that, right? We use a compression scheme carried by a tool, which name is DWZ. So it's an ad hoc compression scheme that we use. I say we, I mean the tool chain people. So what Lee Babigel does here is that it also has to decompress this debug info. So debug info that will take, I don't know, like one gigabyte of space decompressed. Right? So decompressing that has, you know, bring challenges because we have to, you know, try to reduce the memory consumption, you know, to not make things explode. There are interesting things there. So we do that. Bug info. And then we also take the develop RPM packages into account. Because remember, we were saying that you can have some exported functions or global variables that are not meant to be, like, consumed by applications. They're meant for internal purposes. How do you know that these are internal stuff and not public APIs? What we do is that we look at the header files, okay? And we try to be maybe too smart for our own sake. Because one of the main purpose of what we are doing here is that we don't want to look at the source code. We really don't want to do that. We want to look only at binary files. Do you get why? Okay, the reason is that, I mean, there are some profound implications to that. Because today, when you look at the source code, you cannot know what the binary form is going to be. Because in C and C++, we are subject to the tyranny of the preprocessor. Which means that there are stuff missing from the source code. You know, there is information missing. Meaning, for instance, the value of the defined, you know, of macros. You don't have that when you look at the source code. So you have to know the build system stuff. So once you start willing to parse source code, you get into a rabbit hole. Because you need to be able to parse the build system. And we have auto tools. We have mess and we have, you know, it's super messy, the world of build systems. So we just want to look at the binaries, right? And then, well, then you have the binary form, okay? But then, now we want to look at header files to know what the public interfaces are. So what we do is that we are going to, when we analyze the binaries, we look at an interface. And in the debug info, there is information in there saying where, in which file, this interface has been declared or defined. I'll repeat that. When you have a structure that is defined in a header file, a foo.h file, that information is present in the debug info. So you see that struct S is defined in foo.h, okay? And then what we're going to do is that we're going to go get, see if we have foo.h in the header files in the develop RPM. If we do, then foo.h is part of, you know, the public interface defining header files. So we know that S is part of the public interface. That's how we do it, you know, without having to parse the header files. So we need the develop RPMs. So once we have that, we will auto-generate some suppression specifications, saying that whatever is not in foo.h, we don't want to see its reports. So this is how we integrate all this in the pipeline, okay? To reduce, of course, noise in the final reports. Then people were not happy enough. They wanted to do weird stuff, programmers again. So we came up with a tool which name is ABI DW. What does it do? It emits a textural representation of the ABI of a binary. So suppose you have a binary. You want to know what is the ABI of that binary, and you want to save that ABI into Git without having to save the binary into Git, right? Because nobody wants to save binaries into Git or something. So ABI DW just pits out an XML file. We call that in a format that we call ABI XML. So you can just take that boom and stick that into Git. And then afterwards, there is another tool which name is ABI DIFF, which can take an ABI XML file and a binary. So you will compare the ABI of the binary against the ABI of the ABI XML file, which is the baseline ABI, for instance, for you. And that is useful for people to build ad hoc ABI checking stuff that they can run during the build system. There are some upstream projects today that have a make check ABI target that is run regularly. And so they use this kind of stuff, OK? So many people do have different use cases in mind depending on their project. And if we don't have the tool for that, I mean, it's OK to come to us and say, what if you guys could come up with this kind of tool and we will try to write it if possible. And if the libabigail library doesn't have the features needed for that tool, well, we'll just add it because we don't know all the use cases in advance. And there is also Fed ABI package DIFF, which is intended for Fedora specifically. As you can imagine here, using ABI package DIFF can be tedious. You have to get the two RPMs you want to compare. You have to get the debug info RPMs. You have to get the develop, you know, and at some point you just get fed up, fed up. So we said rather than being fed up, use Fed ABI package DIFF. And yeah, I do my best for jokes anyway. So this one is quite cool, even though it's written in Python, nobody's perfect. I didn't write that. But anyway, apparently, writing it in Python was mandatory so that we could use the Koji interface or something. There are always bad reasons to use Python. Anyway, but it's a cool tool. Seriously, I like it a lot because you just say Fed ABI package DIFF, you give the name of a package like, no, Fed ABI package DIFF, you say from Fedora 27, for instance, and the name of the package you've just built, right? The RPM, you just like HTTP something.rpm. And so Fed ABI package DIFF is going to compare the ABI of the HTTP package you just built against the stable one from Fedora 28. You don't even know the version. You don't even know where it is. Just go grab it even if you can do some nice stuff in Python, apparently. Anyway, and it works. So this is typically one of these things that we didn't think about in the beginning and Fedora users met us right. And we have other tools as well, but I won't bother you guys too much with those for now. So, of course, today Task ABI check, so the ABI verifier of Fedora uses ABI package DIFF. As a matter of fact, it was because of ABI check that Cini wrote ABI package DIFF to begin with. So always this, you know, use case-driven approach that we have. We don't do intelligent design that much. We just, we grow like a cancer or something. Anyway, so what we do is that, you want to talk about this one? Yeah, okay. We just compare, so the new package against the old one, as you guys know. And because this Task ABI check uses, uses the TaskAutron infrastructure, there's some lot of nice magic in there to get, you know, the debug info packages, the develop packages. I call that all the ancillary packages needed to do, to perform the comparison. And it's all magic. Boom, TaskAutron infrastructure gives us everything. And we can perform the comparison without TaskAutron knowing exactly, you know, what's happening. And so, yeah, it's kind of, you know, well integrated. And I, for instance, just to give you an example. At the beginning, we wanted, well, I wanted, I like multi-threaded stuff. I like doing stuff in parallel. And we were talking, I think it was with you, coming out with about, you know, like TaskAutron. Is it, you know, like multi-threaded, blah, blah, blah. And well, just because we could write ABI package DIFF, you know, underneath, we put all the multitasking, you know, parallel handling in ABI package DIFF. TaskAutron just calls it and we perform, you know, like, if, I don't know, if your, your RPM has like five libraries and you have at least five cores, right, on the machine, then the five comparisons are going to happen in parallel, you know, this kind of stuff. So it's pretty, pretty neat. So as you know, we get the public interfaces from the DEVEL package and we provide, you know, expression automatically based on, based on that now. So as a result of this, you have to know that, I mean, ABI comparison is a gray area kind of thing. We cannot say, you know, we cannot always say if a change, an ABI change is good or bad, basically. It depends. There are some that are, you know, bad, like, I don't know, a function disappearing, a public function that is in the interface that disappears. A type, I mean, like, for instance, a return type of a function that changes completely. We flag that as not good. But there are other changes that are weird. So the first kind of change, when we know for sure that they are not good, your test failed, failed, sorry, the status of the test is failed. But then suppose you have a structure, I don't know, which gets changed. Like, you remove a data member from it and you add a new data member to it, right? And the size of the structure doesn't change. Is that good or bad? Hard to say. So we flag it, we detect it, and we say that this kind of change, it needs inspection. So these are the kind of gray kind of things. And then there is something I wanted to say here is that if you keep seeing these kind of changes consistently for your project, you know, and in your opinion, there are better ways to categorize these things, come to us, and even for your project, we can figure out a way for these needs inspection kind of changes to become either failed or passed. Is that clear what I'm saying? Yeah, because we need to evolve that way based on your feedback. I don't just say, ah, the tool said this, ah, this, okay, I'm not gonna use any F words here, but okay, this tool, yeah, no. Yeah, Fedora, yeah, yeah, of course. This Fedora tool, yeah, thank you. No, seriously, come to us and we'll try to figure out something. This is how we grow and get better. All these static analysis tools need to, you know, start with input from users. So this brings us to the body updates getting project. So as you guys know, at some point, well, I mean, even at some points, people wanted to use more automated tests to get package updates through body. And, well, we started, you know, enabling ABI verification in that framework. I mean, using this ABI check as a gating, right? Yeah, so, and we have pretty interesting results. Thanks, Ochi. Yeah, so we already have the test running in body updates. In the web interface before. So this, with the current approach, what happens that package manager, package maintainer can go and see in the body test results, whether ABI check has passed or not, or he can go and see in the email notification that the test has passed or failed. But there is no way that we are saying that, okay, go and fix it. Maybe maintainer can go and just ignore it. And we might have seen a lot of, some of at least email threats on Fedora Devil that this package has been updated and there was ABI change and it has not been announced and that leads to a lot of package being failed. So we tried to do in Fedora to get the body package updates using the ABI check as well. So in January end, I think we started getting the package updates with some other tests. There was the ABI check as well. And in this, what we do is that whenever a package is pushed and the updates is available in body, if there is a failure, then it will stop it from being pushed default. It needs to be inspected by the maintainer. He needs to fix it or he can go and weigh with. That was the approach, but it didn't work out and it just failed. The reason we look at it. So basically what happened that we got very quickly some feedback that the tests which are running ABI check is getting them. It's like there was loss of false positive results coming up and they cannot really push the updates available. And then the reason was we look. Yeah. So what happens that in the shared library we drew ABI comparison for all the shared library available in the package. And the issue which we faced was there were a lot of libraries which were considered as private libraries as well as the maintainers. For example, if you go to the LibreOffice package or anything, there are some shared library which is supposed to be used for the internal purpose. The private libraries is like, yeah, it is being used for just inside the project and it is not supposed to be used in the other applications. So these changes were really not supposed to be getting the package from being pushed to the stable. So what we did was we stopped the gating based on ABI check as soon as possible and then we started looking into how to fix that. So there was really no way in the ELF binary to say that this is a private shared library or a public shared library. All are like just a shared library. So we came up with an approach after some discussion that if an RPM package says that we provide these SOs, so we consider only those as the public shared library and the rest of the library would be the private shared library which is supposed to be used only inside the application and got implemented into the Lib Abigail itself, ABI PKG TIFTOLIQ. And another was it was a minor stuff which I think we may start or we came to know about while doing this fixes. So we were, so suppose if you have an RPM package update and there are some sub-packages, so if you don't ship devil or a header packages into that which means that we should assume we don't really want to use your application to be consumed by another packages. So if you're not shipping any devil package, we are considering it as private. Any shared library is shared as private. We do not consider it as being used consumed. So that we implemented inside the task ABI check itself and these are the two things we did and Lib Abigail 1.4 which was recently released contained the fixes of the first issue and it's now again ready because these were the main issue which we saw so far. So maybe we can enable again the gating of the Bodhi upon the task ABI check. So initially it was I think enabled for all the packages but maybe this time we can start with gating some of the packages or obtain basis like some maintainers who wants to opt it. They can obtain and see how it goes. So yeah it's just open to suggestions what we should be doing. There's one more stuff like if you really not satisfied with the current separation which we do like not considering the private libraries and if you still want to suppress some of the libraries or any inside symbols as considered to be private, you can write your own ABI specification. There is a you can have ship dot ABI ignore file inside your package and you can ship it and Lib Abigail and the two links or top of it will take care of it and it will not show you as any ABI changes as a false positive. So that's all and we'll look into the future directions. Yeah future directions. So first of all before even getting into the specifics of what we wanted to do in the future and so on and so forth. I wanted to say again that it's do not please do not be shy. I mean come to us. There are many things that are project dependent. For instance, so Matthias is here in GTK and G object based project in general. There is something that's been in the back of my mind for some time that we need to do and we haven't you know done yet which is that okay you know by default today we only look at interfaces that are public you know not static and we get the types that we look at from those interfaces. These are the roots of the graph that we built. But then in GTK and G object stuff you have some static functions. So they're static. They're not seen outside the binary. For instance the class stuff you know that we should look at because these things defined some of the types that are used in the GTK application and so on and so forth. But because the function that use these types are static we don't see those. So someone can break these things you know these V functions that end up like V functions in the class structures. So I think we can do something like you know detecting that we're looking at the G object thing and look at those static functions as well rather than just ignoring them. We can do that but it's project specific and we have to you know like yeah define how to do that. So yeah it's important that we get to have these discussions even if you know we're like bandwidth stripped. We can still design what we want to do and then when I have the bandwidth I'll you know work on that. So please get you know come to us and so we can do stuff for your project. We even supported ADA recently like if we do that we can do many stuff right. There are some people like who want us to support Fortran. Yeah anyway so yeah we can do a lot of stuff. So first of all I think one of the first thing that people are asking for is speed improvement. We get these like all the time but it's never enough like for instance if you want to analyze Firefox you know. Okay so basically what I think I would like to look into now is doing things more in parallel. It looks like well at a deeper level. So I told you that we're building an internal representation okay but we're loading two binaries usually because we are comparing stuff right. We're loading two binaries but at some point we are building just one graph even though we have two binaries because there are two graphs but we try to share many things. So if for instance we're loading the first binary okay we did load the first binary. We're loading the second one and we see a type which name is INT integer. We've seen INT in the first binary too right. But this time the two INTs are the same. What do we do? I don't build, I won't build another INT you know. That is memory waste. So I will reuse the first INT you know. So even though the two graphs are different they were used there are some nodes that are...