 I come with 40 minutes. Oh no, no, I don't. I'm going to have questions. I'm going to be in an interactive session. Welcome back for a new session. May I please remind you about the feedback. Don't forget to log on to the DefCon website. It's like feedback. And leave your thoughts about the presentations that you're attending. And also don't forget, in 40 minutes, our presentation to go and pick up your tickets for the party tonight. That being said, let me introduce Stanik, who is going to tell us how to be compatible in user space. Hello, I'm Stanik Ablach and I work for Red Hat as a senior software engineer for quite a few years as a team member of LVM Developers. And I've been asking Radek for, if I should give this year another presentation about LVM. And he, like I said, he would like to prefer to get a real developer talk. So I hope there are not people who expect to learn something about LVM in terms of command lines. But I assume that you all came here as developers and you want to either learn something new or maybe you will teach me something new. Please feel it as an interactive session. So if we do something wrong, let's fix us. And if we do something good, so you can learn from that. So what is my talk going to be about? I think everyone knows that the kernel, Linux kernel is compatible like forever. So the API from the kernel space is kept forever compatible and you can rely on it. The problem is that in user space it doesn't apply at all. So what do you all talk about? I will first mention what I mean by your compatibility and why we should be compatible. Then few words, not the few words, but I will talk about what does it mean to write a project in a compatible way and I will mix it in with our LVM solutions. So there will be few words about the solution there. I will even show you code from LVM to demonstrate how we will do the things. So why being compatible? I often hear from my colleagues that they complain that they have to maintain a huge set of back ported patches and they have to work on many releases at the same time. In LVM we take a different approach. We maintain one single upstream branch which on the other hand has to be compatible for all releases. So we have the same code base for Fedora, Rohite, Fedora 21, 24, 23 rails and also the device to Ubuntu and other distributions. The other reason why being compatible makes sense is that you often may want to do a git bisect and if you want to do it on a system which is maintaining compatibility it's easy. If you do it on a system which has like system, the new dev and things like that, the bisect gets somewhat more complex than you would expect because you not just solve the problems from compiling your tree but you have to also change the old your tree to apply to the new state of your machine. So it's often quite complicated. Then the other thing which at least from our perspective is like lesson learned, you often figure out after even many years that you caused some regressions either in performance or in command line parsing for example. So checking for these things is often very good reason why to be compatible. Of course, by doing your project compatible you support your users. They will be grateful for that because you make their life easier and sometimes I miss that from other projects. And of course the very important point is that address may depend on you. So if you provide a compatible API and you stick for that for many years then other users will depend on it and will trust you that you are doing your work good in a good way. So otherwise you are in the risk of this famous strip where basically the people come in and see okay, this is incompatible. I would like to rewrite and do my another project to do the stuff. So to avoid that, develop in a compatible way and you will eliminate most of these situations. So this talk is not going to be about reproducing the Uli interpreter talk which is quite old already. So it was I think released in 2002. It is available on this URL. But I will mention what we do for the compatible compatibility. So one take on that is that you want to be compatible on the compile level. So you provide some header files, functions, structures and you want to maintain the compatibility on the compile style level. This is relatively easy compared to other points. You mostly get instant warning from the compiler. You just need to check on all the distributions that you are still compatible. If you want to be a low-level library and you want to use the latest, the greatest features of your compiler, well sometimes it's not a good idea. So if you really not need them, avoid them. So if you, at least for our project like LVMS, we do not need to use the latest extension of GCC. So if you stick with this or at least you provide wrapping defines, then you should be fine. But you need to check it. Basically you have to run your builds on multiple distributions and check that you are not causing problems. That's, I will show at the end of presentation how we will do it. And now the binary compatibility, which my talk is going to be mostly about or a big portion will be about. We have two kinds. One is backward, which means that if you install a newer, your new library, it will still run the older software. So that also means that the output of the older software is consumable by your new release. So basically, whatever writes the old software is readable by your new software. In terms of LVM, for example, the LV metadata are readable and usable. The other one, the format is much harder one. That basically means that you have to think to the future. So if you have an update in the future, you should not, your new tool should not cause destruction of the older tool. So if you have like demons, which are communicating with each other and you upgrade them, and suddenly the new demon starts to throw a new type of data, it should not cause the core dump of the old one. There should be a communication versioning and the old tool should recognize that it does not understand the new language, but it should not crash like it happened for us, for example, also. So these are all lessons learned. Run time upgrade is something where I think the system is still not doing the best it could. And we try to keep the state of demon past the state of the new version and run the version in a new version and keep the same data. So this compatibility is much harder to think about, but you should not forget about it. So now I will say a few words about how we add the symbols in the library or how we change things in our library. I should mention that we talk about the LibDeviceMap library because that's the only library we provide in a compatible way. So exporting symbols means that you provide correct versioning for each new symbol. There are a few ways how to look on that and we will demonstrate it on the source code. So if you, I don't know, any one of you are playing with those tools like read-off and checking symbols in the library. Ever tried that? Okay, good. Anyone else? So how many of you are versioning your symbols in the library? Never ever? Well, you should start with that because that's what allows you to keep the compatibility to the future. So basically what you have to do is to generate a file which has some structure, but you can have some names and you can specify the function names. You end it with a local one. So basically you have a full control over what is exported from your library. So you really should take care of that and you should not export functions you don't want the other users to use or see. It's important because otherwise users will start to use even functions they were not meant to use. They were hidden for your usage, maybe, but they were not meant as an API. So you are in danger that users will look at the source code and start to use functions that they should not really use. So keep this in mind and hide all other functions. Only export what you want to export and what you want to support. That's an important thing. So the fun begins when you realize that you made a mistake in your API. I'm here talking just about the C interface. I'm not going to talk about the C++, basically one of those things is that LVM is still in C and probably will stay that way for a long time. But even in C you can make mistakes. In C++ it's much harder but in C if you forget that the structures have sizes and you are not returning those structures as an allocated block of memory but you expose the structure in the header file and you realize that you want to extend your structure in your header, then adding new members to the structure seems like a natural way how to extend it and nothing bad can happen. Unfortunately that's not true. Of course if you recompile all your software everything will work but then you have a problem that you can still have a users which do have newer header files, compile the older software and try to use it with your new software and then suddenly some structures passed to your new library will not match in size. So this is something you have to take care about and how to do it. So you will have to provide multiple functions for the same functionality. It's good for you if you can match the functionality in one newer function and you just add wrapper functions to provide interface for all users. So basically what we do in LVM and what I will show you right away is that we create a new function which has the proper versioning. We stick with the dm and version number. So now you will have multiple symbols in your library. One will be with the base. That's basically the sign where we were not realizing we have a problem, we still use the base and from the new version we will have a new... For the new function we will have a new version. And now how it looks in the source code. First maybe I will first show you how we started to generate these version symbols. When we create a new function in a library we put the definitions here in the single file. So we now from version 97 track of the history how we add the functions. And we combine them in a final file through the rule, make file rules and how it looks in the code. So here we have a new function which takes a new structure. And this is a function which gets the latest, greatest suffix with the version 97. And we provide a wrapping at the end of the file where we have the logic and the file for the old functions where we call the function getTaskGetBase and this function has no other functionality than taking the structure with the old size remapping it to the new size and calling the new function. So there are no problems with reading out of memory places. So basically that's all we have to do in this case. So it's not an excuse that if you have to change the structure you do not really need to release a completely new version of incompatible library like we see many times that developers realize that they have made a mistake and generate a new and new versions which are incompatible and cause quite some troubles to many other users. So it's okay if you just develop some plugin for some GUI tool whatever but if the whole GNOME library works in a way that even the minor changes cause that you have to throw up all your plugins, it's not the way it's been meant to be used. So is this somewhat clear to you or should I say a few more words about that? Well I just provided at the end of presentation I provide the link to LVM to LVM sources. It's really complicated because we have also quite complex make file to process those exported names. We even detect the problem if we provide a new function and we have forgot to put the new function in the proper file so we spend quite some time on that so I can probably show you after the presentation. So now what are the other problems? We are not realizing and we realize that after we spend a while on figuring out what the bug report means because we were receiving the bug reports that users are getting strange core dumps and they are installing the libraries and it's not working and we were looking at the core dumps and it's been showing that they were just accessing the memory from functions which were just looking all fine and it was not possible to explain and then we asked what the user did and they told us well we used the latest Koji to build a new library and we first installed this new library to use with our old project so we said oh we have to somehow prevent to do that and how to do it? Well there are a few things I was not aware I came from the Debian world and in Debian you store the dependencies in a very strict way so when you build a package the dependencies on other packages in raw form while on RPM the libraries are expressed differently and you express dependency on the symbols so in RPM it's not strictly limited which library will be used just to satisfy the versioning so whoever thinks that the so-called major minor version solves your problem it will not solve the incompatibility for the package maintenance but it will not solve you problem with the loading libraries which do not have right symbols so basically the problem with the structure size oh I have wrong focus these structure changes were not captured well by the RPM so we had to think about how to do it and if you check with RPM commands what the libraries are requiring or providing you will notice that it's not just glib c6 but also there is some extra symbol and that's what we learn to use so basically this is the way how you will ensure that the users of your library will map the right functions for their binaries and we can see that even in you can see that now the device mapper package contains the tool called DM setup and DM setup has dependencies on all these symbols so whichever package provides them it will satisfy these dependencies so there is one stick dependency from building the package but it's not what you will get if you are using the tirsmap library so this is what you will use in your package to bind the library properly the other important thing to think about is what you should do when you add a new feature to your project and that relates to the fact that we have one single code base in the game so we have row six, row seven still in the single tree we do not develop multiple feature branches and fix bugs fixed in the row seven in the row six and so on so things to think about if you add a new option you should provide a configurable configurable option to disable and enable it if you are sure you do not want to like provide a new feature to the all distro you should do it in a configurable way so the package maintainers of distros may select by single configurable option line what they want to have compiled or not they should not start to play diffing and patching your binaries just to disable something which they are not or they do not want to use then the other thing is you should also provide if that makes sense if it's something that users may want to use you should provide configurable defines because defines defaults may change from distro to distro in one distro usage of something might be a default and in other distro it might be disabled for example lvm we have like demons and you want to have them supported in row 7 and you want to have them disabled in row 6 because they are not supported or not so think when you add new option think about the configurable options lvm has also very extensive runtime configuration file and we recently even scaled up massively this thing so we can generate configuration files for a given version do a diffs between versions of the runtime so these options you provide in the build time you should let them configurable also in the runtime so when you install your package and you want to disable or enable features they should be all that should be all doable in runtime so the thing that developers often forget about is documenting and testing we also sometimes fail to do it sometimes new features flow in such a big flow that some pieces are missing but usually try to catch up so don't forget whenever you add something really meaningful to documentation and also the feature which is not tested it doesn't exist because if you work in the team where multiple developers are doing developing and the features are not tested then it often happens that one coder commits something and breaks something else and when it's untested it's lost and forgotten because if you have like thousands of features you don't have time to play with all the features so testing is important part example from ldm when we added for example feature where we switched snapshots from the old style to using the team provisioning you may select the type at the configure time and you may also select it at the runtime so user is not satisfied with the new defaults he has still the chance to go back and use the old one and if we figure out that we have a mistake in the new feature we can also suggest a very simple bug fix for the user the bug fix in terms that if the user had a previous functionality working and now we break something like we caused serious regression in some performance for example we can just tell him ok we will try to work on that and meanwhile you can disable the new feature and go back to your old previous settings so now testing when you add a new API functionality and you are not quite sure if that's going to stay that way you should mark the API as unstable and you should try to resolve this rather early so the users are not seeing for years this API is unstable so it's fine to release the unstable API it's fine to change it but you should not keep it forever in the state and the versioning makes even possible to if this unstable API is not so much broken you can still keep it in a shape in terms of those version symbols so even if you change the API later you still may keep the combat with it in some way whenever you do a commit especially if you introduce a big feature do not forget about the test cases also problem in LVM not always this is true but we try and that's as I said if you have a lot of features and those are interacting with each other you may easily lose them what is not tested often appears as a bug with 3, 4, 5 years regression so that's why we occasionally have to go back in quite a big history to see when it got broken and how to fix it but I personally do not like very much is that the you really should not release the stuff that you have not tested I often see especially in Rohite that it seems like the package maintenance do not even test what they have built so please if you try to do compatible software think about your users they are not your lab rats really you should check it yourself and enough about that and and let's see what I have here from LVM now it's there are many other projects how to test your stuff we in LVM team developed our solution somehow in-house we used the billboard with each commit we basically test which is virtualizing the whole usage so we collect test cases we try to add more and more of them and we also try to use more machine types like we check on browse 6 browse 5, browse 7 even some LVM other thing is that we do use the static analysis we find the reports from tools like cavity and so on very useful also prevents you breaking the code so let me demonstrate the tests we run so how it looks like this is basically a list of machines could be extended a little bit more easily and every commit causes the whole build of the whole project so by this we catch that we are compilable on many different distributions it also we also try to use slightly different options and configure settings on those distros to check for different pieces because there are many combinations so we even had some other testing machines but they are currently disabled so as you can see some machines are slower some are faster some are even showing that there are bugs which are at this moment it's the row height machine and our test so we catch the mistake which made the author of SCSI debug mode driver where he basically switched the sectors to bytes I think so it's now reporting different numbers and our test is catching that we are quite surprised that the author itself has actually not tested this thing there are many other sign-ons on this commit which seemed like it has been verified by I don't know five other people did not do the same so as you can see we catch that mistake I think it will be soon prepared so so the other noticeable thing in our output which we also have several modes of running like in terms of we have like virtual mode of DM and we have also a full setup of machines so we provide complete RPM packaging and we provide single testing and the advantage of this is that we can lock on the machine and simply check what happened there the other thing our test is doing is we also detect the time of test so we can check for speed regressions and other things like that and I think it's enough and now it's time for your questions if you have some okay well as I said if you have a project which already has an existing API you cannot throw it out and build a new product yes yes but as I said the correct way of course is to return the structure from the library so it's obvious that but if you have already an existing API which has been designed like well 15 years ago you cannot just you can throw it away and you can build a new library but again that's what we try to prevent we want to have one stable library which users may rely on use it and we solve it differently and it's usable so that's what the presentation was about of course we know of course of course it makes a hit but my point is that if you even have all the library you can still provide the compatible interface GLC and provide the support on GCC and everyone should check the deeper presentation there's a lot of things explained in big detail so that's so another question okay go on one of the experience what is the work across the system which is actually great so now when you do have this experience when it's going to be LVM library ready? yeah so the question is that if when we do libdm or libdm library when we do libdm library the question is LVM library will not exist in terms of you will be able to link the library with your code but we are now focusing on providing the like deepass API so the issue I think I do not want to bother other so I can explain why there cannot be link library because there are many memory restrictions and other restrictions that your tool cannot handle especially if it's in Python and stuff basic story is that we locked the memory to the application into the memory and we really do not want to lock 300 megabytes of memory because of Python so okay another question yeah well yeah that's what we get when we run the package whatever the tool is when you build the rpm packages you get a list of differences and you have to accept them so wave them or whatever so there are tools which are checking for for that but on the other side we are not extending our library so heavily that we would like up to any 50 symbols per release so it's relatively low of new news into the library so we can do it by hand and there are 3-5 people watching the comments so that's basically it I understand that if you would have a project which generates a huge API changes every month then it's a problem but in our case it's like few functions few functions per month so there was a bigger change recently actually when we added DM stats into the library but other than that it was it's if you I can show you just here as you can see only the version 104 has a bigger bigger set of changes all others are just one, two functions all the time so you can ask me after presentation it's ok I will show you how we are going to do it it's wrapper you won't be there, right? well, if available it would be better I don't even see how to write compatibility users