 Okay so hello everyone thanks for coming and thanks for having me here. Today I want to share some experience about building LLVM based tools. Yeah just few words about myself. So my name is Alex. I work at a company called PT Scientist JMBH. You can find me on the internet if you want to like my blog or Twitter. So yeah this is the agenda for today's talk so basically these are just like sets of some somehow like random but generally enough tips and tricks that might be helpful if you want to build something based on LLVM. So the tool I'm talking about is called MAL. I build it with some friends and colleagues like for several years like three years something. The source code is available on GitHub. If you want to know more about the tool itself then I encourage you to watch my talk from FOSDEM two years ago. Yeah besides that I'm not gonna talk about the tool so everything will be kind of general. So a few interesting things in my opinion that the tool works in Linux, FreeBSD and MacOS. It may work in Windows but just no one has access to Windows machine so and the nice part also that it can be compiled against any LLVM version between like yeah in between of three nine and seven. Yeah and another interesting part in my opinion is I think it has quite quite it provides quite nice user experience. So here I have just normal program. Yeah it runs like some tests in this case but then I can also do the following. Oops sorry. Yeah I just run my tool against that binary and it works and it operates on the bitcode level so I find it quite quite nice comparing to some other tools that I've seen. But let's get back to the slides and begin. So just to have some framework in mind I defined LLVM based tool as something that operates on LLVM bitcode and it does the like the following things. It parses or loads the bitcode files in memory like creating LLVM modules then it does some analysis and then some transformation potentially and in the end it does some some like processing. So the tool can be something like compiler or a foskator or like in our case tool for mutation testing. So I think this approach is general enough. Of course there might be examples where it doesn't fit but I just want to have some some kind of structure. So the first thing that you should consider of course is the build system and I think lots of articles and documentation like suggest to use LLVM config. It may work if you just start something fast or like some prototype but in the long run it's not not really good and yeah there are a few quirks still. So this is how like usually looks like. So you just use a clank then you call LLVM config you ask for C++ flags and linker flags and so on. So the first problem the C++ flags so it's these are likely not the flags you want but these are the flags that LLVM was compiled with. So here the first one it just yeah just some part of it just an example include yours valid you need it obviously but for example this dash W error I think it won't compile with GCC because it's some clank specific thing and it may also not compile with earlier with older versions of clank. So yeah another point that it's unless you compile LLVM yourself you will likely get optimized version. So if you build your tool using this like C++ flags then it will be compiled like in optimized way all the time. So yeah just you should just be aware of that. Another thing that the LLVM config for libraries and linker flags it's yeah it works really good really well but there are some other issue. If let's say you have on your machine installed more than one version of LLVM and you have flip LLVM dilip then eventually you may get this problem. So if you compile against if you link against like some library specific library like LLVM support for example and you run you may get into this like inconsistency blah blah blah. I don't really understand where it comes from there like something goes wrong obviously with dynamic linking and maybe static linking as well but the solution for that problem is to actually link against just lip LLVM. The kind of issue of a LLVM config that there is not really an easy way to check whether you have this lip LLVM installed or you don't have it. So yeah as a solution I do consider CMake. If you don't know what like well you probably know what CMake is but if you don't know how it works and what it does I encourage you to just read this documentation. It's good enough introduction and from now on I assume that you understand CMake. If not please do yourself favor read about it and then you can get back to slides and yeah it may make sense afterwards. So how do you say LLVM with CMake? Yeah that's straightforward so you just like use the find package and you search for LLVM. When it's done you get access to all the variables that LLVM distribution provides to you and also the libraries. So important part here the you may see in kind of red color search pass they are specified in this way. The reason because as we tend to support many different versions of Clang and LLVM different versions on different distributions of Linux on Mac they have different locations of like different places where CMake looks for. So the recommendation is yeah to do this way. Two other parts pass to LLVM that must be provided from the outside by your user of your tool because if you don't do it explicitly then there is no guarantee like which actually version of LLVM is used and it's getting a bit tricky. The good part that you can also do it the same with Clang find package Clang but it's broken on some Linux distributions especially if you have I think Ubuntu if you have 394 and 5 installed at the same time then it doesn't work because yes some prefixes suffixes with versions just in that case yeah recommendation is to use pre-compiled versions of Clang or LLVM from official website. Yeah so once you've done that you get access yeah as I said to variables like LLVM includes yours and you can teach your targets like where to look for headers and you can link against LLVM support for example as you would link to against any other library in your project except that yeah again you may face this problem and it's really annoying. So the proper solution is to actually check if distribution of LLVM you use if it supplies this LLVM target library then you link against just LLVM otherwise you link against your libraries like yeah any any you want. Okay I think we're done with CMake for now and getting to well yeah okay we're not quite done with CMake so how to support server versions there are two approaches this is the one of them so basically we have just bunch of folders they provide a set of kind of glue code for different versions of LLVM and then yeah the header files they look like this so basically all the places that have some incompatibilities between versions they were hidden behind some functions and also like type depths used and so on it's not much here actually I must admit and you probably may face many more discrepancies between versions but it's I think it's still manageable and it's good enough approach so in fact in our case we also used ORGJIT and it was changing a lot between different versions like a lot and we first tried to come up with some interface to hide the changes behind but yeah at the end we just realized that our use case well that ORGJIT is too general for us and the maintaining this part is just harder than actually crafting ours like our own ORGJIT engine which we did and yeah it just works so far yeah I think the other approach is to use the define like the macros if like in your code base like sharp if LLVM version less than five or something yeah it's still doable but it's a bit messy and yeah I think just as a personal preference I like this approach more yeah in the CMake it looks like this then yeah the real code is a bit more sophisticated but it's just an example taken from the from our CMake file yeah just checks if there is a kind of glue code for the version then it picks it up otherwise yeah error yeah so far I was talking only about case where you compile your tool against pre-compiled version of a LLVM but it does make sense also to compile again sources for the following reasons with pre-compiled version you have like really fast compile time but it's really hard to debug because you cannot dive like deep deep into internals and you only have interface so it's hard to put print apps for for sake of debugging and there are no asserts unless again unless you compile a LLVM yourself and so for example we didn't compile with a source for maybe a year or something and then we've got several bug reports from a person who was adding free BSD support and this person used yeah built with assertions enabled and we just learned that we violated many times we violated some constraints of a LLVM and if you enable assertions then it yeah it crashes but it worked in our case somehow like rather by accident but still yeah so it's a downside of built building against source code yeah you have lots of time to actually build a LLVM so the good part that you can also set up your CMake to support both things at the same time so either compile against pre-compiled version or against sources so the trick is simple it's probably for Gile but it works and yeah I'm glad about that but yeah so basically you just check if the pass if the LLVM folder that a user provides if it contains CMakeless file then you just add LLVM itself as a sub-project so now you own a LLVM and know the other way around the problem here that you don't have things like LLVM includes years and you don't know a LLVM version because those files they are not exposed like these variables are not exposed again there are some solution for that another for Gile solution so you can pick any random LLVM library and read its property for example include directories and yeah it works as well so as for the version yeah you can just parse the CMakeless file and use regular expressions to get the version out of it then yeah it just works yeah and I think now we are done with CMake so now yeah I want to talk about memory management I'm not sure yeah is there any problem here in this in this code anyone knows okay so it may crash or may not it depends on which version you use I think three eight and four it will crash and on the recent versions it it will work I think so the problem is that yeah or rather let's start with the solution you should just switch those two lines the problem here that LLVM context actually owns the modules so and it stores just rough pointers like vector of rough pointers to all the modules that were created and in this example yeah the context deallocates yeah the LLVM context owns the modules and in the destructor it just calls the delete to the actual pointers so this code when it executes LLVM context it destroys like deallocates the modules then happens deallocation of the vector with all the modules and yeah double free so basically you get you get a crash for the same reason this example so this is yeah I think we had this problem before so assumption was that in the nested scope you create a module you do something with it and it's just like disposed like deallocated after the scope ends but we faced a problem on a machine 32-bit machine with four gigabytes of RAM because actually modules are not deallocated and they are stored in the LLVM context so the solution here is to use some local context then your memory is freed at the end of the of the scope another kind of annoying part was was this approach all the modules they're loaded in this the global context and when the program terminates you just see that like nothing is happening and it's because the LLVM just frees the memory so in that approach this work is still done it's still happening but it's kind of distributed across the whole program life cycle so it's not yeah the execution times time takes at the same time but it's just less less annoying or frustrating okay next topic is parallelization so the trick here to to make things parallelizable is to not use LLVM passes it's bold statement but I will elaborate just in a second so I know like at least at least two classes at in LLVM they are not that are not trade safe and that are heavily used it's a LLVM context and therefore like anything you do with modules instructions functions it's it also not trade safe because it's like tightly coupled with LLVM context and the same goes for the target machine so if you use like or compiler for example or if you do any code generation then it's not trade safe so let's look at the example yeah this this tooling is just perfect for fair parallelization because what what it means that you have like ten tasks and you have ten threads and you can just even this split those tasks across threads and you get yeah you gain some performance so just as an example let's say we have three bitcode modules we want to load them into memory parse and see I do some job so if you want to do that across two threads then each thread should have its own LLVM context otherwise it's not trade safe and what happens next if you want to do some analysis so in this example we have six functions it may make sense to split all those six functions across two threads because you just do analysis you do not do any modifications and you can actually do that yeah it works so the there is a kind of bold line where a function escapes its thread boundaries and it's fine but when it comes to transformation then you cannot do that anymore so each function should be transformed in the context or in the thread of the same module of the same LLVM context and these are some numbers so yeah you see the like the steps are split analysis and transformation so the transformation at least in our case it's super fast because yeah for some reason it's super fast because it does much less job job but analysis takes like most of the time and if we parallelize like you can see the green color it becomes just much faster so that's that's why I'm saying that you should you maybe should avoid using passes because they kind of incentivize you to put a transform like analysis and transformation into the same thing so yeah if you split these steps then you can parallelize better but of course you should measure everything all the time so this is yeah in our example in our case just loading like parsing bitcode files loading them from disk takes much much more time than analysis and transformation it doesn't matter like sequential execution parallel so yeah you still can use passes if that's the case for you but yeah getting bitcode that's I think yeah one of the trickiest topic one of the kind of selling point of a LLVM that it has the bitcode that can be shared across different compilers and so on that's still the case but yeah if you want to build some tooling there is not really easy way to get this bitcode depending on the compiler and so on but there are some ways to do it for clank at least yeah for other compilers languages yeah please talk to to developers of those languages so normally like compilation flow is is like that you have an object file so you call a compiler with some flex you have you have an object file then you call a linker and you get executable in the end so with clank there is at least yeah the most famous flag so if you call clank emitter LLVM and instead of object file you get a LLVM bitcode and there is no executable another one LTO so you can call LTO like dash FLTO and you get instead of the object files well you still get object files but they do not contain machine code but they rather contain a LLVM bitcode and at the end you have normal executable so that that works pretty well if you have some well-defined build system like CMake for example or even like make files if you have those intermediate files but if you just call clank and like 10 source files then there is no output of bitcode or no output with object files then it doesn't work and the best approach that actually we use where we adopted recently the F and bat bitcode so if you call this code then at any stage you're either object files or executables they do contain both machine code and LLVM bitcode and you can actually extract it easily so there is a tool called libEBC I think it stands for embedded bitcode so what it does it just yeah extracts the bitcode file from an executable yeah works Boston Linux and Mac and FreeBSD and I yeah the tool the original tool works it just does extract the files and stores them locally on disk so that you can do some post-processing on them and I made a fork to make it work was in memory so everything happens in memory as yeah as I showed in at the very beginning so yeah super straightforward I will be upstreaming the patches as soon so the original version should get the support if yeah if we agree with maintainers but otherwise you can yeah you can check the things out oh yeah yeah these are examples how it works so you just call you see you till and then you get a bunch of bitcode files or yeah you just call the tool I think the last the last point multi-wise support so we don't we are not that many people working on this project and it's really tough to maintain these things so I can just recommend like say two words like vagrant and ansible just like the way to go so what we're grant us it just the system that operates or that helps you manage virtual machines it supports virtual box VMware docker I think yeah anything you can imagine it supports so in that case I have just like one file was that many lines that yeah then I run background up Debian for example and I have virtual machine running ready was everything I need on it so I can just start debugging it right away I can instantiate into it and then when I'm done I just can destroy to dispose the memory and the good part of it that if you have some IDE that supports remote debugging so as for example Selen does have it you can run this set up this virtual machine then connect with Selen and you're like this is it is for example how I work on Mac I do have native experience as a Mac user but I'm running and debugging my software on Linux it's it's very useful yeah the second part is ansible so here you may notice like VM blah blah provision ansible and then a playbook Ubuntu YAML so what playbooks are it just like set of some tasks that needs to be done this is an example of things yeah it just YAML file does something and so for in our case most of the parts they are shared between all the other operating systems like FreeBSD yeah and Linux but it's slightly different on Mac but the cool part about it that I can also run this the ansible thing locally so for example if I get a new Mac machine I just use our Mac like macOS tasks to to to set up things locally yeah what else yeah the other benefit of this the ansible scripts or playbooks that you can also use it like on CI and yeah everywhere you want so at this point yeah I think that's that's pretty much it so I try to to come up with things that are general enough to be applicable for everyone so this is not all the problems or issues we have solved so far but I hope that it will help you to build better tooling around LLVM thank you and yeah yeah question thanks for your talk I found it super useful to kind of pile on the LLVM config hate I noticed you were linking against the clang tooling were you did you try to link against any other parts of clang not LLVM but clang parts and find any issues with like linkage order so the question is if I had any issues with the linkage order when using a LLVM config and yes I decided not to include it into the slides because that that's a mess but we had this problem yeah like with different versions of LLVM and clang there were like some problems with different order of things so we ended up wrapping everything in like start group and group linker flex or on Mac it was like something like dash w like whole archive or something so I mean it just like super hard to support because you have if macOS then you have one bunch of scripts else if Linux other and and so on so forth but with CMake no problems whatsoever just just works yeah yeah thank you yeah so the question is that why don't we use orgjit or mj mcjit and come up with our own solution so one of the problems with that we had with mcjit initially that it requires you to give up ownership of all the modules so for all the modules it's like unique pointer and you need to give up it so you don't own them anymore and in our case we wanted to own it hardly so we cannot use mcjit therefore we switched to orgjit because it didn't have this problem initially but as the time grows like it there was this problem as well and the API changed drastically so it was just too much overhead to actually support our own kind of interface to the orgjet so and we don't need most of the features that it provides like on on request completion and yeah we just lowered program at once and that's it so thank you enjoy the day and yeah see you around