 So, hi, my name is Evake, I work for Collabra, I've probably seen a couple of my colleagues talk here already before and Today I'm going to talk about something that may not mean very much to you, which I'm calling segregated dynamic linking Because naming things is hard and this will do until somebody comes up with something better Okay, so let's jump into it What is it? You may be wondering So let's start with a statement of the problem that we're trying to solve These days a lot of applications are containerized. I'm using that term kind of loosely. They're in some kind of Rapper, they have libraries that come with them The libraries come from the runtime rather than from the host It's just a way of improving portability and stability because you know We all know most people here know about the kind of portability problems we have. Well, that's kind of mostly true It's a little bit of a lie Some of them still need to come from the host Notably your graphics stuff. It's quite closely tied to the hardware There's no reasonable way of shipping it in a given runtime and still having it work with Whatever hardware you end up having to work with Which is all fine for the most part, you know, you can sort of do that But then the problem which we were trying to solve in the first place with our containerized applications Which is that the host and runtime libraries may be incompatible This is kind of a really very primitive sketch of what the problem might look like You can see that we've got a dependency down there And we've got something in the runtime that uses it the main executable uses it and Unfortunately, we also need to have this library from the host probably libgl or something similar It's incompatible It's not going to run everyone's really unhappy because we haven't solved the problem that we set out to solve So what could a solution look like? Well, we kind of want to have segregated linking that we want the host library that needs its compatible version or Incompatible version of the library linked to it But we want the rest of everything else not to see that, you know We want to be generally unaware that there's another incompatible version of this dependency over here And for those of you who are familiar with how linking works So how many people is that by the way, just so I know how many people I'm lying to that's a fair few Okay, so if you spot me telling any lies don't don't tell anyone else. Let's keep the illusion of competence alive So you'll know that this isn't how linking on elf works I mean there are other systems which in which link is a tree and that has its own problems But in general we have one copy of the library and everything uses it Which is kind of the problem that shared library set out to solve in the first place so What are our objectives here? We want to expose only the library that that we we are pulling in so We want to expose lib host, but we don't want to expose this copy here, which is incompatible It as it's a dependency. It's the thing. We're trying to protect ourselves from we don't want to see it We don't want to have to change the code because if we had to change the code of our application or our libraries That would kind of defeat the point, you know, we we could say to people Hey, you're gonna have to rewrite your programs to make them all portable. Well, that's been tried several times We all know how that goes. It doesn't happen We don't want to have to recompile the applications again same thing It's just going to be nonsensical if we say and then you have to rebuild whatever the large application is that you wanted to run and We were very much like for there to be no performance hit. So We also want it to be transparent and it should happen When someone loads their containerized application or game or whatever It should happen pretty much transparently the user should not know anything is happening here and just to make life easy for ourselves We want this to be done with minimal manual intervention, you know Like we want this to be as frictionless as possible because anytime you put a barrier in an application developers way They won't do it with reason, you know, we're asking them to do something and they should say well I shouldn't have to worry about this. It's not my problem. It's yours So well, let's go through this and we'll see how many of these we can achieve So what are the pieces of the puzzle that we're going to need for this? First of all private dependencies a library should have isolated dependencies Normally, there's a single link list effectively of the dependencies in your in your program space Everything will come from there You'll get anything that you mentioned the name of as as your dependency you will see symbols from that and it's just one flat list There's no Segregation going on there. So we need to kind of defeat that somehow worst-case scenario We're going to have to rewrite the linker. I really really don't want to do that. So let's hope I don't have to And the answer turns out to be DLM open So this is a new library call that was introduced in G Lib C in around I'm gonna say 2 not 19 I think it was sort of it sort of existed before then but that's sort of the first version that Works without immediately exploding in most cases It's it's not quite there yet. It's a work in progress patches are still being fed upstream But from then from then on it kind of works and what does it do? It is like DL open it adds a library after you've loaded up to your link map But it can also create a new secondary or tertiary link map and add libraries to that instead It sounds kind of promising. It's it's basically what we ordered for this and Just like DL open it automatically loads dependencies So we have a problem in that we want to pick the right libraries like if you remember from back We had two copies of the dependency. We want to be able to pick the right one now by default It's going to go to your standard loading path and it's going to load the one that's coming from your runtime And your container with your containerized application See libraries are picked from the search path. We kind of need to defeat that Because we will still get the wrong copy for at least one side of our link chain if if we just let it do its thing So we want isolated libraries to come from the non-standard search path We must divert searches for dependencies of isolated libraries. Can we do that? Yeah, sort of the linker loads all listed dependencies But it won't reload items that are already in the link map. It's targeting So if the library is already there it looks at your dependencies and said you asked for libfoo I have something called libfoo. My work here is done moving on to the next one. So if We load the libraries for the isolated library by their full path not just by their SO name not just by libfoo.so We can say, you know run host whatever wherever we've exposed the host libraries We do them in reverse dependency order. So we walk the dependencies backwards We say oh libhost needs libdep. I will go and get that from wherever I've mapped in the host libraries Then by the time the linker gets to it it won't go and search the standard search path It'll look in the link map that it's loading to and it'll say I already have that my work here is done I'll get my symbols for you from here. We can move on so Yeah, we can do it we can get what we wanted well sort of we're part way there We haven't had to rewrite the linker which is always a plus in my book But what we've achieved is we've isolated the DLM open symbols completely They're not available by default to anything So we still haven't met our goal of all of this being frictionless and seamless So we need a caller of our library that's coming from the host to get those symbols automatically And in order to do that we need to understand how dynamic library calls work So we're going to do a little bit of a dive into that now don't worry about it because we're going to go into some detail But at the end of it what we will have figured out is that we don't actually need to know any of that detail because it sort Of works the way we need it to Which is always nice So this is what jumping to a foreign function call of that a call from another elf object a library in this case Looks like we start by pushing our arguments for the call onto the stack and And then we look into something called the procedure linkage table So each foreign function that you have has an entry in the procedure linkage table and we look up its address we read it back from something called a relocation record and The things that are important to note here are the red and yellow sections are read only they can't be touched once They've been loaded. It's only the it's only the blue columns the stack and the rr that are written to Everything else is read only which sort of brings us some nice features of shared libraries For example, they can be shared and between processes Without any problems So this is the first step of calling a function. So we've got an address from the relocation record and Then what happens is that? The first time you call a function you don't actually get the address of the function you get something called the fix-up code The So the fix-up code pointed by the relocation record asks the linker for the real address The linker searches the core the dependencies of the calling object For sort of where it should look for the symbol and then it writes that address back into the relocation record So this this chunk here in green. So this is this blue this happens every time you call a foreign function This happens only the first time you call it And then this happens every other time we read we've read the record back from the relocation record We jump to the text The function does whatever it needs to do Whatever it was written to do The return value is pushed onto the stack and then we jump back to the caller So if we scribble on that relocation record before this first call ever happens The PLT fix-up will never be invoked. So we sort of bypass the normal symbol resolution entirely. It's up to us where this goes The linker never resolves the symbol address. We have total control over where the function core goes This is sort of the thing that a lot of people writing malware use to redirect your functions Thing is they have a rather nice feature in that they need to own it They only need to work about maybe one time out of a million and they've got a successful attack We need this to work every time, you know, which kind of makes things harder for us, you know, their amateurs basically The this is an important bit at no point did we need to know the signature of the function we were calling the caller Puts the arguments onto the stack in the expected way That was handled by the compiler because it had the signatures when the program was compiled The callee pulled the arguments off the stack. It knew what it was expecting But even though we're diverting the call to a completely different location We didn't actually need to know the details. It was taken over just before and just after we did anything So we didn't need to know that which is good because it makes it much easier for us to automate this whole process That's a huge amount less that we need to know. We don't need to know the arguments We don't need to know how they are laid out on the stack Or we need or we need to know which is guaranteed for us is that they are there on the stack when the call happens So the key question is can we find that relocation record and scribble on it reliably? And the answer luckily for me again, no need to relay the linker is yes, we can The link map entry has a pointer to the elf data for each library and LibElf can interrogate this So we've added one dependency LibElf, but I'm pretty sure that's not going to change incompatibly anytime soon so Great we can now put these pieces together We can make a shim library with the target library's SO name so that the linker when the linker loads it And we put it on the search path. So for example, if I want to load LibGL There's no actual LibGL inside my runtime in some with my containerized application, but I can put a shim LibGL there Which will be loaded by the linker And during the init phase of that shim library, I can deal them open the real library and all of its dependencies in reverse order as I've said I can search the alternate library path to do it So I know I'm getting the host one because I want the one that didn't come with dependencies that didn't come with my runtime And then I walk the link map. I'm making this sentence do a lot of work here I walk the link map and scribble on all the relevant relocation records so that All of the functions in my binary instead of going to The plt fix up and then finding the location of the stub function in my shim library They'll actually go to the real one They don't ever realize they've been diverted because I've bypassed the normal symbol lookup code So yeah, kind of we can basically do this now. Obviously I've simplified a lot. There are some details Some problems The self-shaving yet has not yet been invented So I'm just going to establish a couple of terms up front. I've called an isolated set of libraries a capsule There may be a better name naming things as hard If you can think of a better name by all means, let me know And then I've assumed that they come from file system mounted at slash host So within my charoot or container or runtime or bubble wrapped environment or whatever I have mapped in the actual host file system read only at slash host For example, it could actually be anywhere Other problems deal open can't be called from inside a deal. I'm open namespace It causes the c library to explode which is not fun for anyone So I need to use this relocation record scribbling mechanism, which I've already established To replace the deal open inside a capsule With a wrapper that does the following it calls dlm open instead of dl open It does the path remapping and simulink resolution relative to slash host or wherever I've told it The extra light the foreign libraries are located And it doesn't accept rtld global because dlm open Doesn't yet have semantics for that. I'm discussing that with glibc upstream when we've hashed that out It will but for the time being I need to mask that The reason I need to mask that is because Libgl does the It it deal opens itself rtld global to expose symbols to its plugins, which was fun and games when I figured it out But until I did was a puzzling crash as far as I was concerned What else okay dl sim, which is how you get symbols from a dl open library now has to Have a split personality needs a little bit of extra work. Remember we want this all to be seamless But now there are two namespaces not one And that means that when something in our application calls dl sim I need to transparently go ah You may have meant this namespace over here and look in there But I only want it to find the symbols of the directly targeted library I don't want to look at any dependencies. So I need to be quite careful So again, similarly, I will scribble on the relocation record for dl sim And replace it with a wrapper that does these kind of safety checks for me Another interesting thing so the the usage of libgl which was the primary use case Varies quite a lot some things link statically against it some things dl open libgl There's a whole bunch of ways you can use it. So anytime another library That's not necessarily one that I've exposed is dl opened I need to scribble on its relocation records as well Which means that I similarly have to wrap dl open outside the capsule In a you know in the same way so that anytime something is dl opened I scribble on its relocation record. So if it needs I for example a libgl function It gets the right one Uh extra problems Currently each this is kind of a lie in my demo because this is fixed But the patches have not yet been reviewed and accepted upstream. So currently each namespaces its own glibc copy Which works a lot better than you might think in that it actually works at all Um, but it does make anything to do with threads quite It's not actually a deadlock, but it looks like a deadlock deadlock prone Alloc and free is interesting, of course As long as all of your allocation of freeing of a particular pointer happens on the same side of the capsule boundary You're okay. It works. It's fine If you pass a pointer to across the boundary and expect it to be freed On the other side and you've got two glibcs Then yeah glibc immediately throws a hissy fit and says what are you asking to me to free? I've never seen that pointer in my life Corruption pull the ripcord bail out immediately. Uh, we have a disaster on our hands So yeah, there is going to be an extra flag called rtld shared Which is currently being reviewed and what that will do is allow me to put sort of the fundamental libraries like the glibc cluster Have the same instance on both sides It's building on work that's there in in glibc anyway because as you can imagine there can only be one copy of the loader So it it's sort of half already does this but it needs to know to do it For glibc as well. And that's kind of one of the things that I'm feeding upstream So the workaround I had before was to replace the alloc and free clusters inside the capsule Which kind of in a hackish sort of way tried to dispatch The pointers to the right place to be freed It doesn't really come up much when using libgl because libgl doesn't really Allow allocation to pass across the boundary. It's api, but for other libraries it it might be relevant Right so and finally does it actually work. I'm going to postage the fortune here. I'm going to attempt a live demo So these things have all been tried And worked. I'm going to try glx gears and then if that works, I'm going to try open arena so Where are we going to this? So this is uh This is a cheroot which has had the libgl surgically removed from it And I'm going to try and launch glx gears you should see some spew from the capsule Which is where you can see it's mapped in the different things and we've got a glx running And you can see there it says oh DL open wrapper cannot pass our tld global. So I know I'm intercepting these calls Um I'll still where's my pointer gone someone see my pointer. Oh, there it is So if we page up a bit You can see I've got this is the default namespace. You can see I've got some shim libraries mapped in here Which are stubs. They don't contain any actual code. They're just a placeholder So that the linker can map things in and then I can scribble on the relocation records later and then if I go up a bit There we go We can see there's a capsule there which has host libraries mapped in which is where the real libraries reside And these are isolated from the from glx gears Except where I've redirected them with relocation scribbling So that was one test now. That's a relatively simple test because It has libgl linked in not statically linked in but linked in Now open arena uses sdl Which is going to deal a deal open libgl Which is kind of a more complicated case because things are going to happen much later if I'm really lucky this will work And I've got about 50 percent with us for this demo. So let's see if we can get get that percentage up a little higher and Let's put the window here. Why is it the window here? Let me I don't know if I can move the open arena window Up dammit. Let's put it on the wrong screen Maybe I can just turn this around demos they never work, right? Okay, it's tiny but you should be able to see Open arena actually running with a diverted libgl Uh, okay. I think that's Kind of the end of the talk So, uh, the code is there. It's all kind of published already. Uh, this talk as well if you're into that sort of thing is available there Any questions? I didn't understand then what makes the Like your open arena demo only 50 percent reliable Oh, uh, that was it was kind of unrelated to this. Um, it was a Pulse audio not being happy running in a charoot problem, which I eventually tracked down, but uh, yeah As I it's one of those things it breaks and quite often because there's a lot of moving parts So a colleague of mine, uh, so this work was originally well and still is sponsored by valve Because they want to improve game stability is one of the things and so a colleague of mine Simon Gitti is working on the other side of this which is Assembling all of the right libraries within the runtime sort of hybridizing your runtime libraries With the bits from the host that you need You need to pick the newest libc of the ones that are available And a couple of other things so he's working on that side of things, but those pieces don't exist yet So I kind of had to hack it together by hand, which is why it was unreliable There wasn't anything intrinsic to the project So I would like to ask if you have support for at the RT deep link to go to the next symbol in the chain Sorry to the support for deep linking when you load the symbol from the next library in the chain Uh, no, I don't think we do. Um, it is pretty much just gotten to this stage. So not yet I'm going in a different direction. Sure. Uh, could you use this to make, um Um symbols like Soft linking where you you resolve a function if the library is installed it is the library and if it is not installed just returns I don't know an error Yeah, yeah, um, what happens when this one is actually I didn't download it because my auto generated stubs direct everything but Yeah, if you uh, you can have uh, what would happen there if a symbol wasn't available And you called it you would immediately get a backtrace saying you called a symbol that wasn't available in the target library Here's the backtrace where you called it from here. So, yes Let's answer your question Yeah, but I don't want to get a backtrace. I want to uh So for example, I have a program and the program links to five different compression libraries Sure And then if one of them is not installed I would just want to say no cannot decompress this, you know, like try a different one or something like that Yeah, yeah, you could do that. Okay I was wondering if you can speak a little about uh, what uh, volv is planning to do Longer term with this where they're going. Okay, sure I mean the idea is is I mean pretty much as I laid out the beginning So, you know, you you need libgl or in the future you'll need your um, your wayland drivers to come Vulcan drivers Sorry to come from the actual system because they're the thing that's closely tied to your hardware And there's there's no real sane way to put those cram all the different ones into a runtime You know, they would quite like a game that you bought 10 or whatever number of years ago to keep running even though you're on a completely different system So the idea is that the game will come tied to a runtime Uh a version like pretty much similar to way things like flatpack work, you know You've got your application in the runtime. You'll have a game and it will specify I want this flavor of runtime at this version Um, but then you need to parachute in the most recent libc out of the two that's available And you need to parachute in the graphics drive the stuff that's tied to the hardware needs to be brought in from the host And so that's what it's going to be for and the idea is that When my colleagues work is done all of that will sort of be assembled into your sort of hybrid runtime So hopefully we'll the idea is that in the glorious flying car future We will improve game stability a lot, right? If you have an old console you can plug a gaming that you bought x number of years ago and it will work Now they have the advantage that it's all baked into the hardware. So they don't really have to deal with things changing, but They would really like That to be the case because what their thing is that they want to sell you stuff electronically And they want you to have the confidence that whatever you've paid for it goes on working. So that's the goal here Sounds great. Is there an ETA? There there sort of is but it's tied to other things that I can't talk about so okay. Thank you So what if there is a different system that uses the same Mac for example a heap profiler that wants to replace all melox in in your entire program Uh, what what if there is uh, the uh, I don't I don't know what would happen It might work. It would depend on the order in which things happened Um, is that was that we've saying would they clash or? Yeah, I mean can those interact is there a way to make it work? I I don't know the answer to that. I'd have to try it I wonder if you could talk a little about the surface area of gl that's relevant to change because i'm In my memory, there's like a couple of apis where you put uniform in and then you like hunk a giant String of program text and that goes across a terrifying barrier into the drivers Okay, and then like But that is like a function that says int and like gives you a void star and stuff And so it doesn't really know much versioning to be honest. I don't actually know that much about how gl worked I mean there was a possibility that I'd need to learn it to do this but I kind of got to the point where it worked and then I just stopped looking because That's you know as as as you may know, there's there's a whole number of different ways of using libgl There's the simple glx gear's approach of just linking against it. There's the sdl approach of gl opening it and finding the functions There's the one where you get a specific function from gl and then you ask it for the function pointers That you that you want to call but they all sort of work with this approach because it's only the small set of well-known functions That people can call that need to be proxied in this way and everything else just kind of works amazingly enough Okay, uh for the um dynamically linked glx gears, can you just show what It looked like when you ran through ldd. Oh, yeah, sure Don't seem to be able to get focus back. Where's he gone? Oh, right. Sorry giant screen. Let me quit that Oh, there we go So you can see libgl is going to a shim Um, if we looked in that shim, we would just find stub functions with no actual code in them Well, there's a little bit of code This code that generates a backtrace if you ever actually managed to call those functions So the the search path the shim search path you added to the containers ld conf Yeah, oh, okay Yes, um That's gonna work from here Oh, isn't it? Oh, okay. I'll I'll make make it public You're the first person to actually ask so Uh, yeah, okay any other questions? Okay, um, I saw that you had So why then here is uh, so you see like the fit the fit to last line there Why is ldd saying that glx gears which you didn't have to recompile or anything? Why is it dependent on lib capsule now? Let's come from the shim The shim depends on lib capsule. So lib capsule is kind of rather than baking the code into every single shim They link against lib capsule, which does that Okay, um, that seems to be old questions. So we're there it is segregated dynamic linking