 Hi, everyone. I'm Gary Rensen. G-Benson on IRC. Usually I work for Red Hat on the debugger team, basically on GDB for the past few years. The project I've been working on is called Infinity. You turn the mic up, I'll be talking about that. It's not a hashtag. Yes, I'll speak louder. Sorry, I'll speak louder. First up, what is Infinity? It's a little kind of single sentence. It's a platform independent system for applications to expose some extra functions to software development tools like debuggers, monitors, another analysis tool. I guess throughout this talk I'm going to say Infinity is tool agnostic. It's not aimed at any particular tool, but I'm probably going to say GDB and debugger a lot because that's kind of where I'm coming from. So that's what it is. Why? Why do we need it? Some aspects of some applications need to be kind of understood by the tool that you're using. So for example, you may be working on an application and it's using multiple threads. On your platform, the thread-in functionality is provided by libpthread.so, which is the thread-in library. So any tools that you then use to examine this application, they need to know that if they see libpthread.so, if they see the thread-in library, then your application is using some extra. They need to do this extra bit, which for the thread-in library there's a second library, the thread-debugging library, which then the tool, the debugger or whatever, has to then find a load. It can use these functions then to introspect your application and find out, in this case, the threads that your application is using. In a kind of diagram it looks like this. So you've got your application up there on the left and the tool, the debugger, the monitor is accessing your application somehow and that's the red line. And the tool has loaded the thread-debugging library to see what your application is doing with the thread-in library. So already there's a fair amount of knowledge that we're baking into the, that has to be baked into the tool to make all this work. So any tools that are going to deal with threaded applications here need to know that if they see the thread-in library, then they need to load the thread-debugging library. And they also need to know that if your application is statically linked that it may be threaded anyway, so it may need the thread-debugging library anyway. The tools all need to know where to find the thread-debugging library that relates to the thread-in library that you've loaded. And there's no standard way to do all this. It's just a pattern. So the thread-debugging library is just one example of a debug library. It's probably the most common. But there are other debug libraries that are used to examine runtime linker internals. And the OpenMP, the parallelisation stuff, is either proposed or have written their own debug library. So this kind of stuff has to be, you know, each debug library has to be written kind of individually. And the rules for finding the debug library have to be written. And also for how to check versions like the thread-in library and the thread-debugging library are kind of, the thread-debugging library is looking at internal structures in the thread-in library, so they have to make sure that they're for the same versions or whatever, in the same compiler. And all these issues can be, and they are currently handled, you know, the thread-debugging library works, and it has, however many years it's existed, for most situations. So if I go back to the, this is the graphic we just looked at, so the tool, the application, and the two libraries. But this picture's not complete. There's, in here also is the C library, the system C library. So your application is linked to the C library, the thread-in library is linked to the C library. And your tool is linked to the C library and also the thread-debugging library. And this is fine if you're using like a homogenous environment if, you know, everything's running on your laptop or everything's running on one computer somewhere. But it's not fine in a heterogeneous environment, so if you have different architectures, say, you know, your tool is running on your laptop and your application is on some ARX64 server somewhere, or, and it's also not, it also doesn't work in containers where maybe all this is running on the same computer, but the C library in your application's container is not the same as the C library in your tool container. And in this particular case, the C library is going to look like it's the correct architecture, but it may well be built for a different operating system. And it may well then just, it may just crash your debugger when you load it. So this is kind of, this is kind of why I've started working on this project. Obviously you can make special builds of the debug libraries that, you know, it's built for this thread library, but it's linked again against that thread library. But then you have a kind of matrix of different builds of the debug library that you have to maintain and it kind of gets out of hand. So, okay, so in order to kind of replace LibThreadDB with something else, can I have a look at like what does LibThreadDB do? Well, it has a bunch of functions that it exports to the tool. So in terms of the functions that GDB uses in the thread library, they're all information to get, they're all functions to get information about a specific thread or to, and there's a function that allows it to iterate over all the threads. But unlike a lot of other libraries, LibThreadDB is kind of, it's like double-ended. It exports functions to the tool, but it also imports functions back from the tool. So it's kind of importing amongst the functions for the tool. So the debugger or whatever would say, can I have the, I don't know, the priority of this particular thread and then the thread debugging library then comes back to the tool and says, okay, can I have the contents of this block of memory or can I have this register from this thread? So the thread debugging library is essentially driving the debugger, the tool. It's kind of handing over the steering wheel to the thread debug library just for a little while and then it comes back with whatever the debugger is asked for. And the solution that I've picked, I've kind of been working on for Infinity is to have platform independent functions, the kind of equivalent of what the thread debug library has stored in the actual library. So in LibThread, for instance, if you have your application is using the thread library, the thread library contains the actual thread library code that it usually does and as well it has these little functions that allow a debugger or a tool to call into it and say what is this and it would come back. So this approach of having platform independent functions in the actual, the kind of regular library means that locating the functions is trivial. Like if your application is using this library then the tool can see that here's the actual file, the library that your application is using and here in that file is these infinity functions that I can use. It also means that the tools don't need to know that some particular library is accessed via some other particular debugged library. If you connect your application with a debugger and it finds infinity functions to access certain threading things then it knows that okay your application is maybe threading so I'll maybe call these functions to find out what about it. I'm going to head over myself a little bit. Yeah, sorry. And also this method works nicely with static link and it doesn't really matter where the functions are, where the infinity functions are. So if your library got its threading code from dynamically loading the pfread.so then the debugger can see that. If your application got the threading library because it was statically linked in then as long as the infinity functions have followed it then the debugger can see that. And graphically this looks like that. Yeah, this is kind of that as a graphic. There's no kind of separate library you just have your debug functions in the regular library. So what can this be used for? The obvious first case is thread debugging because that's the one that I've been kind of working on already. Another potential use is for when applications load libraries into different linker name spaces which has been relatively uncommon on Linux at least for up until now maybe because it didn't work properly in G-Lib C but it's been fixed. Yeah, applications basically the debugger would at the moment if a debugger loads if the application loads a library in a different namespace then the GDB at least won't see that. There's no way for the C-Library G-Lib C that has no way to export that. So as far as GDB is concerned your program has gone off into some code that it has no idea what it is. Another potential use I mentioned earlier is OpenMP which has its own debug library to access various things about what it's doing. A-Sanitiser and T-San that address Sanitiser and Thread Sanitiser from Clang. Is it Clang do people say Clang? Yeah, they currently use hacks to calculate things that are not exported and these kind of things where a tool bakes in knowledge of the kind of internals of something else is the kind of basic use case for it. The final potential use I thought so far is pretty printers that G-Lib C has like currently they're written in Python and the actual Python code has two parts. One part is the finding the thing you're trying to print and the other part is actually printing it. So the printing bit is obviously would still stay in Python but the actual bit of finding it and coding all the structure offsets and stuff is really nicely suited for infinity. So what are the components of this? There's a compiler which takes a kind of textual source and turns it into dwarf byte code wrapped in a little chunk that goes inside the binary or shared library and there's also a tester which allows you to write unit tests and it allows you to write basically a unit test and the function itself at the same time which I kind of think is pretty important because these functions are kind of a tool and so like you know the software development tool so that it puts a kind of like a late discovery of error into it where things like the thread debugging functions in G-Lib C will not really going to be used in G-Lib C until you start debugging an application with it so it could potentially have a G-Lib C gets built and shipped and if it has a broken thread debug note then the first thing you're going to find is if some user tries to debug something and the thread debugging is broken so I wanted to make it as easy as possible for people to write complete test cases in the code this ship the other thing is that there's a client library which is written in C and that kind of allows applications to kind of manage and execute the functions and I've got currently a set of replacements for the red db functions that GDB needs there's kind of five of them I think and there's also a lot of, since I last presented this a lot of the questions were basically on the lines of well we'd really like to use this but we'd also like to still have a lib thread db so that we don't have to change anything so I've kind of also now written a, I've kind of been integrating this into G-Lib C's lib thread db so that if you are if you're trying to debug a program using G-Lib C and you load lib thread db like normal you'll, if your lib thread db has infinity support and your G-Lib C has infinity support then you'll use infinity rather than the kind of existing setup and I was just curious, what role does the client library play? Like what sort of? It's like a, it's a byte code interpreter basically but so Like does it find the byte codes for it? Because then don't you need that to kind of call back? It doesn't, yeah there is a call back it doesn't, the client library knows nothing about ELF so I'm kind of anticipating that any program that's, any program that's kind of looking at binos and things has already kind of got some kind of ELF decoder in it so you'll kind of, you kind of provide the client library with the notes it's just the blocks of data that are, you kind of pass it to the encoded functions which are the moment the only place to find them is in ELF notes so Well like you have to dig that out You have to dig that out Yeah yeah I mean it's not, I mean even there's a certain amount of things like I'm going to dig myself in a hole here now because I I mean there's other stuff as well like that you know if you'll say you're a debugger you know accessing this stuff it's not just the actual, those bits of the ELF file that you need you need other stuff that you need to be able to relocate and stuff like that to be able to read the addresses so it's kind of like I kind of thought of it I can't think of a way an application that would use it that has not already got this stuff I can't think the application could could support LibFredDB as it is without already having all this stuff so any kind of other stuff is that extra of this future Yeah and I've mentioned that GB integration is the last part Initially that was kind of the final component and there was no wrapper inside LibFredDB but now the GB integration is kind of into the past version 1.0 or whatever so that's not a current thing Kind of how long does this talk go? Is it still half past? I'm going to skip I've got some code but I'm going to skip it I think You still have 20 minutes I still have 20 minutes All right, I'll make you look at the code Anyway, I'll do it really quick because nobody wants to look at code Basically this is a function in LibFredDB it's getting the address of a symbol it's doing some little special case logic with it and that's what it looks like in C and the infinity version of it is basically the same length it's loading the thing doing the little special case Do I go here? Do I go here? Do I go here? and that's it I'm not talking about that anymore I guess this sort of shows what you will be writing if you're writing notes so it's kind of the kind of general features the functions have a name and a namespace in this case the function has one argument the actual functions are written in Dwarfbyte code which I'm sure you all know what it is it's a kind of stack based language so this function has one argument which means when you arrive at the top of the function the caller has put that argument on the stack and this function here returns two values so your function is expected to return these two values and this is checked by the compiler it's checked by the compiler it's also checked by the client library when you load it but at the moment the compiler checks the stack but it doesn't lay it out because I don't know how to do that I'm hoping that at some point in the future someone else who knows more about compilers than me will be having to write these notes and they'll go, this is easy, I'll do it but not now I'm not going to walk through this function but this just kind of exists this is to point something out this is another thread debugging note to get the address of a TLS block or thing to get some kind of value from third local storage or the address rather and I've put this function in just to point out what this one's doing here is it's calling another infinity function but this function is in the runtime linker so it's kind of to illustrate that they kind of come from multiple sources like at the moment LibThreadDB is accessing runtime linker internals in a way that means that if you were to have a LibP thread from one G LibC with the runtime linker of another with different structure offsets it would fail whereas if you're using infinity it's accessing it through an infinity function in the runtime linker so it kind of opens the door for that kind of thing to be done I'm not sure if there's any demand for it but it means it's kind of safe so when your tool kind of attaches or starts to inspect your process the way I'm doing it in GDB is it's looking at the actual files that you've loaded so when GDB goes through and starts adding the objects of the application that you attached to or whatever it's going to add the runtime linker and then those functions from the runtime linker get given to the client library and then it carries on and it adds the thread library and it adds those functions to the client library so I guess the answer is the client library links them there's kind of a callback mechanism so when you add you kind of give the client library the encoded functions and then it tells you that functions have become available so a function only appears in the debugger when like in this case if you added the encoded form of this function but the runtime linker function wasn't there then this function would not appear in the debugger so the debugger would kind of assume you had no TLS support so the functions arrive at the debugger kind of more piecewise than the debugger would live for a DB it's like the idea is that if it is platform-independent I understand that when we see calls to, for example, underscore a link, map TLS mod ID that looks like relatively specific we will probably not find this if we use gdb to debugger windows with the application this thing here is the name of another infinity function reuse some of the names but this then should map them I don't know, I don't know nothing about windows but the idea is that the set of functions that the debugger sees is kind of like an API so if the debugger sees that you've got this function thread getTLS address then the debugger sees that you've got this function and then you have a TLS address and then you have a TLS address and then you have a TLS address getTLS address then it knows that that concept is there so it doesn't matter how this is implemented like this is the implementation from glibc because a windows implementation would almost certainly look completely different and it maybe wouldn't have the concept of a runtime link or a module ID I don't know does that answer your question? cool this is a test case for one of the functions the way I've been doing the test cases is this implement thing it kind of just like stubs out all the functions that it's going to use then it calls a function and checks that the result is as it is it's a test case so you have a Python wrapper of the client library? yeah so yeah in this Python you're calling the library so yeah the current status of all this lot the node compiler is complete but it needs documentin the node tester is complete but it needs documentin and it needs more tests the client library is complete but it needs documentation and some more tests the client library actually has quite a lot of tests but it doesn't have tests that would validate the specification which in itself I haven't completely finished writing but the specification says that this bytecode should do exactly this and exactly that so I need to write a lot of tests for bytecode that doesn't do that which is not there I have the stuff to create the broken bytecode but I haven't sat there and turned away a right I've got a set of glc the ones that GDB needs they all complete most have tests the API that they export isn't documentin but that's relatively simple I don't know if there's only gdlibc people here but I'm hoping to get a patch mailed by February the 16th that's my date so I'm working on getting this all in fairly soon hopefully the lib thread DB, the integration part of it is there there's a little GDB modification that it needs which is currently native only so that's future work and the GDB integration bit that I mentioned earlier is kind of post 1.0 so it's not done yet How stable is the node format in the bytecode? I'm hoping very stable I don't even remember why I decided to call it infinity but as things progressed I spent a lot of time thinking if something is going to change here in the future how do I make it so that it fails gracefully so that it's hopefully got a lot of built-in stuff so that you can easily say that this is not relevant anymore this is unsupported, it is now supported so that older stuff using newer features can gracefully fail time will tell if I see manage that I'll see if I can find I'll show you a demo but it's not very exciting because it's just it's essentially looks exactly like GDB normally let's see if I can get this come on alright yeah I'm trying to get this to um there we go mindrowing completely blank and I couldn't remember what key hello change colours then that's horrible to see change the colours it's good is one of these going to be black or white? it's mindrow it's there isn't it grey or black black or white can everybody see that okay so there's an example process five seven three so if I open another tab is that going to go back to being small again so if I go tell it to load the should have done this in advance okay there we go this is going to be such an item to climax what did I say it was five seven three there you go it's got a block and we can go in this is a statically linked program with two threads and so it's also got a TLS variable which is so but wait a minute it is loading lib thread DB but that lib thread DB is using infinity it's a big because at the last time I did this talk everybody said this is great but can we actually have a lib thread on our system looking like the old one I'll show you if I show you it with the debugging like tracing switched on type all that stuff in again what was it set lib thread DB search path this actually hurts five seven three you should see a ton of byte codes get executed so there you are there's stuff happening so if I print the TLS variable again you will get some more byte codes not a huge amount you can see it's it's actually quite a complicated where's the start, the start is about there and it's going into this function get TLS base and it's also calling this is all thread debugging stuff but then it goes into the runtime linker notes and starts pulling stuff out from there and yeah print some number and that's it I think the client library is actually running the byte code yes, yeah that's kind of the byte codes column the ones with the x are kind of extra ones that the client library put in they're not actually in the note well no no these ones are in there saying did you see I'm very stuck on the pieces no do you implement the piece what's that? probably not that's probably an answer as no yeah okay I just put all this stuff on the screen and then any further questions I will answer one is does this let you use lib3db from gdb server currently no but it would kind of at the moment the way it's going to work is you load gdb is still going to load lib3db but I the next thing that I have in my kind of sites is the multiple namespace thing and there is no debug library written for that and they don't intend to write one now so once this is all in glibc in their lib3db and in the libp thread my kind of next thing is then to make gdb access to infinity functions by itself without needing lib3db and then it will be able to do it and it will all happen in gdb will it happen in gdb? no it probably will make sense to have it in gdb server as well yeah I think it would be nice to have it not in gdb server if you have a gdb server implementation it's just an added complicated thing it has to do so you have your gdb server and your application creates your new thread to communicate with gdb to handle that or does that? it's just evaluating some little script and getting some stuff gdb can do that if lib3db is implementing internally using infinity there's nothing blocking gdb using a lib3db that then interprets the alcos in the host even if it connects to a gdb server then you just have to load some version of lib3db that's compatible enough with the functions defined by infinity if you load some version that's using some opcodes that's not familiar with the opcodes that are in there if you load a new enough lib3db should be backwards compatible I guess that the kind of ultimate idea is that gdb just won't need lib3db it won't ever pick it up then it becomes a little implementation detail it's just that piece of code that's a bridge between gdb and the infinity library it's either a chunk inside gdb or it's a separate library it's the same thing it's like a lib3db and you copy the new implementation you copy it inside gdb for just thread debugging it's no difference whether it's in lib3db or in gdb but once you start adding extra things like glm open it doesn't make sense to it obviously doesn't live in lib3db it doesn't make sense to me to write a new library because then you're going to expose a few entry points in infinity terms and those will be defined entry points defined in terms of infinity functions and something will need to be aware of these entry points and there will be some C code that will call into some infinity interpreter to run this infinity function and there will be different tools that will want to do the same thing and call it to the same entry points thread debugging both gdb, lib3db and other debuggers they all want to call the same entry points that can be wrapped in a small shim library that exposes like five C functions and you don't see infinity at all it's implemented inside that shim library they can do different things like at the moment all the thread debug stuff is in its own block whereas once debuggers can actually access just a single infinity function then there's no need for to have functions blocked up like that what if you want to just access one single structure do you want to write a whole library for that or would you like your debugger to be able to to call a single function to get that I see it more in terms of glibc is going to expose a set of infinity entry points so you could have a matching library for glibcdb for dldb and that will grow entry points just like the infinity points grow I see that as a viable option maybe I'm wrong I think maybe I need to sit down and talk about it a little bit more comfortably than standing here but like I still have more questions are these notes like in a loadable section get them out of an inferior memory no, not at the moment, there's nothing to stop them being made that way if you want them to be they're not allocators I think they live on the disk but they don't live in memory I've kind of made it that way at the moment I've made it that way at the moment purely because all current for gdb to do this stuff now it needs the files on the disk so it didn't make sense to put the stuff in memory for all the inferiors when it doesn't make gdb be able to do anything extra but the client like the client library is it doesn't care where the functions come from it doesn't care whether they've pulled out of memory or taken from disk because you just give it to them I had two ideas one is if it's coming off the disk can you attach to a running program who's executable and deleted if it's coming from memory having an allocated section has pointers to pls things because some of that is pointer mangled at the moment they're coming from the disk and so you can't you can't attach to something that is executable has been deleted it kind of went through my head oh you can do all these things with it but then if the executable has been deleted there isn't a lot you can do with it and from gdb terms like if some other use case has that it is useful that denotes to load it into memory and the other thing is things like I think the thread debugging stuff is I don't remember where it came from but there's a kind of requirement that it can work without debugging for installed whereas some of the other things like pretty printers maybe if the pretty printers used infinity functions and those infinity functions could go in the debugging for if there were a lot of them so sometimes people want to be able to use gdb and debug something that has green threads or understand its thread system is it possible that those systems could export these same functions without actually work or are there other dependencies on the thread ID system until you don't need to teach gdb stuff I mean if the if the threading system you're interested in can fit into the api of the notes that are currently there then yeah I guess no extra there will be no extra work but I'm not familiar with any other threading system there are two levels of things here if you want to do that like for support teams better support for go routines in gdb gdb is not aware of go routines you debug and go you schedule a side sequence and change where they go to the front and to support that you need to come up with some api that debugger queries like some higher level entry points like please tell me what is the kernel thread this go routine is running on right now or I want to schedule this go routine that's currently unscheduled please hack the schedule to make it wonderful right now but these are the entry points you need a level that the debugger needs to access how it's implemented to find the scene how does the component discover the structures inside the render program where to poke and schedule how do you describe the structures that can be Python that can see they have these advantages they can be infinity and then there's the advantage that that it's architecture independent so an infinity solves that layer of architecture and architecture independence but it still needs the higher level entry points it needs to be designed and specified one other comment before you asked about moving the thread to gdb there's one disadvantage to that which is latency because you asked for print me variable and you notice the bunch of op codes that ran when they showed that the block enabled a bunch of memory accesses back and forth just follow some pointer and pointer chasing if you do that on the server then the host is going to ask the server give me CLS base that's one long trip on a potentially slow latency link a few milliseconds perhaps if you move that all this computation to the host you may have milliseconds for each of those and if you have a handle then it can be noticed it can slow you down I'm just thinking like first of all I had the thread to gdb support to something and it's a nightmare and I don't want to deal with that again it would be nice to just remove it well and then there's the callback stuff but it seems to me you'll need something like that with this scheme because it will require you to find and extract some section from it like gdb server doesn't know how to do that and generally some other implementation probably wouldn't know how either because nothing needs it right now so you can either make out a new requirement or this live thread db and we'll have to use that same simple callback api to just look up a different symbol like magic symbol meeting send me the but it's just some code you have to know about the way I am imagined it working for gdb server was that gdb and gdb server would both have the infinity client library and gdb would feed the notes to the one that it held so gdb will be passing the elf because I guess if a gdb server you'll have the thing sometimes you might have some strip binaries over here and you have binaries with symbols over here I think that's pretty common so the gdb would be loading the notes and then sending them over to gdb server which would then kind of maintain its own copy of the kind of like each gdb server would have its own copy of the notes and they would execute there to avoid the kind of latency gdb had handle any symbol look ups that it had a relocation just shipping the data over and it was like that part of the gdb server and the call back thing is also just it's very nasty to implement you don't ask me why it's done that way and you know it is how it is but it would be nice if something new didn't work that way it worked it's more like bizarre because interface with copy console and so on function point of base register this call back well I don't mean the lip thread to be part I mean the part about gdb server there's a special thing in the protocol to do this dance to get some little backup that little symbol look ups I know the bit you mean is that can it be yeah yeah yeah cool is anybody else okay okay yeah I've kind of made it like as generic as possible so at the moment the node can probably use its gcc which generates elf but there's nothing to stop there's nothing forbidden in the standard or whatever this back it's elf at the moment because that's what it is but they're self-contained units they don't rely on anything else from the elf file they don't need the byte order or anything like that they exist kind of on their own cool alright thank you all for listening