 I think everyone can hear me correctly. I guess so, okay. Great, my name is Nicala, and I'm going to talk to you about LTTNG and containers. I'm the software developer at ATC Sales, we're the company behind the LTTNG project, so I'm the developer of this open source project. My background is in the system administration, but also in development, and I do mostly CI working, and so on. And I maintain the LTTNG packages in the network. So first, what's LTTNG? Well, it's a suite of tools, mostly a kernel tracer, which I hope some of you are familiar with, and also a user-based tracer. Also, other parts of the project are the tracing platform app, which is called CTF, the common tracer platform app, which is used by all of the LTTNG tools, and also by other tools, like JDB, for example, with its trace, and well, if it's used in JDB, you can do also tracing it the same format. And also the LTTNG project, the LTTNG project has a single kind of line tool, which is used to drive both the user-based and kernel tracer, and it's also a security that can be used in other applications and used through bindings, through Python, and other languages. And there's also a common line trace viewer, which is basically reads the binary trace formats and convert it to human-readable text. And also multiple graphical interface, the user interface, viewers like TraceCompass and LTTNG Scope, which gives you the ability to go from tracers, export them, run some analysis on them, do some graphs and other stuff. And then why would you use LTTNG? Well, I won't try to explain why you should do that tracer, but I'll try to tell you why you should choose LTTNG if you're looking for a tracer. It's very low overhead. It's been built to be used in on-politician system and it is today used on-politician system by multiple companies. And unlike some more traditional lagging frameworks, which are comparable to tracing in some use cases, it can be enabled and disabled at one time, so you don't need to restart or reconfigure your system. You can actually trace in, and start it whenever you want without affecting the rest of the workload. And also there's a lot of flexibility in how you would store your traces, so the traces can be, the default mode will be to store them on disk, but also if you have like this system or systems with a little storage, the traces can be streamed over the network. They can also be written to memory in buffers and on these snapshots when you need to get the actual data, so the system can be running a long while without using any storage in them, just when you need to be actual trace, you can extract it from memory. And then for the container part, well, what is the Linux container? I'm sure probably most of you want to surprise to know that there's no actual concept of a container inside of an external, it's multiple subsystems like C-rubes and the same spaces that are combined to make different third-party implementations like Docker, Rocket, LHD and many others. So supporting all this in a kernel tracer can be a bit complex since they use different ways of, well, each container technology has its own way of using those features. So the least common denominator in here is the main spaces, so that's what we chose to use as a first implementation of how we can integrate with containers and LHDMG. So what's the current status? What can you do today without LHDMG on a container host? So the kernel tracer can be used on the host, needs kernel modules to be loaded, but there's no way to filter events per container, so you get, like, syscalls and all kinds of events, but you cannot really easily match them to a specific container. And then for the user space tracer, it can be run inside the host, but it can also be run inside a container. You just need to run two instances, like one instance per container. But again, there's no information that would be, or you would be able to correlate like a trace from the host side with the trace from the user space inside a container right now. So how can you fix that today? Well, we have currently some patches where we have the support for main space context. So what is the context? In LHDMG, a context is basically metadata that you can attach to each event in the kernel. So for example, existing context are TI, for example. So when you get a syscall, you would add metadata to it to say which TI this syscall was issued from. And so we've added support for main spaces. So basically you can now, when you get a kernel event, you can add, for example, the TI name space to it and then you'll know in which TI name space the process that called this syscall was. And also we have another feature that is called state dump. It's basically when you start tracing, state dump events will be issued that will basically give you a current state of the system. So another thing that we need to add is like a dump of the current name spaces that are on the system for, or the tree of main spaces for each process. So those patches are currently not upstream, they're feature branches. For example, here I have the text view of a kernel event for a syscall, the GitLang of a syscall. So with these patches added with the main spaces I just talked about. So we can see, for example, in green, we have the pit ns, which is basically the I node number of the name space in the process then because that's the only unique ID we can get in the kernel to differentiate one name space from another. And then we can see other useful name spaces which are the TI and VTIG. Basically the TIG is the unique ID of a process inside the kernel, so like this trade ID in the root name space of the kernel. And then the VTIG, which in this case is one, is the process ID as seen inside the container, inside the root name space of the container. So in this example, we can see that the process is called RIDIS server, and it's VTIG is one. So we can extrapolate from that that we're running inside an application container and that RIDIS is running as the ended process. So an application container that doesn't use an end process. So in this case it would be a RIDIS server about your container. And then the state dump. So this state dump would give us initial state about each process of our running inside containers. So for example, here we have the system D process and we can see that it's, sorry, we can see that it's TIG is 1527. So that's the, that's the static kernel. And then we get two TIG NS state dump event. So we can see that the namespaces are nested. So we see a, the first event has a level of one. So this is the namespace, that the namespace of the container and then the other one has a namespace level of zero. So that's the host root namespecodes. And we can see that the TIG never changes between those because it's a unique ID, but we can see that the VT ID as will change. So in the NS level one, it's the VT ID of one and in the kernel side 1527. So with this information, and we do the same thing for the user space tracer. Just call this thing. And with all this information now, we are going to do some filtering. So we can start a tracing session and say, oh, I just want to trace this as call that comes from this container. But right now it's not very integrated because we need to know what is the TIG namespace. I know number of our container to filter it. So I don't know if you can see, but we can, with Docker inspect, we can get the TIG from the process. And then with LSNS, we can get the number of the namespace. And then using this information, we can create another TNG session that will filter this. And the resulting trace will only come thanks to calls that come from process in this container. So we're starting to get somewhere. But then it would be nice to be able to use container names, for example, instead of having to get the actual number. So to do this, we need a way to map kernel IDs to human readable names. So right now the easiest solution we found was to use a small, well, basically a shell script based on what I've shown you before that will iterate over all containers. So you need to write a helper for each kind of containers you're running in your system that will basically dump this information inside a user space trace that can then become combined with a kernel trace and then run through some analysis script, for example. And here we see that I wrote a basic implementation that outputs events. And you can see that we just get container name and the container type and it's the DNS that we can use to correlate with the events we saw before. And with all this, you can use analysis tools that we already have, which are called LTDNG analysis, which are basically kernel based, there are analysis based on kernel traces. So for example, in this slide, it's probably not very readable, but this is a CPU type analysis. So it will take kernel trace and input and it will do a graph of CPU usage per process. But I've modified those and you can maybe see it at the, down on the slide where there's a bit of color that you can also now see CPU users by container and you can see the container name, it's ID and it's type. So there's already a lot of those analysis that were provided by LTDNG analysis project and they could all be adapted to the container where. And because of the analysis here, it's a memory allocator analysis based on page allocation, page three trace points in the kernel. So you could usually filter it on some pension, but now you can have container filter and see which containers that are on your system are giving page allocation and page allocation. And so with those tools, a complete scenario would be to run a kernel and user space tracer on the else. So you can get kernel traces and get the state dump or the name resolution from the container run times. And then for the containers where you're running applications that you could instrument with the kernel, sorry, the user space tracer, you could also run one LTDNG and it's in each of those containers. The version could be completely different from the OS system. So for example, if your OS is running an older version of Linux with a different kernel way, you don't have to have the same version. So it's inside the container and outside of it. So for deployment bits, it's easier. And then you could stream those also out of those traces too and really even also over the network and run some analysis offline to get the information you're looking for. So and what next? So all the things I've shown you are not yet upstream. So we're working on merging this maybe for the next release, but we're not sure yet. Another thing that someone else is working on is adding the same kind of things for context and speed up for C group values. So that's being worked on and should maybe land in the following months. And longer-term improvement would be to instrument container runtime so that we could get events about the lifetime of containers. So get events when the container is created, when configuration is changed and all interesting information that would also be integrated into the existing analysis tool to get you a better view of what's happening in the system. And also add better integration with the existing command line tools. So for example, instead of having to type a big filter, you could just like use the health community filter dash dash my container container name and that even would resolve the container name based on what's installed in the system and so on and so on. So having tighter integration between the command line tools and the different container run times. But what we're most interested in is what do you need. So I would very like to hear about you guys what you are interested in, what your needs are, what are your use cases. And finally, do you have any questions? Okay, I've got two quick questions in the morning. Do you have more questions? Yeah. First question, what do you do with nested opinion space? Nested opinion space. Yes, well, in the state dumps, we get nested events so we can resolve, know what's the chain of. I'm sorry? You just put it on the top level right now. Yes, yes. So this simple example, yes, but we have the information about the hierarchy of the data in space and also the user in space, I think, which are obviously hierarchical. Okay, the second question is about, did you see the insurance decision around the audit container name that's going on? I think, yeah. I've heard about it, but it didn't seem to be major enough yet to be something that's used. That's fair enough. I mean, eventually we should have some kind of way of tagging containers in the commands. I'm assuming we're gonna be using that at that point. Yeah, well, but it's tagging so it will probably be different for each container on time, but yes, that's information that could be more easily, instead of having to use both the kernel and the space tracer, we can use only the kernel tracer to get this information. Other questions? Yes. So the test process, is it feasible to do that? Maybe the analysis will be using another tracer, for example, if it will be helpful. To use the analysis? Yeah, to do the analysis, using other tracer. I didn't get the outfitted of the technology to do, to work with condensers. Well, I'm not sure. Well, if you can... Maybe the same thing, using just QADPF tracers. Or any of the other tracers. Well, if you can convert your tracers to the CTF format, then you might be able to run the analysis, but that would imply that the event names and so on are all the same, as for the LTNG, so it might be a bit complicated. Apologies about this, but what would be the point of running both the user space tracer and your container, and also the host space tracer? Because we want to maintain isolation between the host and the containers. So, and in between each container, so basically having, if you run, well, you could technically run the user space tracer on the host and have it trace process inside a container. But, well, first, a couple of simple patches would need to be applied as there's expectations that are going to be not met, but basically by fine mounting a directory with the proper sockets inside of your container. I mean, it works, but the goal right now is to not break isolation, so it's totally doable to run a user space tracer inside the container. Also, for deployment purposes, if you have like different distros in different containers with different libraries, it could get very messed up, so it's a lot easier to just have one user space tracer per container. But, like, there's no technical, there's nothing that would prevent it in the end to do the one tracer inside of us. Other questions? Oh, like, use cases? I'm not sure. All right, thank you. Thank you.