 OK, next up we're going to have Miquel Jansson who's going to be talking about tracing and containers with LT-TNG. So hi, can everyone hear me all right? Yeah? OK, so I'm here to talk to you about LT-TNG and containers. And I want to say LT-TNG stands for the Linux Trace Toolkit Next Generation, which is a name we don't really like, but it's been like that since the beginning. And it's stuck with us, so pro tip, choose your project name wisely. So my name is Miquel Jansson. I work at Efficios, which is the company that does most of the development work behind LT-TNG. My background is not in development, but mostly in system administration. So I do mostly all the CI work, the porting to different architectures, build system, and so on. And since last year, I started developing new features integrating container support into the LT-TNG toolkit. So just a quick overview of what I'm going to cover today. So this is a follow up to last year's talk. If you were here last year, I apologize, because there's going to be a lot of the same material covered, because last year I was explaining a proof of concept, but now we're in the implementation phase. So yeah, I'll cover why we'll first, what's LT-TNG for people that are aware of what it is? What do we mean by supporting containers in LT-TNG and our progress since last year talk? And I'll go over the features we have now and the features we're planning to implement this year. So what's LT-TNG? It's a suite of tracing tools. So we have two tracers, one kernel base, which is kernel modules you insert in the next kernel. And the other one is a user space tracer, which is a set of libraries which you link into your project and then add a trace point in them. So how can you describe what is a tracer to start? It could be described as a fast, non-blocking, structured binary logging framework. So basically, logging, but faster and more flexible. So the kernel tracer is used to get low-level information about your system. So you can get syscalls and their parameters, the scheduling event, AOI events. So it gives you a really in-depth view of what's happening inside your kernel. And then the user space tracer will be used to, well, applications have to be modified to use the user space tracer. So trace points have to be added to them. But afterwards, if your application is instrumented, then the two tracers can be used together to correlate information with the kernel formation. So all our tools use a common trace format, which is called the CDF. So it's what's produced by both tracers and then what's consumed by the tooling and the analysis scripts and so on. The CDF format is also, it's a spec format that's also used by other projects like GDB for their tracing functions too. The tooling is called LTDNG tools. So it's a common tooling that's used that you use to control both the kernel tracer and the user space tracer with the same kind of commands and primitives. And then we have a command line trace reader which is basically a command line tool that's used to read the binary tracers, the binary traces and convert them to a human readable text format. And also could be used to convert between different binary trace formats and so on. And we also have multiple graphical trace readers which are UIs that gives you a graphical view of the content of traces. So there's one that's called trace compass and there's also a LTDNG scope. So why would you want to use LTDNG? Because it gives you a combined view of kernel and user space. And also it's a very low overhead logging and tracing solution. So it can be used on production system that have actual production workloads well without having much of a, well there's always a small impact but the impact is usually low enough to be used on production system. So the difference of a logging framework or if you would set your logging framework to be a verbose debug output would usually have a very big impact on the running system. It also can be enabled, disabled and reconfigured at runtime. So if you have a running system and something happens instead of having to reconfigure the demon to output more information and then restart the service. With LTDNG you can just start a tracing session in the background, attach it to your running system and then get the information you need to troubleshoot the problem without having to restart or reconfigure the system. It also provides you flexible storage of traces. So traces can be just basically simply written to the local disk but they can also be streamed over the network or they can be stored in memory ring buffers and only get the disk on certain events when you really need the data but otherwise just spin in memory and have a very low impact on your system. And now in the context of LTDNG, what is a container? Well, not only in the context but what is a container and how can we observe it with LTDNG? So the problem we faced was that from a kernel perspective, there's no single concept of a container so you cannot read the data structure in the kernel that would list all the existing kernel, the existing containers and their names and so on. So we have to find other solutions to get this information. And also there's multiple container runtimes like LXD, Rocket, Docker and many others that all use the same kind of facilities in the kernel but all do it kind of their own way. And if we want to be a generic as possible and support all the use cases of all your users, we have to support all those different container runtimes. So what we came up with is that all the containers are all based on the same kernel subsystems. So basically the namespace, CROOPS and also other isolation and security subsystems depending on the runtime. So what does it mean to support containers in LTDNG? So we can divide this task in two main objectives. The first one is that the traces we produce, the data you extract from your system with LTDNG needs to contain information that related to containers so you can basically like filter events based on, let's say I wanna know all the right syscalls that are coming from this container instead of having the list of all syscalls that are done by all the containers on the system. And the other main task would be to adjust our tooling that which is currently which currently assumes like a monolithic system which was designed before containers existed. So adjust our tooling and our deployment strategies so it's easier to use LTDNG in container systems. So the main task we tackled last year since my talk last year was the trace content. So the features are developed, they haven't been released yet but they are queued for version 2.12 of LTDNG which should be released around April of this year. So the main features are, so we added to the kernel tracer a state dump of the namespaces so that will help you to get the state of the system at tracing startup. Also we added context to classify and filter events with namespaces and we already add the syscall events that will be used to track namespace changes. So I'll go over more details on each of those features. And the same kind of support was also added to the user space tracer but only the namespace context was added because from user space the view you have from a container is much more limited than from the kernel side. And what is still missing is having some more events that come directly from the container runtimes to get a view of the life cycle of the containers and some more metadata from the containers. So for the specific features, first was the context. So what is an LTDNG context? It's basically a metadata field that you add to a trace event. So if you have an event that says basically an event from the kernel for example which is a syscall for example, you can add information to this like a lugging statement. You would add metadata. So for the context we currently have are like process ID, threads ID, process name. So you could basically enable those contexts on your tracing session and each of your event would be tagged with the proper information. It's useful for when you manually read traces, text traces, but it's also very useful for filtering. So if you want for example to only get the syscalls that come from a specific PID, well you could add the PID context and then add a filter that would say only read the events that come from this PID. In the context of containers, so we added context for each of the namespace type that are in the kernel. So, and this enables us to do filtering of events per container. And on the user space side, we can have the same context or basically have the same container ID to the user space events and then it will allow you to correlate events between a kernel trace and a user space trace that would and you would be able to know they come from the same container. So here's an example of a text representation of a syscall event, the get time of the syscall event. And you can see that we added namespaces. You can see a namespace, sorry, the namespace for the PID NS namespace context in green. So basically it gives you an ID for this PID NS. Inside the kernel, there's no concept of containers as I said before, but there's only namespaces and namespaces don't have names or there, they can only be referred to by the iNode number of the link in the proc file system which they are exposed to user space. So that's the only unique ID we found that we could use to identify namespaces. And so you can see in this example that the event is tagged with the context PID NS so we know from which container it comes. And you can also see that since this is a PID NS then the PID would be different inside the container and outside of it. So you can see in the fields TID and VTID the view in orange from the host and the view inside the container. So in this example, this is a Docker container running Redis server. So the init process is Redis server and you can see that it's PID is one inside the container but on the host it's 11,000 something. And so the same thing, well, we did the same thing for the user space tracer context so you can see the same kind of values and the same kind of filters. And so if you wanted an example of the commands you would have to run to enable a tracing session of all the syscall events that are coming from a specific Docker container on your host. We also have what we call the tracker feature which is basically a simpler filtering system that's much faster and we already have an implementation for PIDs so basically to trace specific PIDs on the system without having to use the filtering rules and we're planning to add the same feed while adding namespace support to this tracker feature so you can basically trace all the events that come from a container on the inside the kernel without, much more easily. And the other feature which we worked on is the state dump. So what's a state dump in LTTNG terminologies when you start tracing there are certain events that are emitted which gives you a view of the current state of the system. So for example, a list of all the PIDs, a list of all the OpenFDs, a list of all the block device and so on. And so we added to this information to the process information and the namespace context. So now you can have an initial state of your containers when you start tracing and you can use this when you do analysis afterwards to build an initial state of the system. And so we added this information to all the same namespaces and also the PID and username space are hierarchical so there's a hierarchy you can have containers and containers and containers so this information is also provided for modeling and you can see the events that are generated when you start tracing and you can see that they now contain the ID number of the, for example, the PID namespace. And so the second big task we have would be to make our tooling aware of containers because currently all the tooling in LTTNG expect to be running on a single system with a single view of the amounts, the no ID mapping. So a single view of what are user IDs, what PIDs are but when you're in a containerized system depending on where you're running or from where you're observing a process that you have and I appeared that's different from if you observe it in the host or in the container so that's all things that we have to adjust in art. We have to adjust those assumption in our tooling. And also if we want to give some easier solution for deploying LTTNG in container system we have to review how we do security because right now it's pretty simple, it's based on UNIX IDs and fast system permissions and so on on UNIX socket. So if we do like bind mounts to expose UNIX socket from a demon running into the host on inside containers then we have a whole problem of like what's a UID and who can control tracing? And so well, I'll go over that. So currently the support deployment is pretty simple is that you have the kernel tracer inside the host and you have your one instance of a user space tracer inside of each container. But what we want to go is to have a single instance of the tracer in both user space and kernel inside the host and then give access to these resource through the containers so you wouldn't have to do a full installation of LTTNG inside of each container and you could control the whole system as a whole instead of having, seeing the containers as each whole system. And so that's the vision that they couple the container or tooling vision we have for the future and we really want to know what are your needs and your use case. So if you use containers and you use tracing or if you don't use tracing right now but it's something that you're interested in we really want to know about what are your use cases what's your current pain points and how we can make things better for you to use tracing. So you can contact us on IRC, Twitter and we also have a development and user mailing list. So does anyone have questions? That's gonna make things easy. Oh, all right. Yeah, that's not gonna make it to the camera. Well, about things that we are fixing right now for example, you have like several containers with an orchestrator like Kubernetes being able to trace something running in a pod in a node in a cluster, it's a hard problem right now. So I'm sorry I really can't hear you well. Tracing in Kubernetes, so you have like a Kubernetes cluster with several instances of a run application running as well nodes having a tool to be able to trace these applications without like going inside the pod or the container or going to the node and then tracing from the node. Okay, so if you have a cluster with containers and you want to trace a specific application inside the specific containers let's discuss a bit container. Well, yeah, we don't have a solution for a full time. In the line of problems that we are facing, the last question, so I think that's one of the problems right now, like tracing on clusters of containers, not single containers. Yeah, I know that well, a lot of people that are using containers right now that are talking to us, it's that there's a shift in before systems were static and they were, you would configure manually each of them but right now they're spawning containers and we would need to have some way of auto-configuring tracing into the containers and having remote administration but that's something we're interested in looking at but we don't have it right now. All right, out of time, thank you. Thanks.