 Okay, everybody, let's continue and welcome our next speaker, David Bechwajik, who is a container enthusiast and evangelist, and he'll tell us about tools, checking tools inside containers. Let's welcome David. Okay, let's begin. My talk will be kind of quick, and it's about some stuff which is in a very, very early state. So, basically, what I will be speaking about, it's about injecting tools inside the containers, because, you know, Docker and all the containers are very popular now, and it's hyping, and it's a little bit fragile environment. They created a lot of new concept, isolation, reproducibility, and all these stuff. So it started, a lot of people start to use them, start to run them, move them from there to test to production. And sometimes it happens that something is not behaving good in your container. And basically, you want to do something with it. So, first, we can look where you can lie around containers. And there is some kind of evolution, how it went from the beginning for me. Basically, when I started with Docker and all the stuff, I used it on my local host for development and really stuff. So if something went wrong in my container, I was just able to get in or just use my tools on the host, change the space, and look inside. It was pretty good. Then stuff like Atomic Rancher Core OS came and they limited my tools on the host, which was still kind of usable, but it was getting harder and harder to look inside the containers and what's happening there, for example, how network works. So then the orchestrators came, like Kubernetes and all these stuff, and they started to throw containers through all my environment. So it was just hard to say even when the container is running, how it's operating, what's wrong with the node, it's getting worse and worse. But still, typically in that environment, you can manage your host, you can manage your nodes for containers and all these stuff, so you still have some access. But now, even containers hosting came, so you can have OpenShift Online, you can have Google Cloud and all these stuff, and basically you just run your containers inside. And if something is wrong in your container, you usually can get some command line to it, but if you look in, that's not good, because there are best practices for how to create your containers. Typically, you need to create them to be fit to not contain a whole operating system. Ideally, it should be only some static link go application there and run. But if you are in this way, it's not good for debugging, because you don't have any tool inside, you cannot ping anything, you cannot look at the racer, you cannot use sTrace, almost nothing, so if your application is busy, you have your logs, and you have almost nothing else, so it's okay to guess what's wrong with it, sometimes not. So basically, I was thinking more and more how to debug the containers, even if I don't have any access to the host of the container. So basically, I am limited to the local container and its internals. So, I was trying how to get some tools inside. I was thinking, okay, so typically, if we have some tool somewhere on some system, we can try to mount them. It's not possible inside Docker, because you need root privileges for that, you need access to your host and these stuff. So, second thought was, okay, I will try and use Fuse, but Fuse doesn't work at Docker containers. You need some capabilities for it. At least you need Kafka's admin, because it's still mount, and even if you are running on some online service, there is no defuse, so you need at least Knot for creating the device, so you will probably not have it. Another option was, okay, maybe if I needed to, I can copy it inside to the running container, but that's not so good, because you need to clean it after yourself and you are changing container, and probably you want your containers to be immutable, so if you are using some tools to debug them, or to just look what's wrong inside, you just don't want to mess with your container, mess with the process in the container and all these stuff. So, I've started to create some small library, which is called intercept file system, and basically it allows you to get your stuff into the container. So, how it works, basically it's very easy. It's one small dynamic library. So, to use it, you need to copy it to the container, and you need shell in the container also. So, basically, two stuff you need to use this is bash inside the container and this library. And how it works, basically it's a very small library, which is supposed to be already preloaded to your shell, or something like that. And it implements some basic calls like open, start, and so. So, I will keep in between these two sliders. So, basically, it's that situation. If your application wants some access to some tool, it needs to make some IO requests. So, basically, just open the file, run the pane, so it needs to look at it. So, in this approach, if some application inside your container wants to do that, it will get to the, the IO code will be get caught by that small library, and it will check path. It's the path is for the real stuff or the whole slide slash root slash user. It will just do nothing and defer the stuff to G-Lib C, and everything works the same way. If, if it hits some special path for my POC, I choose slash XXX. It then communicates to another container via 9P and use 9PFS and basically allows you to use tool from different container. So, this basically described here. So, if you hit this special path, you will get two containers. There is another picture. So, this is much more brief concept. So, how it works in reality. You typically have some application container, which is, for example, running your Django application, your Wallfly or something like that, and something is wrong there. So, you take Docker client or you can take OC command, OC client. You put, you just, we are Docker copy, put your library here, and you will run it via LD preload and bash. And then also you will deploy second container, which will contain all the debug tools you need. There can be stuff like strays, trace path, everything. And InterceptFS will enable you to access the, to run command from this container inside the, inside your application container. So, basically it will just bring everything inside, but it will not copy data inside. So, basically, when you exit your process from application container, remove this library, your container is in the same state like before the stuff. It's really, really easy. And it seems that it's some time works. I will probably show you some demo now. But keep in mind, it's most experiment. It's a small library. It can, it still can't do a lot of stuff. It's only can run simple commands and so, and don't run in the production. It's just a very, very experimental stuff. So, I will really show you how it works, because it will be probably better than speaking about it. So, I will get my terminal sessions. Okay. There's a lot of containers. So, you can see I am running some Federa containers, Federa containers which is named Tools, and it's running some 9PFS server. This is my container with Tools I want to use for debugging other stuff. So, we can look inside the container so we can see that there are a lot of stuff installed. Basically, it's a container. This container is based on Tools for Atomic Container, for CentOS Atomic or RedAtomicToon. Enable you to use the default stuff on your host. So, you can see that there is basic stuff like think is there, trace path is there, strace and other handy stuff. Okay, this is container which has almost anything for you to use. Then, I will spawn second container, which I will call up. It will simulate my container's application and it's based only on Federa. Oh, okay. I need to remove the... So, now it should work. Yeah, I'm in. And, as you can see, there are no tools I need. So, basically, if this container will start misbehaving, I can't do almost anything. I can only look to the logs and all these stuff. But, what can I do? I will copy my library to the container. So, that's easy. You can see it's just copy some file into temporary directory. And then, I will copy my command here to not write it. I will explain it. So, I will go back to my tools container where there's nothing still inside and I will run this command. What is it doing? It's basically saying that it will run Dash in some special environment. It will preload this library and it will set host and port variables. These host and port are important because they are saying where my 9PFS file system is listening for the containers with the tools. So, now, as you can see, there is still nothing. But, now, I said that there is some special file system which can be, like, which start with triple X. It's just hard-coded for now. And you can see that a stress is here. If I will exit the shell, the one with LD preload and will try to do it again, you can see that there is nothing. So, basically, this is very, very easy and almost, how to say, not intruding way how to get your stuff inside your container. It basically enables you to get any tool, any command you need inside your container in almost any environment. It will probably work on any environment which has internet access. Now, it needs direct TCP access to the 9PFS server, but it's not so important. You can use HTTP, probably, as a server for your tools. It's just anything you can hack inside. The library is still small. It's just support only 9P, but I will probably expand it further in future. So, and that's probably, I think, almost everything. Do you have any questions? Yeah. For now, yes, but it depends how you, if you will set, in future, if you will set your lip pad correctly, it will look to triple X directory and find it. But for this POC, it has issues with finding dynamic libraries because of the linking. But it's on my to-do list to fix it, but I still was unable. Okay, any other question? I think that you can use almost anything, but it's not implemented now. It's just because it's very easy to implement. But I think that what I will try to use, I will use 9PFS for now, and then I will be creating some plugin architecture for it. I will probably make plugins for some HTTP backends because I thought on GWR some file systems which are exporting data through HTTP because it's usually what you will be able to get in public cloud and it can go through proxies because usually, if you will try to use NFS or this stuff, you will usually hit some firewall issues. So basically it's... I think it doesn't matter what it will be behind, but it's good it will be able to go through HTTP, or HTTPS. Okay, any other question? Okay, so I think that's... Oh, sorry. I didn't see you, sorry. I think it depends. If you will put your tools container somewhere on internet, it will require internet access, but if you will put your container, for example, next, or if you will use some public service or your intranet, if you will put your tools container next to the container, you will just need access to the tools container. So basically it depends where you put your tools. And it also doesn't need to be at the... The tools doesn't have to be at the containers. It can be just anywhere in your network. You can use your own 9PFS server there, which is just one process. Okay. Any other questions? Okay, if not, so I think that I will leave you some more time, and that will be basically everything from me. Oh, that is... Get up early if you want. Test, test, test, test. Awesome.