 OK, everybody will be starting the next presentation soon. I just wanted to remind you to go and rate the presentations at our website, also tweet, blog, send telegrams about this event, and stuff like that. And please enter and exit the room quietly and not to disturb the speaker. And so I think it's time to welcome our next presenter. I hope I pronounced his name right. It's Jan Pastziora, right? Yep. Correct. So give a warm welcome to Jan. Hello. For the past year and a half, I was trying to put things into containers in a sensible way. And being from identity management team or with identity management group, the things that I tried to put into containers included free IPA server, which is something like Active Directory for Linux. And the client parts, the system service SSSD. And lately it was on Atomic. So I would like to share some experience with that journey. And it will be both about Atomic and about SSSD. So hopefully you'll find something interesting for maybe your endeavors in putting things into containers. There are applications and there are system services. And for some reason, that will become obvious later, I care about the distinction. So application is something that is the reason why you run all those computers, right? The web services or the GIMP or whatever. And the system services is something that supports each particular host. So for applications, often you don't care on which host it's running. And we've seen many, many presentations that showed how easy it is to set up containers that will take all your workload and move it over to a different country. For system services, you don't want some orchestration layer from all your machines to take all the caching name servers and move them to a idle machine so that you would have 50 off caching name servers on one machine and none left of the others. You want system services to stay on each particular host. So examples might be time synchronization, logging, authentication. Authentication is something that I was dealing with. So even if you take a minimal Fedora cloud image, there will be some system services running there, right? Network manager, Yej Client. So typically, how do you get there? Well, you install the software. You tweak the configuration manually or using some tool. You enable the service so that after it brews, you still have it running. And you start the service and you are there. So you tell me, no, you don't do it manually. You might use some configuration management system, puppet, whatever, to do it. But eventually, underneath, they would do exactly, or in some way, these steps. So SSSD is a demon that you might want to have on your machine if you want that machine to support external users, authentication of those external users, host-based access control, pseudo rules. So it, besides authentication, it enforces policies for access and other purposes. So there's the demon. And there's also a nice setup tool called IPA Client Install, which sets things up for you. So let's try to do it. I have IPA server. I have no host besides the server there. I have Fedora Machine. This is not atomic, so I'm on normal Fedora Machine. And I would like to be able, because I have some users in my external centralized identity management system, I would like Bob to be able to log in to that machine. And if I now check if Bob is there, Bob is not there, apparently, Bob was not so lucky to get into Etsy password, and that's eventually correct. So I would like to IPA-enroll this machine to be part of my free IPA domain. To do that, I need to authenticate to the free IPA server, I can do it using either credentials of some admin like user, or I can pre-create the host record in free IPA server first, and then use one-time password for the host. Which way would you like me to do it? So who wants to see admin's password? No one, so you all want me to go the one-time password route, right? So I create Fedora here. I click force, because we don't have it in DNS. I say set one-time password. I set the password to Yezhek. Nothing has happened, but if I now go to Fedora and run IPA client, install password Yezhek. What should happen is it enrolls the machine. Wow, what happened here? Yes, unsecure is good. So it configures certain many aspects of that operating system. And it should end with telling me that everything is fine. And how do I know that it's fine? Well, suddenly, the machine knows about Bob. So that's pretty cool. But we can check that as part of this setup, the setup script has started and also enabled, I think, SSS DDmon. So suddenly, I'm able to log into the machine as Bob. So I can do. And I can show you that. Do you want me to authenticate using password or Kerberos? Password? Someone? Kerberos. Kerberos. OK, you like Kerberos. So Kinead Bob, example test, secret password, and SSH Bob at Fedora example test. I'm in on that machine. It complains about missing hostname, but you would be able to tweak that yourself. This is not authentication matter, right? And I can do sudo bin ID. And I can, as Bob, I can now run commands as root. Why? Because, as I said, part of what free IP can do for me is sudo rules. And I have already enabled that without authenticating, I can run command bin ID. So with one setup command, I was able to basically pull my machine and make it part of externally managed domain. Yes, please. Thank you for the question. I hope that somebody would ask that question. I certainly hope not. So and because it's a long output and it might have been added to the middle, we've checked that it was not. So what we can check. So if I now stop SSSD, the SSSD is no longer running. And I run ID, Bob. What would you expect to happen? Well, you would expect Bob to be away, to go away, but that's not the case. And the reason is that SSSD, besides being a process, it also creates some fancy caches. And if I ask about Bob with disabling access through those memory caches, now I'm really not finding things in somewhere like Warlib, SSS, MC, I believe. This is where Bob was at it. So the Bob record did not go to Etsy password. It went here. And some libraries and SS libraries might still find Bob somewhere there, even if the demon is not running. It will expire eventually. So OK. That was setting up system service on a normal machine. Enter atomic. Atomic is minimal. One size fits all. You get that OS3 image. And it's written only. So let's check atomic. Because we would like to be able to do the same for atomic. Because we saw that it is pretty cool. So I log into atomic. The problem is that while I can create files in Etsy and War, the rest of the system is read only. So even if it is based on RPMs, I cannot install SSSD there. Because I don't have any YAM or DNF. You are not supposed to install additional software to atomic. The provider of that distribution of that image made a selection of tools that are essential. Atomic platform is somehow meant to run containers. So if you want something which is not even an application, it's not your web application. It's not your eShop. It is just SSSD. You would like Bob just to be able to log into this machine. You cannot do that using normal tools that you are used to. So how can we get that system service there? Anybody? Install a service in a container. So we want a container which would somehow be running and providing the operation or functionality of the service. So you create a container from a Docker file. You install some packages. You run the container. Now what do you run in that container? Because we saw when we looked at that non-atomic machine that there was that pretty crucial part of setting things up of somehow IPA-enrolling the machine, for example. And also the system services, many locations, many files, directories might get set up and modified and expected by other parts of the system. Let's step back a little bit. This is a typical containerized application setup, the way you would expect to run it in OpenShift or Kubernetes. You have your database. You have your application. You have some front end HTTP proxy or whatever. They might be all high availability balanced, and they are connected through DCP links, essentially. You probably want that database data to be stored on a host somewhere. So there might be some volume, so-called bind mounted into the container, but there would be one or few of those by mounts per each container. Typical system service. But it might have dozens of files that it cares about. And you want that files to be stored not in the container, and you need those files to be readable by other services. If that IPA client install changed PAM configuration at cpam.d, SSH should be able to find it there. So if you were to run a container with system service that includes SSSD, you would have quite a lot of bind mounts to specify. No one can do that right every time. So what can we do? Well, the Docker images now support so-called labels. So you can specify a run label. And in that run label, you can define mapping of all those dozens of volumes that your system service might need. So that's the first example. We have a long list of dash v, and the dash v is take this from the host, and make it available on this particular location in the service. And you will notice that the mapping is basically one-to-one. What we expect, we don't do anything fancy. We don't change the host names. We want a Kerberos key tab to land exactly in the same place in the container. So that'd be good. But the problem is if you run this container, many of those files would not exist. We still want to enroll the machine using that container. So the good news is you can specify more than one type of labels. You can specify install label. And yeah, I skipped even better news. And even better news is that there's a tool called Atomic which can understand these labels. So if I define a run label which specifies a long list of parameters to Docker run, I can use Atomic Run to basically execute exactly this command. And it will do some environment variable-like replacements on the way. And I can do the same for install phase or setup. On top of that, if I run Atomic install, the name of the image, and additional parameters, they'll get passed to that container, to that Docker run command. So now the question is, can we actually run IPA client install in the container while being able to pass parameters to it? Can we? Yes, we can. So I'll show you so that you're excited that it works. But then we'll talk about the details a little bit. So this is the Atomic thing, right? Let's see Fedora. Yeah, well, it's Atomic because this is 3.0 only. Fedora really stays the same. So I can do, I can pre-create the host here. Where's my identity? Where's my host? Or would you like to see Admin's password? No, you don't. OK. So I have Atomic here. Admin's password is pretty secret. So I use different one-time password here. They must match. So this is good. And now we do Atomic install Fedora SSSD password critic. Let's see what happens. It runs some Docker command. It passes, let me scroll up a little bit. Well, wait until it finishes so that it does not move under my hands. We pass it dash v. And we say, just take the whole root and make it available in the container s slash host. And that's not something mandated by Atomic or something. That's just that the clever guys said that it would be nice if you did it this way because it's easier for everyone. So we were able to run IPA client install, the setup tool in the container. And we landed back, we are not in the container. We landed back on the Atomic host. And we can see that SSSD was created, configured. So you would expect ID bop to return bop, right? Who would expect ID bop to return bop? Yeah, OK. So let's try it, ID bop. Now, what could be the problem? It's not running exactly because we need to run it. And we have two ways to do it. The Atomic way, which I want to use, and the system today way. And what is probably happening? Well, I'm probably running the Atomic run using in the start script of that service. So now I have bop. And if we check the processes, let's see. There will be Docker demo running here. And as children, and I did not exec if you were to one of those previous presentations, I did not exec. I'm just sleeping and looping in there for some reasons. But the fact is that SSSD is now running in the container. And it has bimounted all those files. So would you like me to try logging into that Atomic machine? Yes, you would. Can it? We do it as Alice this time. So SSH, Alice at Atomic Example Test. And we kind of expect, I have multiple VMs running there, so it's a little bit slow. But eventually, we would expect Alice to be there on that Atomic machine, right? So that's pretty cool. We were able to add a system service to Atomic machine, which did not allow us to install any software. And it provides service to things like SSH, to PAM, and other components of the system. Feel free to ask any question at any time if you get lost. Yes, I believe that what you are hitting, the question is, why do we have so many locations for the configuration and caches and data and not have it in one location? And I believe that what you are hitting is reality. This is how things were set up in the past for AGs. And Atomic is using the normal RPMs that are in Fedora or CentOS or Predator Enterprise Linux. So it would be a nice project. And it would be a long project. So even if we could do that project, if we want Alice to be able to log into Atomic today, unfortunately, we cannot rely on that. And of course, Alice can do sudo in ID. And she cannot run date. Just to show you that is just a sudo that is enabled. So what did we do in that install phase? So the important thing here is that the definition of the runtime and definition of the install time is in a Docker file. So it comes with that container. And we have essentially what I have shown in the slides in the definition. So in that install, we want to be able to run that setup tool. And that setup tool needs to be run in that container, in the context of that container. But there might be some existing configuration on the host that is different from the container. I don't know, nsswitch.conf might be different. We don't want our container to override that configuration. So what we do, we first copy interesting locations, interesting files and directories from the host to the container, then run the setup tool, then copy things out. And we restart on just to make things really nice. The real install sh is a little bit more complex. And it supports, for example, joining to Active Directory using Realm around D and so on. So in a sense, we want to create environment in the container that is similar to what we have on the host, or that matches what is on the host. We might need, eventually, in some cases, for other services, maybe to merge the configuration. That's what your install script would need to do. Then we run the setup script. And then we copy the resulting configuration and the data out. One of the nice things is that, essentially, this is a part that copying it out to host takes some time. This is an atomic operation. So only once the whole IPA client install has finished successfully. And we have a new configuration that we want deployed on the host. We do the copying out. Now, I've shown you that, after running atomic install, I have SSSD service. And how do we define a SSSD service? Well, we use that user-lip system D something, which is read-only. So we cannot create, define that service. Luckily, we can use Etsy. So we create a service. And now, things get a little bit tricky. Because when you run Docker on, it is just a client which talks to that Docker demon, which is running there. And the Docker demon eventually starts the processes in a container. Container is just change-route, maybe with some more fancy features. So we need to define the service possibly as one show. We want just to run the start, or run the container, and assume that it did not fail, and assume that it is still running. We heard that some work is being done in notification business. So eventually, this might get a little bit more complex. And we would have better control over what the status of that service of that container is. But in general, we can make that container behave as normal system D service. Issues. I mentioned one. How do we know that the service is still running? Another issue is we want for the processes in the container to be able to access those files and directories on the host. And to do that, we probably need to make it run as privileged, which is running in a SPC underscore T, a Selenix type. And we probably need to add some Selenix rules, for example, for debuts to be able to talk to processes like this. It might have been nicer if we could say we want to run all the processes in this container as SSSDA underscore T, which is how they run on normal system, except the transition from the Docker to that particular Selenix type is not permitted. So so far, we are running privileged. I don't like it that much, but that's the best we get. Restart is another issue. Do we want to rely on Docker? Do we want to rely on system D? Both have their own problems because if you rely on Docker and you disable that service and you reboot the machine but don't stop, Docker will bring your container up after a boot, so that's probably not what you want. Another problem that we have, SSSD and the whole IPN client suite comes with, for example, SSS cache command line tool. We would like to be able to make that command line tool available to admin on the host, right? And we can easily run that command, like make that command run Docker exec and basically run the same command in the container. Now, the problem is, where do we store that utility? User is read only. User local bin, that would be a natural place, is read only. I looked at making postfix system service, which would just take the local email and send it over to wherever you want it sent, make it run in container on atomic. And you have user-lip-sandmail, right? The typical interface, which is already on Fedora, a sim link to Etsy alternatives, except it's not installed on atomic. So do we want to create a bunch of sim links from user to Etsy so that the containers could tweak it once they're installed? Maybe. And another problem is that container is not exactly small because it can do not just that runtime SSSD is running and it's resolving users and groups and so on, but it includes those setup tools. So even after the install phase, even after the configuration phase, we still have those setup tools there. So maybe we want different container for running and the problem for setting it up. Now, that atomic tool is primarily promoted as a way to upgrade that OS3 on atomic, right? So we've seen presentation about that. But to me, much more important and nice feature about it is that it can understand the definitions of running the container and those definitions are stored in the same image that you try to run. So you don't need to have another layer, another wrapper, another service definition. It comes in that image. It's pretty cool. So the feature is not limited to atomic. Let's see. I come back to the Fedora machine, that non-atomic machine, root at Fedora example test, touch user A. So it's the non-atomic one. Where are we? We don't have system D running, but we have it configured. And when I say system D, just imagine your own service that you might want to run in Atomic. Don't get to hook up on SSSD. So if I do Atomic, install Fedora SSSD, and I have a special migrate option there, what should happen? Yeah, we don't have Docker running system. Start Docker, okay, now we have Docker. We run the same system, system setup install script. And it says, well, and that option is somehow coded in such a way that it just says, well, okay, you are trying to set up containerized system D application. The setup is already there. We will, status SSSD, we'll enable ourselves in Etsy system D. And now when I do system, see, tell, start SSSD, I might be cheating because the original system D on the host might have started, right? But we'll check if that's the case. And now it is running in the Docker and it's providing the services. I can remove SSSD, who does not trust me so that I can remove that SSSD RPM, but yeah, you trust me. So the nice thing about the Atomic tool, again, is that finally we have a way to actually say, not just install the software, install the software define external port, but we also have a way to say, on a typical system, this would be run like this. And especially the bind mounts and the mappings of the files are there. When you want to develop a system service for Atomic, it's not that easy. You don't have Git there and you cannot install Git there. So you need to use another container and it gets so long that you rather take your normal non-Atomic system. You create a container there. You can debug it there. And then, of course, you verify that it still works on Atomic, but... So, I'm getting back to Atomic as the tool. It's pretty useful if you want to do something about system services and not just in system services in containers and not just on Atomic platform. Now, you might want to do that, I don't know, to try a newer version of SSSD. You just take that existing configuration, you create image, maybe, completely different operating system, Fedora on Rails or on Fedora, whatever, and you just start the demo and you can verify where your Kerberos and host-based access control still work. It gets me to conclusion, but before conclusions, do you have any questions? Okay, you might come to questions during conclusions. Conclusions are labels, are nice, use them, use Atomic tool, and take advantage of being able to separate the setup, which can have access to the whole host root file system from runtime where we only make available certain locations and some of them read only. Moreover, as a result of this effort, we now have, and these images that are showing, they are available on GitHub, so we now have SSSD service for Atomic platform and you can use it. It will be a little bit big. On the other hand, if you have another container there with the same basic Fedora, at least that space will be shared. I have a bunch of links and I'll welcome your feedback, both here and on my email and elsewhere. Yes, please. How did I figure out the list of my mouse that had to be bind-mounted? Docker diff is a nice thing because if you start Docker container and run IPA client install and you don't bind-mount anything, you can run Docker diff on the host and it will tell you what files have changed. So it gives you the initial list, of course, different parameters to that IPA client install to that setup tool might lead to different configurations to be modified. So it's ideal if you know the software. But unless you're putting Postgres, everyone knows that thing has data in varlapg data and especially with system services, chances are that there will be multiple locations. Man pages, Docker diff, asking authors or just general experience might help you. It was iterative process. Other question? Since everything that the container should possibly care about is bind-mounted, could the whole container be made read-only? Docker diff SSSD? Yes, even if some of the files get modified. So I take advantage of not being forced to have it read-only. I would have loved to be forced because that would lead us to the necessity of basically resolving how do we want to deal with these things because I would probably need to use that new TMFFS to bind-mount those locations as well, but then, yeah, possibly to let the container do a bogus changes in places that I don't care about. Yes? I believe that if you just download, if there's a new image and it gets to your host, it gets started. The next time, the Docker run might actually fail because it tells you that there's a newer version and the old container using the old image is still around. We might need, that's another thing that I might want to put into a list of issues, we might need a way to distinguish stopping the container and killing it altogether. If you remember how I had the service defined over here, instead of running atomic stop and atomic stop running Docker stop, we might want to change that to run Docker RM and just delete it so that we don't have a container laying around. Another way would be to have some way to, but how do I know that I should run it? Then says atomic update should help, but I might need a way to, that will tell me there's a new one and you want to run atomic update. Well, yum update is a manual thing that was done. This one is somebody got me a new version to my machine. Oh, so back to your question. It should just work unless it doesn't. I'm out of time, I'll be outside, thank you. Thank you. I need to, I need to. Mike McGrath, I don't know if you're here. Yeah.