 Hello everyone, please welcome Jan with his presentation about turning complex application to a service. Hello. Have you got a virtual machine and you have installed that software there, maybe it's a wiki, maybe it's a, I don't know, time tracking system, maybe it's a monitoring stuff for yourself or for your team or you have a complex application and I will show an example of that one later. Typically it's not just one demon, typically you would have, you know, it would have web interface, there would be a database, there would be some additional stuff that you happen to add throughout the time and maybe you have friends and they come and say, well, we would love for you to set up the same thing as you have in Amazon EC2, do it for us, but you don't want to maintain it for them and you know that if you let them maintain it, they will not upgrade it so eventually, you know, the machine will get compromised and they will yell at you again. So the plan or the idea is to be able to take such a set up and make it easily deployable somewhere else, maybe a little bit lightweight because typically such a virtual machine will sit idle for most of the time and also maybe converting your solution that could be started by your friends or by the other teams in your organization whenever they need to without you being bothered. So we assume that there are multiple services or demons configured on that machine, it's probably a Linux machine. There might be some crown jobs because you know you have a wiki but you want it backed up back to your, well, storage or update it from your storage, maybe you have email send out so that you know you get some log watch reports here and then. You probably have SSH there so that you can log in when you feel like it and you upgrade it, maybe you've installed it with Ansible or maintain it with Puppet. The goal is to containerize that application without breaking it into pieces, without converting it into multiple containers and yes, you've been to those other presentations that told you that you should never do that but if you have such a set up, why would you want to break it up to pieces just so that you can go to OpenShift and put those pieces back together. You would have to learn all the JSON things to basically recreate what you already had and what you were sort of happy with. So the plan is to convert the application into something which could be running containers but not do it the right way and the hope is that there is still quite a big gap market for such an approach. So if you want to containerize your application, how do you do it? And I sort of assume that you have all built a container or two. So you probably would install the software in build time when you are creating that image for that container, right? And the settings that could be done, that are not specific for that instance, you would do in build time. So if you take a typical Apache installation in Fedora or CentOS, you want to tweak the configuration to disable modules that you don't need and enable those that you need. You probably want to look at your application and figure out what's the configuration and what's the data. And I now talk about the configuration and the data of the whole application setup, of the whole stack of that whole instance, not about those individual components because you have that setup and it's there to play together. You have multiple demons, multiple services, but only as one unit, logical unit, they make sense. Figuring out what's configuration and what's data, it can be an iterative process. You might want to say, well, it's easy. Configuration is in Etsy and data is in Wara. Well, not the whole Etsy and not the whole Wara and maybe they live somewhere else. The goal is to take the configuration data and store it to persisted volumes outside of that image. So that when you start a container, you give it that configuration and the data and if you kill that container and maybe move it to another machine, you will be able to recreate it by marrying, again, the software and the configuration and the data. If you do this and you give that image to your friend, you will likely want that image during the first time it starts to configure itself, set up the passwords and whatever customizations and initialization needs to be done in runtime. It needs to be done in runtime simply because you don't know the parameters that your friends or that other team will want to use. By the way, yell at me or ask questions whenever you feel like I missed something or whether you disagree or something. So, and there will be multiple services to set up, right, that there will be the database to set up and the schema to populate and the Apache needs to be configured properly with, I don't know, TLS certificate or whatever. And in runtime, whenever that container gets start again and again, it will just start the services around, right? So, if I want to execute multiple services in a container, how do I do it? Well, there are multiple ways, but the easiest way is to do it just like you would do outside of the container. So if you use in it or system the outside of the container to run multiple services, well let's do it in container as well. So, let me try it. Dockerer, is it big enough? Yeah, Dockerer run container equals Dockerer, I'll get to that in a minute. Ferrar 24 user has been in it. I have started a container and run system D in it, inside that container. We can check what's in there. There are some target services, whatever, enabled and running there, right? So it looks like a virtual machine, it is a lightweight virtual machine. There are running some defaults, services there, you know, things seem to be going fine. You want to use that option, container equals Dockerer to get that output that we've seen from, you know, the status output from system D. The only problem is that it will not work on Fedora 25 and newer. That's why I started Fedora 24 to sort of show that it used to work in the past. If you tried to do this like a year ago or earlier, you know that you had to bind mount using that dash v option, cgroups file system and specify something for slash v. slash run and slash dmp. It's no longer needed if you have the OCI system, the hook package installed on the host. So it will do these mounts for you. We can check it actually. I'm in the container and if I do mount, I cannot do, yeah. We'll likely find stuff mounted already for us. So there is some magic in the recent installations and if you use Docker, Dmon installed on Fedora and CentOS and well from the distributions or from Red Hat, it will pull in this dependency automatically, so it works. We've seen that the command to start it is pretty simple. So if I want to start a system D in a container and configure the services upon the first time, how do I do it? One example or one possibility is to do the configuration and then start system D. I can exec from that first process and set up everything and then run system D. The only problem is if while setting things up, you need system D running, you sort of have a problem if you did not start system D yet. So another possibility is to create a configuration service which will be possibly started before all the other services that you have and which will do the trick for you. So, yeah, you can configure before system D or from inside of system D. You have to be a little bit careful or a little bit clever because to pass parameters from outside of the container to the container and to that configuration service, environment variables don't really survive execution of services and system D. Unless you configure, you know, specify the configuration, putting things into some temporary files might work for you. It certainly works for me. Also, if you would like that set up process to spit out something so that you can see it, again, your services might not necessarily be able to write to the console when you start the container. So that's something to think about as well. So what does a typical system D base setup look like? This is a Docker file, so if you've never seen one, this is how you specify content of your container and let's try to build the image and run it. Let's scale on the one that we have here. I have the Docker file. Where do I have it? I have the Docker file here. So in build time, I install HTTPD and ModSSL just so that I have something to work with. I specify the environment variable with that Nth directive. I enable HTTPD, which is very clever of me. You know, whenever you then start the system D, it will start the HTTPD, so I can do it in build time, right? So let's build a container. System D from system D, HTTPD. It was very fast. It's not so fast. So you might want to do it along. Don't do it here because it won't be that fast. Everything was cached for me. If you want to try it online, you might want to log out to some machine which has better networking connectivity than you have here on the Wi-Fi. Anyway, but I have the image built and now I run it. Docker, run. That's system D, I think. Do I need anything else? Ah, I don't think I need. I don't see any output. Why? Yeah, because this is from Fedora 25 and as I said, Fedora 25. System D will not show you, well, the status, but we have container running. We can look at what's inside and we can look at status of the HTTPD. My laptop is fairly slow lately and I have no idea why. Yeah, so I see processes running. I executed the system CTL in the container. I see HTTPD running, right? I can crawl... What's the IP address? Inspect. When you want to know the IP address, you grab for IPA, so you will find IPA even when you are not looking for it. So you see that I have Apache running in the container and I was not starting it by specifying that Apache CTL, dash D, or whatever you find in all the Docker files around. I've enabled and started it via system CTL, enable HTTPD in build time. Let's us look at it again. So this is how I defined my container image. Now, so things work. Are you happy with what you're seeing so far? Any complaints? I will continue with the assumption that you are happy so far. So what do I have? I have software installed in user, right? When I install the RPMs, it will likely also populate some stuff in war and at sea, so I have those defaults in there. I have services enabled. You've seen that I've enabled HTTPD and it has started, so that's nice too. I have configuration, I have data. So where is the configuration and the data? There's a nifty command, Docker div, which I can use to see what has changed in that container but on top of that image. When I start the container based on the image, it creates a writable overlay or writable file system on top of it and all the changes in the container go there, that happen in the container go there and you can see that there were some changes, new files created in Warlock HTTPD, but you can also see that when you start the container, it will remove some stuff in WarLip RPM. You can see that it will even update your HC password. So we've just enabled HTTPD and look at what gets changed. A lot of things get changed. The reason why the HC password gets modified is, anybody put venture gas? Sorry? Now the HTTPD user, UID48 gets added when you install the RPM in post-install script. So the image already has that record in HC password. So it gets done in the build time as you would sort of expect. Any other guess? Last login what? Last login should not really update HC password. That goes to WTMP and ClassLock. I will not tell you, you try it. Yeah. So a lot of things get modified besides what you would expect to be modified and not all those changes really are the changes that belong to your applications or that are part of your application. At the same time, so you want to take the bits from HC and VAR and maybe other locations and put them somewhere. And what I sort of thought was clever three years ago and what, since I've realized is essential by now, is that you want to put all those configuration files, data files, directories into one location, one volume. So that when you start your application, you will have it all in one directory and when you bind-mount your data into the container, you just bind-mount one directory, not 15. One reason is that if you have 15 to care about, you will likely miss some. So you will have your application running with your data but not with your configuration or with your configuration without the data or half of the configuration, half of the data. Also, when you upgrade your image later on because you want to add some additional clever feature, you would then need to update the definition of how things get run. You would have additional volume to bind-mount. Well, if you just put it all into slash data, the next time you run the container, it will just get added there, populated by some default content and off you go. So the approach that I propose and that I use and shout if you have some better one is to do SimLinks. So I install the software, I tweak it in build time and then I pick the changes that I sort of think define the instance at chttpd or at csystemd system or warlock. And I move them aside to some directory. Data template was my pick back then. And then I create SimLinks back where the stuff used to be. And if you look at where I moved the data and where the SimLinks point to you, what do you notice? They don't match. Why? Because if I moved the stuff into slash data and then I started the container and bind-mounted into slash data an empty directory from outside of the container, I would lose access to that content, right? Because I would see that empty directory. So if I move it to data template, I set the SimLinks to point to slash data. And then when I run the container and I sort of realize that the data volume is empty, I copy over the template. I populate it with the template. So suddenly the SimLinks are all right again. And with the cp-n or some use of tar or something, you can do it every time you update your... You start the container with maybe new application. So fill in the new additional locations that I was talking about a while ago. Yes, please. Yes, there is. Yes. There are two shades of gray. But you don't probably see them. So yes, the first one is in it and the second one is a snippet that I would add into a Docker file. So I take that in it, I feed it into user local as been in it and I say instead of running user as been in it, I will run user local as been in it. Any idea why I name it in it and not just my clever data moving in it? Well, it's a wrapper and it execs the real system on the cert line. But by naming it in it in the image, the OCI system, the hook kicks in. So it does not really care about where the script is located as long as the base name is in it. So this way I get the seagrows mounted and all is fine. Any other question? Sorry? Well, you just happened to grab source directories about sources of various stuff and you figured it out. So we have a container. If we have enough time at the end, I can show you how these things work for the simple case, but we have more interesting stuff here. This is a system-based setup in a container. Now, containers are fine. You start them with the docker-on-command, right, or docker-compose or whatever. Now, let's move to services. If I want to make something a service and I gave that example at the beginning, I want to give my application or my setup to my friends who will be able to set it up themselves or to my customers who might want to set it up themselves. I really cannot assume that they would be able to tweak the installation, install stuff, start it, and then use docker-exec and system-ctl-enable here and system-ctl-mask there, and so on. So we really need to plan for it and have a reasonable number of parameters that will sort of do reasonable number of actions during the initial startup of that whole thing. A very important thing here is that with containers, you can sort of treat containers and the storage that they have in that varlib docker or wherever those volumes or the images are. You can treat them as the instance unless you remove them, they will be there and you can start them again. With services, if you start this thing in OpenShift, that orchestration layer can take that port and move it to another node. It can remove that container and start a new one, a fresh one. So everything that sort of defines your instance, the configuration, the data, the logs, whatever, needs to live in volumes and you sort of need to be rather diligent in figuring out what is the status. The fact that whatever is doing it, I think system D, is messing up with varlib RPM for some reason. It's not something that defines the status of your instance, right? So, the iterative process and some work with Docker diff helps here. In OpenShift, your containers will get removed and created elsewhere. So you need to have the persistence worked out and you probably want to have just one volume which holds it all because otherwise it's a complete mess. Another thing which might be sort of special about or that you need to take into account is when you move to services so you would like to run your application in some PAS provided by someone that they might not allow you to run your processes as route. So, UID namespaces could be handy except UID namespaces don't really work well at least checking Docker and system D in the container. So, we have to wait for that to be resolved. You might find out that when your container gets started in some environment like OpenShift that the hostname of the container is not really yours to pick. It will be assigned by that orchestration layer. So if your application sort of thinks that it could look at the hostname of the environment where it is running and use it for something, Apache using it for determining redirects or something, it probably will not work. And for things like identity management which manages identity of other hosts including its own identity that can prove to be a little bit tricky. That system providing environment might be targeted at web. So if your application is about something else than web, those HA proxies and other stuff and clever setups might not really apply to you and you need to find other ways to make your application accessible. Also, you might have an application which has master and some replicas and those replicas need to be set up against the master. I found out that it doesn't necessarily, it isn't necessarily well supported just yet. So an example of a complex application, FreeIPA server. FreeIPA is an identity management solution. So we have data in LDAP. We have Kerberos distribution center. We have certification authority. There's a web server on top of that. There can be DNS server, other services all configured with one script. So typically you would configure it in one virtual machine. It will set up multiple services under system D. It has many services listening or provided or many ports. It also can do replication quite well on its own. So we have the necessary bits to run it in containers on this repo if you want to check it out. Here we go. On GitHub. We also have released RAL-based container as technology preview with RAL Atomic 7.3.1 recently. And now on top of that containerization work I've tried to make it run as a service. So I have OpenShift running here. I'm clicking to project. I made my preparation. So I've already imported a template for the free IPA server. So what name should I use? Free IPA.gov. Image. You have choice of Fedora 24, but it will fail. Through not my fault. Bugs have been filed. Fedora 25 or CentOS 7. Which one do we want? CentOS, OK. Why choice? It will be free IPA CentOS 7. Let's give it an IP address 7, whatever. Here are the options that will be passed to free IPA installer. So you can just trust me that there is a script and it takes some options and you can specify the options here. I used the defaults. Well, whatever. Password so that we can log in there. And I will hit create. So I have created an application. And if things go right and I don't see why they shouldn't, I should get a pod created. Pod is OpenShift sort of logical unit where you can have multiple containers in it. We only run one container there because as I said at the beginning, I did not really like the idea of breaking it into multiple containers and then just putting it back together in some JSON. There are logs here. And you can see that my free IPA server is being configured. Running in a container in OpenShift environment. I could start another one but it's running on my laptop so it would be even slower. We'll come back to it later when it finishes. So the definition of that service, if your container is done right, is then sort of just trying to find out the correct lingo for your particular orchestration provider OpenShift in this case. If you want to try it right now and the JSON files are there in the GitHub because the free IPA guys were kind and merged my request, you might want to... It will not work in OpenShift Online because in OpenShift Online you cannot run process as ruled. If I go to the machine where I started to that OpenShift machine, where's my OpenShift machine here? So if I check the containers running, you will see user local has been in it. So I was not tricking you when I said that I will use this approach. And we can look into the container which was started on this node. It's a single node OpenShift file environment running here. The hostname is a little bit strange. It's not a fully qualified hostname even if we specified one. It was sort of mangled by OpenShift. So be careful if you care about hostnames. Sorry? No, because I made it work. You would think that it would, but it... Strangely enough it does not. And if I... So I was in the container now, I'm on the host. And if we check the processes running here, we will find... I'm probably putting too many options here. Which one should I remove? Yeah, so the processes are running here under the Docker container demon. There's that system that I've started. But what I wanted to show you, so let's put that option back, is that it is running as root. On the host, this process has UID 0. The Java stuff, which is used to run the certification authority, the doc tech, is running as UID 17. I don't have that 17 defined on the host in my Etsy password. That's why you see 17, right? Is that clear or not? So in OpenShift Online, you will not be allowed to run your process as UID 0 on the host. If you have an on-premise installation where you feel that it's okay for you, you have to tweak the security context definitions of OpenShift and allow it. So in my setup, I've created a service account and I've added a policy which says you can run it as root. Yes, please. Kerberos does... The question is does Kerberos require proper DNS resolution? Yes, it does, but well, it likes to have it. We are actually setting a DNS server in the free IP container as we speak, so it will all work out nicely. If you are troubled by the fact that the host name of the free IP server is different and things still work, I was sort of troubled by it as well, but then the guys assured me that if I keep using it that way, it will become a de facto standard and that they will keep supporting it. Let's check the log to see how far along we are. So we have everything installed, there's some upgrading going on, so it will be another two or three minutes. So in my setup, you define some persistent volumes because you want the data to live somewhere and be persistent, right? If you're just playing with something like I do, you can look at a JSON file which defines host path, so you don't need to set up Glastrand and so on. And then you start the application. I've used the web UI, but you can use command line if you prefer to do so. And we have a bunch of options, so yeah. Again, I will repeat it because I feel that it's important. You want your service. Well, if it was a single demon, it would be fairly natural to have just one volume. For some reason, people get scared when their application setup consists of multiple services and they think that they should have multiple volumes, one for each service, but to me, it's a completely false approach. So we have one volume mounted to slash data and the rest you have seen. There are simulings, things that update it, things work. We have defined the cluster IP that one, seven, two, whatever so that I can access not just the web interface but other services as well. Unfortunately, it's master only right now because no matter how I tried and I talked to Michal for it extensively, so it's not like I did not try to ask. Neither Petset nor Stafelsets, which is a new name in the Kubernetes 1.5, worked for me. I'm using Kubernetes 1.5 alpha one here and they still did not work. It is supposed to work in such a way that it starts the first port and once it is running, it starts the second one and the second one sort of would set itself up as a replica. I did not get it to work. Ultimately, what you want is run the container with read-only file system, meaning that all the changes that it needs to do will go to the volume. Nothing needs to be changed. Well, the layer of the container is read-only. Some software is not happy about writing through simulings. Bugzillas have been filed. One important thing to note is you will lose a cylinder separation if you do it this way. On the host, when you run Apache, it runs as Apache underscore T. If you run Postgres, it runs as, I don't know, Postmaster underscore T. If you run things in a container, they all run as the same a cylinder type. So, if you are bothered by that, this approach is not for you. On the other hand, if you've used a cylinder disabled until now, you will not even notice. Okay, before concluding and before giving you a chance to ask questions, the thing says that free IP server is configured. Does anybody remember the IP address that we used? Well, seven, yes, but there's more numbers than seven, right? I cannot give it just seven. Let me find the IP address and you can start thinking about the questions here. So, if I change my resolve conf from the default to use the DNS in that IP server, does anybody remember the host name that we used? Here it is. If I click it, I'm talking to, well, it generated a self-signed certificate, right? So, this is not a bug. This is actually what you want to see. Admin, the password that we used, and I'm logging into the free IP server that I'm running in OpenShift. And I could show you enrolling a client, but you really want to start asking questions now. So, the server works. We have some things to flesh out. The UID namespacing, the replication. On the client side, well, the server is sort of nice. You can use it as an example for your setups, converting a legacy VM, which is sitting somewhere into containers. But for using the identity measurement solution in OpenShift or similar environment, you really want those other containers to enroll themselves, get identity, and consume those services, right? And I've seen a mixed feelings about that. Nobody really was able to specify how exactly they want the things set up. We are trying something, but we are not just there yet. So, you can have another container which is enrolled to this IP server, just like you can have a container enrolled to IP server running completely outside of container. So, yeah. If you want to check out the work that the source is there, the free IPA is built in Docker Hub, so you don't need to build anything. There are JSON files which sort of define those image streams, so you'll find them in the repository. And there's an older presentation about this approach also available. Questions, please? Yes? So, you told that when we had something like CMS, so when do you think it breaks? When do you think it's not a good idea to have such simple approach? Okay, so I've explained that it's an anti-pattern, and now I should explain why, when you don't want to use it. You don't want to use it when you have time and resources and knowledge and the patience to set up things the right way. So, if you feel like that CMS system that you have set up should scale upon a bigger load, by all means you probably want to split it out into multiple containers so that it would be run on different physical holes in that open-shift environment. So, I'm not saying that the default container mantra is wrong. I just say that there are use cases where the effort would be so enormous that it makes sense to explore other ways and this is one of them. Any other question? Yes, please? What were the problems with that replication? What were the problems with the replication? I start a service. I tell it I want to create two replicas of padsets. The thing starts the first port. It starts the first container. It says that it has started the first container, but it also says that it is running two replicas. The first container is green. I will show you that when we started the container, it was light green and now it is dark green and dark green is good because it means that some red in its probe figured out that if the circle is blue, it's good. If it's light blue, it's not so good. So, I have blue circle. I'm very simple-minded. When I see a blue circle, it's like good. I would assume that in that situation, there would be another gray or light blue circle and it never appeared. And the OC status output was confusing. I'm waiting for the final 1.5 release of Kubernetes and OpenShift to give me a solid foundation to work with because I think that there's something buggy there. So, I did not get the second port, the second container to even start figuring out what is needed on the application level. Yes, please. The container that was created for VAP in OpenShift, is it a privileged container? No, it's not that privileged. The container is not privileged. It's a container running as PSZ. I'm looking for that init, right? Running... Well, this is init1, so that's not the one I want. It is running as a normal and privileged container here. Yes? FreeIP server is a federation... Well, the question is, would it be easy to scale the LDAP? And the answer is, FreeIP was set up as a federation of getting things set up, multiple services. It assumes it is running on a single machine. If you want to replicate it, you use IP tools to create the replica. It will replicate that LDAP and everything else set up the replication agreements and sync things work. But merely replicating the LDAP will probably not buy you much because you would not have a use for that. So we really want to be able to use the application level or application supported replication scenarios here, not just replication of some of those components. And we cannot put those components into separate containers because the setup tools, which is at least half of the value of the whole solution, does not support having LDAP here and Kerberos here. It assumes that it's on the same machine. That's the reason of the whole exercise. Another question. That means that... Oh, thank you. I'll be around if you want to ask some more questions or want to see my client IP enrolled to my containerized service. Thank you.