 done. So hello everyone, I'm Tomasz Tomeczek, I'm working for Redhead and I would like to tell you something about Docker in production today. Okay, so let's start, I mean the title can be a little bit misleading, so let's start with like what this talk is not about. So it's not about existing success story, so I'm not going to tell you about how we deployed our application in production in containers and I'm not going to talk about that and also I'm not going to show you any production cluster or how it's running and I'm not going to do a demo about an update or a live update or something like that. So what this, what the talk is actually about is how you could run Docker in production, how you could like put your application into containers and run it in production. Then when you do that what you can expect from such an environment from such a change and finally I'll cover what's current state of the art of Docker like what they are up to these days. So this means that this talk is about like how you can make your success stories. Is anyone running Docker in production or nice? So I hope you guys will do a talk yourself someday, I'd like to hear it. Okay, so and finally I'll introduce myself, so this talk is presented by this guy, by me and Tomas. I'm not an operations, so I'm an engineer, so I usually do code which then is being helped by operations. I'm also contributed to Docker project, I'm not contributing to Docker engine but to other projects and I really like containers, I mean I'm interested in them in two years and I'm running some of my applications in containers so yeah I like them. So let's start with the first section and that's like steps needed for using containers. It's actually pretty easy, this is the very simple workflow you need to do like to run your application in containers. So first you need to take your application from its native form and put it into like container native form, so in this case like Docker image then you need to distribute the Docker image on their nodes and finally deploy to production, like it's pretty easy. So let's go through it. So before even you start building your Docker images you first need to containerize the application, like another term could be like the package it into Docker or container native form. Okay so it means that you usually need to produce a Docker file which is the template how to package the application or let's say it's like a recipe for building an application, building an Docker image. And also there's not enough, like there's just an image, there's this file system with some metadata so in order to make it a container you need to figure out an arguments like how to run it, so for example what volumes needs to be mounted with environments variables need to be available in the container and stuff like that. Yeah this is usually easy if you have like simple web application with I know with database or work service or some cache or something like that this is fairly easy you can learn this in like days and do this repeatedly. Yeah there are some cases when this is not easy and the examples is like graphical applications so for example you can run your IDE inside container or your web browser or for example your application requires access to hardware or requires some special kernel modules or some special kernel features and that's when not easy. This requires a lot of knowledge like how the Linux works, how the Demons works and yeah that's not easy. Okay that's the first part of the build like how to get to the point where we have the image before we can even assemble the image we need to pick a base image so that's the like minimal Linux environment for your application and I like to stress out that content of the image really matters so yesterday Honsa Horak had an excellent presentation about this like you need to pick base image which is like regularly updated doesn't contain CVs doesn't contain bugs and not like something that someone posted on internet it's something that's really stable and you can use it in production. So now we have base image now we have the recipe how to create the Docker image so we can do that. Yeah this is easy well it might not be easy again because this is done with Docker command with build sub-command and it has some issues and some of these issues were not addressed in like one or two years and community still complains about it and I mean Docker sometimes tries to do something about it and sometimes they just ignore it so let's take some of these issues so first thing is layer management so when you have Docker file like with all the versions of Docker every instruction in the Docker file produced one layer so layer is just root file system and metadata and these layers are when you deploy a container these layers are stacked onto them and you have like unified view on the file system so these are layers and it's not usually what you want to produce a new layer for every for every instruction because then your image gets really beefy and it really takes a lot of time to start it and it actually affects performance so community asks like okay so Docker we want new feature in your in Docker engine and that's we want you to be able to squash layers and it was like I mean issues and pull requests related to this were for like two or three years and none of them were merged or but right now there is one it's being opened by a Docker in employee and it's possible that it might get merged so let's say in Docker 1.13 we finally might be able to squash layers night natively so what community ended up doing they community feared our own tools for squashing so there were some of them luckily there's one right now which is being developed by a reddit employees called darker squash and if you need to squash layers I really advocate for this tool okay let's go to another issue and that's build time secrets so usually when you build or assemble your Docker image it's often that you need to access some external service and you need to authenticate with it or let's say that or I'm bio authenticate I mean that you need to have some password or some SSL certificate or SSH keys to like get to get get your sources inside the Docker image so for example get clone from a repository which is behind SSH the thing is that this this problem is not addressed I mean it's very easy to lead these secrets into the final image and it's really hard to like get the content from an authenticated resource inside the image so there are some best practices how to do this but it's not like addressed still there's plenty of more issues I had a talk dedicated to this problem on DevCon so if you are interested you can watch it or you can reach out to me and I can discuss it with you after a talk after this talk and finally related to building images is image hierarchy usually when you have your application it's not like one container it's usually several containers or I know like 10 or 20 and that means that you need multiple images and it's it's best practice to share as many layers as possible within your images so this means that you need to create a hierarchy of your images so you have like base image let's say Fedora and on top of it you have your own base image with some common packages installed and to figure out this hierarchy you need to like really think it through spend some time on it and like optimize it as much as possible and it really takes time it's not like you can just like right now okay so we have our image built so it's on one machine somewhere and we need to get it to our nodes in our cluster so we need to distribute it there are many ways how to do that and there's just one which is usually being done so the first way how to do this is to like produce an archive for your image you can do this with Docker it can like get a single file representation of an image yeah this is sometimes this is used but like you need to like automate it to the process of like exporting the image and then getting it to cluster so it's not used much but it's a possible solution the issue with this is that Docker usually changes the format of the archive so if you like export the archive on Docker 1.8 and you import it on 1.10 it won't work sorry so be aware of that another okay so when I did this presentation internally a week ago so this item was like oh I wish it would be if it would be possible to like share the images by a shared storage so you have it on one place and you just share it into our cluster and you don't need to care about like distribution this kind of stuff that was like that would be nice RFE and then yesterday I came to Dan was talk and he talked about this that they are playing to do this and I was really happy about it so yeah this is being worked on and in like a couple of months we'll see this one comment for this is that it would be very nice if we could share the content from Barlib Docker where Docker stores all these images to share it across the whole cluster like like the older layers but unfortunately it's not supported like some of the members of the community try this and the response was from Docker was that it's not supported sorry so what's being used for distribution is actually the Docker registry protocol and their service so it's called distribution so it's in their repo and you can use it the Docker engine client speaks the protocol so you don't need to even automate it or anything works yeah but it wouldn't fit on the line then yeah so yeah sorry about that I expected that if I put GH that it would be obvious it's deep up so yeah I'm sorry I can fix it yeah and finally when you start distributing your images with the Docker registry you need to figure out the naming scheme so for example I advise to use like semantic versioning so this image is alpha version this is production image and if something fails you know that you were using let's say like QA image not production image and again you need to spend some time on like figuring out the guidelines and finally we can finally deploy the deployed application yeah this is the easiest one no so this is the most complex one actually so let's start simply and we get to the most complex topics so that version of Docker engine matters so there are releases coming every I mean frequently and in the real matters what version you are using I mean the best thing is use the last one but what I mean by this is that specific release of Docker contains specific features and bugs so whenever you are hitting a bug you can upgrade to newer version and hopefully it's fixed and it's also possible there is a new set of bugs which will affect your application so I really advise you to test with a specific version and be the latest one and hope that it's there are no issues for your application if you have single node setup if I just one server running Docker and your application deployed there it's just most simple scenario but that's not very common usually you have multiple nodes to work cluster and you have your application there or even applications and that's really complex so for that you need an orchestrator or an orchestration system which will take care of containers and and the for the whole application life cycle of your application yeah so what can orchestrator do I'll tell you in a second but the important thing is that the orchestration system is like a whole new big service it's not just Docker it's also an orchestrator and you need to like spend some time for education for your stuff to learn it see how it works before even starting using it so what it can do so for example orchestrator will figure out networking for you just need to set it up and then your application will have correct networking between multiple nodes then there is storage it's also like hard to figure out but usually orchestrators have solutions like how to do storage rolling updates when you update your application you know just you tear it down and then you put it up and you have you don't have 100% up time that's usually not you don't want to do so orchestrators help rolling updates and finally monitoring like some of them do have monitoring some of them don't but you also likely you also need to use like your another type of monitoring like Zabix or something like that to like check health of your application okay so this is for deployment and let's do quick recap so before even starting using containers you need to establish a workflow like how to what the pipeline will look like from like build to distribute to deploy then you also need to figure out base images like on top of which base image you want to stack your application usually part of the workflow is like CI CD so before even thinking of putting something to production you can test it in CI so whenever you some of your developers do a commit you can put it to see CI and check like the builds are passing and then you can easily do CD on top of it some of the orchestrators even support this by default so that's very nice but you need to pick the correct one so and that's not easy I mean you need to check what features are available what requirements you have and that takes time okay so we have containers in production and let's see what can we expect from that let's start with the good so you have unified environment I mean usually it was I mean in the past it was pretty common that you have a team of developers and everyone is using something different this one runs Linux this one run Windows this one has Fedora this one has Gen2 and they use different version of packages and for example what works for them on their machines doesn't work in production or doesn't work on machine of other developer so with containers they have all the same environment and you can be sure that if it works in their machines and in CI likely will work in production then everything is automated everything is scripted you don't need to like run scripts to do the release or something like that just automate it and it works also it's tracked so you know that when something fails in CI you can see what image ID it was what version of application was in there what commits were part of the release so you can easily check it and you are in control of the whole system hopefully so in ideal world you all you need to do is just take your release push it to version control system and just sit back and enjoy your BLM and check the pipeline how everything's smooth okay let's start let's continue with the bad expectations what can happen so this is the new type of infrastructure so if you had like processes previously tools and script in place this will usually not work for your for the deployment with containers although you need new people to manage this or educate your former staff and I would like to talk about this a bit more so since containers I mean are so popular right now there are many releases in darker or in I don't know open-shift Kubernetes so these projects evolves very quickly so you need to pay attention what's going on in upstream and basically what features are being added and if you are interested in them you can even like contribute to the projects and polish them so it will work for you and at the same time there are also plenty new bugs being introduced and there are regressions in the releases and this is not something pleasant and that's what's also pretty common is that some of the issues are not very not very easy to suppose and reproduce for example there there was a bug in Docker that for example one in ten connections was like not established in your containers and how would you how do you resolve it or even reproduce like one of the ten and sometimes it happens sometimes it doesn't so okay let's continue with the state of the art like was the current state of the Docker ecosystem so I'd like to start with this quote it's by Solomon Hikes and he said it on this year Dockercon which happened I mean two months ago and it was on his keynote he said that nobody cares about containers and if you think about it it's almost I mean it's true so what we care about is that we care about system which is very easy to use and we can use it for deploying for developing an application and deploying the application so you have one system easy to use and you can use it for the whole life cycle of the application and the containers are actually just an implementation detail I mean it could be done with virtual machines and if the system would work fine I mean we would use it right so what Docker started doing like two or three Segar they're starting building like technology around containers so they are used I mean so they are easy to use but over the time they figure out that it's just not enough like they need to earn their money so right now they're building whole platform like whole platform for deploying for yeah for deploying applications and let's check the platform so I think like three days ago they finally released they produced final release of Docker 1.12 and let's see what's inside so the most like controversial big thing is that they put orchestration inside Docker engine so in the past Docker engine was like service which only could run containers on your one machine but since 1.12 is able to join the swarm cluster and it's able to like run on in the whole data center so you don't need a special service for that for the orchestration or for the multi-host thing so the good thing is that it's optional you don't need to use it and if you want to use it there won't be any overhead so just that's just great and it's also backfab compatible so if you upgrade to 1.12 it's not like that it will break all your applications with the orchestration comes the new API it's called service API so in Docker there wasn't any like there wasn't any elements for services so you may know this from Kubernetes there are like ports and services and right now they're also in Docker so services basically composed of several containers so you can basically have like HDBD service and the service and the containers itself might be running on multiple hosts and they have also another primitive to I mean the service has another primitive like replicas and stuff like that so this is also right now part of the engine and also finally I mean Docker also came with their own implementation of like multi-container applications so it's called Distributed Application Bundle like it's almost some like Docker Compose I mean Docker Compose produces the Distributed Application Bundle so and it's very similar to the Compose file format okay there's also new instruction to Docker files it's called help check and it specifies a command you can use to check healthiness of your application so if you have like web service you can put curl there and can check if your application is responding or not and then you can see the status of this in Docker inspect this one's very nice so in the past when you restarted Docker engine it stared down all the containers and they were gone or they were just up that wasn't very pleasant so with 1.12 there's a new demon flag called Light Restore and if you apply it whenever you restart Docker engine it won't stop all the containers so they will live happily even if you restart it if you even restart demon so that's very nice there was also split in packaging so with 1.12 there's not just one Docker binary there are now two binaries so there is Docker binary which is the client there's Docker D binary which is the demon and what's great that Docker 1.12 is available in Fedora since I think yesterday so if you have Fedora you can I mean if you have Fedora roll height you can try it out I also mentioned some issues in past slides so I'd like to talk about this a bit more so stability and issues so for 80% use cases you very likely you won't hit any issues and for some very specific use cases like for some esoteric networking or storage it's possible that you might hit something for example since Docker 1.10 they put a DNS server inside Docker engine and since then there was there were multiple bugs related to DNS and for example I know that some community members were running an application inside containers which were doing a lot of DNS requests and they were hitting a lot of issues so it's possible there are also lots of code changes inside releases there are there are multiple pull requests flowing through the project and they change a lot of code so this means that it's not I mean there are there are bugs inside these like code changes you can see it in release notes if you read release notes there's like 10 new features and 50 fixes to bugs so releases are done basically every two or three months so usually when there is a bug it's very it's very common that it's fixed in next release or even next minor release which is like a couple weeks after a major one so this is very good and as I said some of the some of the issues are hard to reproduce and even harder to fix so here's a list of some interesting issues I compiled from the issue tracker so I'd like to go through it rather quickly there is a name of the issue I take it directly from GitHub and then there is issue ID and so so the list is split into several sections so the first thing contains some race conditions then there are some issues relative to DNS and connections and then there are issues related to graph beckons namely overlay FS and device mapper so I'd like to go through it quickly so the first thing is darker demon hangs under load so this issue is what is open since June June 11 2015 so it's more than one year and the issues about that there's a race condition in Docker and some of the users were so the issue is that the Docker stops responding like basically you do Docker PS and nothing happens and the issue is open for more than one year there were multiple like people commenting I have the same thing and still not being fixed so yeah as I said some other issues are harder to fix then there is issues related to DNS because in one dot 11 they to they employed inside the DNS server some trades so for example you could only do like 100 requests per second and if your application does more like they will be denied so some of the users were issues with that then another then another like area which is had issues and they are easy to interact with device mapper I mean there are issues with that so and finally fuse overlay FS I mean it's really fast and really great but it's not very stable so for example if you are using Python packageer and you are using overlay back end so if you install Python package in lower layer and in the upper layer if you try to update it it won't work because the big packageer will try to remove the previous installation but since it's in the lower layer and you can remove it it will fail so but I think that this is being fixed in colonel so in your version colonel you shouldn't encounter this yeah so this list is just it's not so it's not so hard to encounter an issue with Docker or containers so this is basically last slide of the presentation summary so for me the hacker engine has won containers I mean they developed and like so they made using containers very easy and everyone loves it so they have millions of users and billions of downloads of images and they definitely want it but what's happening right now is battle for orchestration so there are multiple orchestration systems and none of them is like one won the whole market so some of them are good for this some of them are good for that and that the battle has just begun and it's really up to you to pick your platform what suits your workflow the best and before even doing that I advise you to learn about the platform learn about Docker and pick what suits your best so this is further reading so if some of the issues or topics I discussed here are described even further so you can read it and this last slide so so that's a link to my GitHub repo with the slides so they are completely open source and if you get in touch with me that's my Twitter handle and if you have any questions we have plenty of time to discuss anything yes so I decide to file an issue of it how can I help them to resolve it I mean can I take the corner of Dr. Demon is that is that useful do I have to analyze it for that for example so I see like new teams or state of channels messages are in use or are there any tools for like big building applications in production because if you are deploying stuff into production you should be able to debug the stuff and in case that your application depends on a lot of her because your application yeah that's very good question I was actually thinking of putting a line how to do that or what could help but I haven't so I'm glad you asked so I won't answer for the whole goal line because I don't do rolling and I don't know and for doctor specifically so what you can do is that whenever you encounter an issue and you'd like to send doctor like some meaningful data which they can use to analyze the issue and fix it you first thing that you should run doctor in debug mode she's just doctor demon dash capital D and the second thing is that you can send sick user one signal to the doctor demon and then it will respond with the whole trace of all go routines so yeah there will be stack trace of all gold routines and then they can use it to analyze the issue and and ideally fix it so so I know that in redhead like there was a new answer already it's like a couple of weeks or months ago and it had some like improvements in case of doctor so there are some new ansible modules for that so I mean you can use that to basically I mean the modules are able to like create the doctor image without doctor files so we can use an ansible playbook to create a doctor image she's really great and then you can use another ansible module to deploy basically so I haven't tried it much I mean I play with it what work it was working I liked it but I haven't used it in like production or extensively so I can comment on that does it answer a question or yeah okay thanks yes please yeah that's that's good question and I okay so I feel like they are trying to push for doctoring production like running containers in production basically when you go to doctor come usually one of the first questions to the audience is who is who is using doctor for development so basically the whole audience rises their hands and second question is who is using in production and for example every year there's more hands being like put up I feel like that with latest releases they are trying to go for containers in production but I mean as the time goes it feels like that the platform is more stable so there are more bugs being fixed and so I feel like that it's stable with it more stable with every release but it's I mean that's not really an answer to your question and I don't feel like how could I answer it's like yeah yeah yeah yeah and that was like two years ago so it should be even more production today there are very large like 4,500 companies running doctor production scale yeah I'm yeah yes please since we're talking about the production what's the correct way of copying data inside some container for example some databases okay so so ideally you have the database that data in a volume and you can pick a volume plugin to manage it so it can be like on the file system or it can be on Blaster or some shared file system so definitely best thing is to put it on volume and ideally have the volume being backed up and on some shared storage or some like enterprise storage yeah I mean as I said I'm not operation so I don't know but my suggestion is put it on volume and pick the correct volume plug-in and store it in there so definitely don't put the database data inside the anymore questions System VIN Spawn? Yeah, what's your opinion on that? I've been using it for three years now and production was really good for when the hardware and honestly I was locked, I've fallen in love with doctor 3 years ago when the time passes and since it might be a one-and-a-half year I'm getting more angry and more angry Because from basic container rendering, it started to be a full shipyard. And the stability is getting lower and lower. Imagine. When we use your production, we had to learn by heart what not to do. And then we know how to use that in the stable manner. So we started to feed and to test system, the N-Spawn, which is basically a very tiny block without any porting installed, et cetera. And it actually looks to work perfectly right now. We are in this situation where we're trying to think about what to do next. Yeah, so I'm glad you shared your experience. Basically, I don't have any opinion of N-Spawn. I mean, I think that these two guys really appreciate your feedback because they are working on system D and they are part of system D upstream. So maybe they could answer your question better. I'm not sure if N-Spawn recommends using N-Spawn production in the market, there is a regret. He doesn't. I thought about it. He doesn't. But it works. But there is always a great serve. It's important because the original use for that was just the system D because if you are loading the system you, if you're running on your own, it will break it. If you're running on your own machine, it's just clumsy. So equal design is to do the system D. But what's, well, if it's about the N-Spawn spaces and C-groups and all the tasks and many of the stuff you need, so. Exactly. You are not the first one. We thought you were trying to do the system D because we're on the C-groups. Yeah, I agree. What made N-Spawn so much more exciting to not use everything for the storage of the N-Spawn here? So we want to split the, to run the containers which you can see for the application that it's trying to see. So this right here, N-Spawn, you can describe your container, you can use it when it strikes the first time. And at the same time, if you want to run something right on the N-Spawn, the most common system based one is actually working on a regular desktop. It's better than that. So it makes things much easier to develop. OK. Yeah, but this is definitely interesting feedback. Also, as Dan Walsh talked about his presentation yesterday, they are also planning to, like, not use Docker for running containers. They are planning to run C and on top of that, build a simple service which will babysit the containers. And that's it. So I totally understand that some similar service for running containers works for you better than the full-blown Docker. So when I have now a container, I would run it 50 times, just small one, not really that much. So that's what I would recommend. The latest one, or is there a stable version that you consider most stable? Yeah, that's a tricky question because it really depends what you want to achieve. I mean, if you need the latest features, then definitely the latest features. OK, so I mean, as I said, it's a tricky question. For example, I would definitely go for what's the latest release inside the distribution, for example, in Fedora or REL. So yeah, that's stable, and you can use it easily. Yes? Do you know maybe if Python API is stable now, I was trying it, and it was quite difficult to find a correct documentation in Docker with the DOS page, I think. Pardon me, what was the name of the project? The name of the project you mentioned, I'm sorry. I'm talking about Docker Python API. Aha, Docker Py, yeah. Is it stable or not right now? Yeah, we had the same issues, basically. When we used an older version, it wasn't working for us, then we had to upgrade to a newer version. And I feel like that with the latest releases, it's pretty stable, and they are also following. So whenever they put a new feature inside Docker Py, they basically, OK, what's it called? So they use deprecations and stuff like that. So whenever you use something which is deprecated, they warn you. And at the same time, you'll see that this feature can be used with this version of Docker NG. I mean, I feel like the latest releases are pretty stable, and I'm actually using it in one of my projects, and it's working well. OK, so the problem is the Docker API seems to get re-specified every six months. So it will be stable for a while until they deprecate it. And they do give you warning, but then in three months it's gone, so bring it to self. Yeah. Is that, I don't know if that. Well, it doesn't matter. The point is, there's been, like, Docker's existed for like three years, and they get re-designed for APIs six times. Well, you can do it. So version one, and then one dot one, and then one dot two, and then v2 happened, and then when v2 happened, there were two iterations of that. They're not compatible. So yeah, I mean, yes, the library is stable in the sense that they don't break it, but they've got changes often, so. So I mean, the good thing about their API is that they version it. So for example, right now there's, like, version 23, and you can pick the specific version, and it will work. So you can even do it with Dockerpile. You say that I'm interested in version one dot 20, and you can use it, I think, forever. But Adam is right that, yeah, they are changing the implementation details, and structure, and stuff like that often. And it's something that you shouldn't rely on, like, on their internals. But the API, yeah, is being changed often. And I asked, does ID, like a personal pet, because the API was not the same as the documentation set, so. Yeah, that's pretty common. Yeah, that's common. Actually, I have commits in Dockerpile truly because of documentation, because I had the same problem. I wrote something to documentation, and it blew up, and so I traced down why, and yeah. They were excited in the world, and they have matches in there because I tend to sort of pick stocks a little bit, but that's a hard one. But, sorry, Mikal, but whenever I hit this issue, I go to the API documentation of Docker itself, and usually it's, I mean, I haven't found any, like, anything there, any issues. So I use that one, not the API documentation from Dockerpile, or I browse the sources of the Docker engine if there's really something wrong. Sorry. I just installed Docker 1.5. I didn't notice that RPMs, like, 120 max. Yeah. And because of that, it was 140 max. I mean, system impact, which was 11 max, and people were like bashing us, that it's still too big, 11 max. And it has 140 max, it's bigger than the Docker in the line. So, both things, going is statically linked, and they bundle everything, and now they bundle swarm in there, it's like, they took, like, all these projects and just threw it together. What do you mean? It's a little bit, like, going, I don't know, right? Yeah, we do, but they're not dynamically linked. It's all, so, when you do the depth change, it just pulls it into the build route so that it statically leads them to the binary. I mean, the binary is static, it can run. Yeah, it's, I mean, it's, like, in the end, it's a native token right now. But the tool change, do we? We don't see it as static now. But, but, Seagull, it doesn't do that, I think. We don't provide one, I don't think. In one die, seven, I'm gonna sit right on the board and share that, but we do see it as static. Of course.