 Hello We're starting right at 215 All right, we'll get started here in about one more minute. All right, I think we'll go ahead and get started Good afternoon everyone. Buenas tardes. Como están todos? bien Great for those of you who don't speak Spanish. Good afternoon. How's everybody doing? Good. All right. Sounds like we're on the same page Hope everybody had a good lunch. Hopefully not too big of a lunch though I don't want everybody to take a siesta here. Well, I'm talking about some some interesting things here My name is Jared Smith. I've been in the Drupal community for a long long time Helping out in a number of different things Wanted to give a talk today about Docker and specifically how we use Docker in the in the new Drupal CI test infrastructure Before I get started though, I want to talk a little bit about photography because I I'm an amateur photographer I don't claim to be that good, but I have fun with it. It's it's cheaper than a hiring a therapist And in the world of photography, there's a saying that says that the amateurs worry about gear and Professionals worry about money and Masters worry about light And so in in my talk today My goal is to shed a little bit of light on the topic of Docker and then and then Docker's interaction with the Drupal CI test infrastructure I don't claim to be an expert, but I have been around the block a few times and Hopefully I can share some of the knowledge and light I've found with you guys here today I do like an interactive presentation. So if you have questions, raise your hand If if things if you're following along and things are making sense nod your head up and down like this Everybody practice up and down If things aren't making sense or if you have a question shake your head side to side like this That lets me know that I need to you know rephrase Something or say it another way or go into another explanation It also just lets me know that not everybody's asleep. So that's good I do I do like I do like lots of feedback. So please feel welcome to interrupt me with questions or comments or complaints or Suggestions or if you know this stuff better than me stand up and teach me something. That's okay, too We're all here to learn right? Okay, so quickly a rough agenda for what I want to cover today. I want to talk about a quick history lesson Talk about what is Docker? How does it work and then dive into the into the Drupal CI stuff? Figure out why why is Drupal CI using Docker? How are they using it and then talk about some of the warnings and pitfalls and gotchas of Docker because it's not all there yet But it's still kind of fun stuff. So let's let's go through that and kind of roughly that order But like I said, I want to start out with a history lesson Let's say you wanted to go sailing on a ship since we're so close to the ocean here in Barcelona such a beautiful city Let's go sailing. What are we going to need? A boat. Okay. We've got a boat. Here's our boat. What else do we need? We're gonna. We're gonna sail to Boston How long is that gonna take us? Two weeks three weeks six months on a boat like that. It's gonna take a while. So we need maps. What else do we need? Wind Some did I say did somebody say food? Food we need food, right? So how are we gonna get that food on the ship? Well, you're gonna wheel a bunch of stuff up to the ship and you're just gonna throw it over and some of It's gonna be in bags Some of it might be in barrels. I that's probably not wine, but that may be whiskey I don't know. We'll see now some of it may be just wrapped in canvas and Apologies to any vegetarians in the room for this next slide. I'm sorry some of it may not be in containers at all That's frozen meat Looks like a bunch of frozen pigs. That's right So that's kind of how things worked up until roughly the 1950s Every time you wanted to load a ship full of cargo. It was in different size sizes. It was bags or barrels or containers or crates or boxes or pallets Everything was different size and it took a lot of time to get that freight loaded on a ship Loaded a back off the ship. Maybe it goes on to a truck gets loaded onto the truck The truck goes to a different city gets off loaded off of the truck That loading and unloading took a long time Until this guy came along guy by the name of Malcolm McLean Malcolm McLean was a started a trucking company in the United States started out with one truck Before too long. He had a whole fleet of about 25 trucks But he always hated waiting at the port for things to get unloaded off of his truck onto the ship or offloaded off the ship onto his truck and he says There has to be a better way And so he came up with what we now known as intermodal containers. Here's your standard eight foot by eight foot by 20 foot container What did this do the shipping industry? Come on. I said I wanted feedback It standardized transportation. It really revolutionized shipping It cut shipping costs by more than 80% It sped up the you know the time that now things could get you know Onload it, you know put onto the ship or offload it onto the ship and onto a truck or onto a train and vice versa And it just made things standardized. So it was so much easier to do international shipping To give you an idea today more than 90% of all non-bolt cargo So if you're cutting out things like shipping wheat or coal or things like that that just go in big open big huge open Boxes about 90% of the world's freight is is shipped in in containers like this And they estimate that just since 1990 alone The the amount of cargo shipped in shipping containers like this has multiplied five times In the last 25 years It's growing just just just a little bit. So standardization is important It's amazing now that you can have a container full of T-shirts or shoes or whatever it is, you know manufactured Let's say manufactured in China and shipped to the United States and you know 25 26 27 days later It's there. It's unloaded at your front door and that containers probably never been opened So it's really revolutionized the way shipping shipping. It also has many of has improved Manufacturing there's a lot of companies that get their raw materials in containers like this And the containers will show up within hours of the time they need it for their production And then they ship their products out the door and containers just like that now on the way out So quite quite an interesting story now if you want to read more about it There's a great book called the box by Mark Levinson that is really interesting about Talking not only about how containers changed the shipping industry, but how it changed the world economy as well I suggest you may want to go pick that up. It's it's a better read than you'd think Now I have a particular interest in in containers and in Offloading and unloading containers onto ships because one of my first IT jobs while I was still going to the University Was working for a simulator company and we built simulators for these gantry cranes So this is called a gantry. This is the thing that picks up one of those containers It connects to the top of it in the corners here It's got a little mechanism that locks into the top of the container and we'll lift it And then you sit up in the cab clear up in this this little cab right here, you know Stories and stories and stories in the air looking down at these containers You got you kind of get this kind of a view looking down on the containers from the top picking them up moving Oh picking them up setting them on a truck or putting them from a truck onto the ship And in that sort of thing so so I had a lot of fun, you know Designing simulator systems to do this and so kind of kind of an interesting Passion of mine But obviously we know today that there's these big huge, you know container ships that that to carry these these containers around some of these Container ships will hold 15,000 containers And they keep making bigger and bigger ones all the time in fact part of the reason that Panama is expanding the size of the Panama Canal is because they can get bigger and bigger ships to come through the Through the Panama canal interesting so that kind of sets up the The story for Docker Docker is really the shipping container of application virtualization now What do I mean when I say application virtualization? We've probably all heard about Virtualization over the past five or ten years Whether it's you know operating system virtualization platform virtualization application virtualization The best analogy I can I can give is you know is that running in you know a single set of applications on a single box Is kind of like a house right? It's kind of self-contained. It's got everything a family would need When you get into virtual machines That's kind of like row houses or town houses where it's a bunch of houses stacked right next to each other But they're still pretty much self-contained. They don't have a lot of shared resources between one town house and the next When I'm talking about application virtualization, I'm talking more like an apartment building There's a lot of shared resources there. There's shared plumbing. There's probably shared, you know electrical Hookups there's probably you know some other shared utilities along those sorts of things what that does for the individual You know apartment owners is is it reduces their costs and they can share some of those costs with the other the other people in the apartment building There's also some drawbacks in if you know if the apartment building catches on fire Who gets to pay for it? Well everybody does you know you share the responsibility as well So the idea behind Docker is to really provide a standardized container For an application that can be easily moved to reproduced from one machine to another So reproducibility is one of the one of the key key things of Docker So I want to take just a minute and talk about kind of the architecture or the infrastructure of how Docker works. There's There's there's several foundational pieces that make Docker work and work well The first thing is is namespaces within the Linux kernel We have namespaces for for process IDs for the network for inter process communications for mount points for For all kinds of different things in within the within the Linux kernel What that this means to an application is it's it thinks that hey, I've got a user with ID 501 And that's great But in the in the host system it may be a totally different user number It may have different process IDs what the what that container sees is not the the whole world view It gets kind of a Microview of what's going on it sees you know its own view of the network even though on the host machine may have a totally different network That sort of thing we also leverage what are called control groups in the in the Linux kernel Who's familiar with control groups? Control groups are pretty cool They've come out in the last oh four or five years within the Linux kernel What that allows you to do is within the kernel say hey this this process over here Can only use this much memory or can only use this much CPU or can access this stuff over here But can't access this this stuff over here and so it's it's it's a very nice way when you're dealing with virtualization of constraining a Process or a user or or those sorts of things so that they don't take up all the resources that they can't take more than their fair share of The resources whether it's memory or CPU that sort of thing Docker also uses that what's called the union file system union FS which makes it easy to kind of layer Pieces of the file system on top of each other and we'll see we'll see more about the layers here in a minute And then last but not least Docker came came up with kind of a container File system format that they call lib container that it's a standardized format that you could take a container for one machine Ship it over to another machine start that container on that other machine and have your application run up and running very quickly on that Other container So defaults to using that container format. You can also use another container format out there called LXC that's used by the by the Linux kernel So that's kind of the that's kind of the foundational pieces here Let's see how those fit together Docker comes with a Docker host or a Docker Damon a server we're going to say that's that's right here. Here's the Damon. Here's here's the machine. It's running on and Then there's a Docker client that you can run from the command line to execute commands against that against that host Now it's kind of neat the way that works that client talks to the to the Docker host to the Docker Damon using simple rest Command so anybody in here that's done HTTP and done rest kind of calls The command line client is actually calling making rest calls out to a Unix socket to talk to the to the Docker Damon You can have the Docker Damon listen on T and a TCP port if you if you'd rather communicate across the network Now one of the drawbacks to that today is that there's no security your authentication or anything on that So you may want to be careful with that if you play with that That's something that's supposed to be coming Hopefully in the near future and Docker in Docker 1.4 Roughly, but the idea is that from your client you say hey Docker build this build me this container or build me this image Hey Docker pull this this image down from from some sort of external registry You know repository of existing images that people have created Hey run this run this image and create me a container based on on this particular image So these are the really that the three pieces you need to know about as far as real real objects you're going to be dealing with are images and containers and Registries, so let's go into a little more depth on each of those an image is a pre-built System that's a pre-built application Templates so to speak And so you may have one for something like Ubuntu or you may have one for CentOS or you may have one that's specific more specific to an application like engine X and Then what you do is you use a Docker pull command to pull a copy of that image down to your down to your Docker host And then you do a Docker run and say I want a container based on that Particular image so you want to say hey, I want to I want a new Ubuntu image You say Docker Run that image and it would create you a new container based on that image so the image is kind of a read-only template of The container and then you can instantiate as many as containers as you want based on that image So you may have four or five or six or two hundred containers all based on that one Ubuntu image in this example does that make sense? The registry like I said is just a is a is a place that you can go out and pull Existing images that have been created or push your own back up to the repository Now the Docker community has one that they call the Docker hub That's kind of the central repository for all the Docker images But you can certainly run your own in your own infrastructure and I'm going to strongly suggest that you probably want to look into running your own For reasons I'll talk about a little bit later in the in the slides But the idea is that that if you don't already have you know an image created the way you want or you want to get an updated Version you would go and pull that from the registry You could create your containers if you want to make changes to that You can also push it back to the registry if you had the permissions to do so Any questions up to this point? Not seeing a lot of nodding heads, so I don't want to make sure people are falling. All right Now now is the part where everything goes up in flames because I'm going to skip away from the slides here for a minute And then I'm going to show you some real-world examples and hopefully this works And then maybe we'll have internet access in the room and that sort of thing and we can have fun with that Sound good. All right, nothing like nothing more dangerous than standing up and doing a demo live, right? Okay, so the first thing I'm going to show you is if you run the docker Images command this is just going to show me the dot the images that I've already downloaded here to my local machine And so I've got a bunch of them. Is that big enough? Can you guys see that? Okay? You can see I've got a bunch for Drupal ci and we'll get into exactly what each one of these does here in a minute When we talk about the Drupal ci piece of this but just down at the bottom you notice I've got for example I've got fedora 22 I've also got one called fedora latest got one called hello world And if I wanted let's just run that hello world one. That's that would be a good example, right? If we wanted to run that We would just do docker run hello world And what happened? It took that image created a container out of it ran that container all the container just Spit out some information and then and then exit it The idea of these containers is that they run one application They start that application. They live as long as that application runs and when that application stops running The container goes away. It's done disappears Okay, so in this case a hello world example is pretty trivial All does is print some things to the screen and exits and the container is gone Not that exciting right? What if we wanted to run a container and And you know leave it running in the background and interact with it that sort of thing So let's do that. Let's go ahead and I'm just going to use one of the ones that comes from Drupal ci since it's it'll be lots of fun Let's do docker run Drupal ci My sql 5.5 Okay, I'm going to add a couple of options here First thing I'm going to do is do a minus D to tell it to run in the background run as a daemon When I run that what happens it just spits out a big long number any any guess as to what that is It's a container ID. It's an identifier so that when we go docker ps to see what what containers are running You'll see that the the beginning of this that this container ID here is the beginning of that actual long Long hash there. It's a hash of the value right now. I can by default each of these Containers is also going to get a made-up name unless I specifically tell it to to give it a name So here's here. This one's called goofy right if I created another one I've got one called elated brown and goofy right Each of these containers you you start is going to get its own name or you can assign your own tag Your own name to that if you want to you know, give it a more useful name than an invented name that that comes from Docker You can you can see what's going on with one of those containers by doing docker logs and you can either give it the name in this case like goofy right or I could give it the the The the container ID or the first several characters of the Of the container ID and that would work and this is telling me what's going on with that In the logs what happened is that is that container ran and it looks like if we if we scroll back a little bit It looks like it started up It installed some system tables in my sql it killed so it filled up some help tables It printed out some information We connected and we were unable to connect for some reason here Jeremy's going to figure out why later Then we were able to connect successfully. We added a user called Drupal test bot some some commands Happened some some grants some permissions were granted and now my SQL is up and running Okay, how can we tell that it's up and running? We could accept we didn't tell it we didn't tell the system how to map the network port inside this container To a network port on our host system. So right now my SQL is running, but traffic is not going to get to it So let's kill these off and then we'll start them up by by specifying a port So again, we can do Docker ps to see what containers are running Docker kill That one Docker kill that one and again tab completion is a beautiful thing It just works. So now Docker ps. We're not running anything Let's go ahead and restart our Container here this time. We're going to say connect port Since I'm already running a local copy of MariaDB on my laptop here on port 3306 Let's put this on 1306. That is that sound and Connect it that the 3306 on the on My SQL 5 5 does that make sense? So we're going to tell it that's where that's at Again, we can do Docker logs except we don't know the the idea of the of the container. So Docker ps Ecstatic McLean. Hey nice that it chose McLean is the name remember Then the McLean guy that built the shipping container. That's kind of cool, isn't it? Okay, so now we can say Docker logs Ecstatic McLean. He's ecstatic to know I'm talking about him. We see that my SQL is up and running Let's try to connect. So if we do my SQL minus H 127.0 to 0 local host port, what did I say 1306? The user is Drupal test bot And I'm not going to tell you the password Although it's easy to find And with any luck here Why didn't sorry say that again? Let's let's find out on 3306 Huh that worked when I tried it when I was building the slides Let's try one more time here It doesn't doesn't matter with my SQL Let's try this Yep, I do it all the time Let's try that. There we go. And as you see it's normal standard out of the box MySQL database So arc it the container that Drupal CI That we use in Drupal CI it installs my SQL it Creates that user it grants permissions to that user All is part of the container startup and I'll show you that here in a second when we get into that way How does how does Drupal CI use that does that make sense? Okay, so that's In a nutshell to show you how you can get up and started using using Docker But Docker is no fun to just use if if you're depending on somebody else's images, right? Let's show you how these things actually get built So I'm going to log out of here I'll go ahead and kill that Container no more containers running Okay So I'm going to start out with a very very simple example to show you how Docker containers and images get created Typically when you're creating a new Docker image you create it by creating a file called Docker file And a Docker file has has several different parts I'm going to try to walk through those in a couple of examples here The first thing you'll typically see in a Docker file is a line that begins with from in all capital letters See that up there. I apologize. It's cutting off the first letter there. This is from FROM and this is saying what what base image Do we want to start with in this case? It's saying start with a base base image of Ubuntu trustee You could say Ubuntu latest. You could say Fedora 23 You could say Fedora latest you could say CentOS whatever whatever kind of base image you want to start with But typically people start with an operating system as their base and work their way up from there kind of makes sense, right? And then There's another line that says maintainer the idea of the maintainer line is just put your name in there or some way to recognize Hey, who's responsible for creating this Docker file? Who who should you go yell at if it's broken? The next thing you'll see is is one or more commands that start with run These are the things that docker is going to do to build this Particular Docker image so in this case it's going to go download the Ubuntu image and then as part of the run command here It's going to do an app get update app get install Apache to app get clean and remove some temporary files Okay, so what's that going to do? It's just going to get that machine up to date and make sure Apache is up and running on the box or installed on the box at least That's what those run commands and you can have as many of those run commands as you want But you want to group things together that belong together like everything you're going to do with App get should probably all be as part of one command and that's for for the caching reasons I'll talk about here in a minute when I talk about the caching layers and Docker The next thing we've got an example of here is the ENV Command those are just setting environment variables. So in this case We're setting an environment variable of what user and group that Apache should run as What's the log directory for Apache? That sort of thing the exposed command here Says which port is going to be exposed to the to the host Damon in this case. We're going to expose port 80 None of the other ports on this machine will be able to be connected to from from outside of the container so that's kind of your your Poor man's firewall for for lack of a better term It's a way of only exposing the network ports that you really want exposed out to the outside world And then right here we have cmd command and this is the command that's going to run This is the application that's being you know containerized. So in this case, it's user been Apache 2 And then anything after that is just you know Arguments or parameters that are passed to that that command. So in this case, it's running user has been Apache 2 Minus capital D foreground to run Apache in the foreground When Apache dies if Apache were to crash for some reason what's going to happen the container goes away. It's done Okay, so whatever you put in your command here is what's going to run and keep running for the life of your container The question back here so So so the question was here. I only specified one port. I only specified port 80, but when I when I when I instantiate this image How do I how do I assign that to a port on the on the dame on that on the on the host? Right? Is that your question? Right? You can Right, so so here is what's exposed here But when you actually take this image and instantiate it into a container Then you would map hey I want port 8,000 on my local system to map to port 80 or maybe you have two or three of these running and maybe you want one On port 8,000 and one on port for port 8,001 and what that one on port 8,002 You do you create those mappings at the time you create a container This is just the image that it's going to use to create those containers, but if it's not exposed here You're not going to be able to map to it when you create the container. Does that make sense? perfect All right Any questions up to this this point on this docker file? Not too complicated. Oh, here's a question Yes 80 colon 80 yeah 80 map to 80. Yep. That's correct Okay, so that's that's a very simple example Let's let's do a little more in-depth example here to get a better feel for for docker files And since I like postgres I'm one of these crazy people that runs Drupal on postgres. Let's Jump in and look at maybe a more little a slightly more complex docker file here Again where we're saying from Ubuntu in this case just did another tag out there 12.04 Again some some kind of identifier for the maintainer to see who maintains it We've got some environment variables. We've set here. This is just so we can specify what version of postgres Do we want what's a user name and password we're going to create for that user And then we're going to run this this set of commands here We're going to set up some of the lists for the sources for for app get We're going to get the the public key so we can make sure we're getting something that's been Cryptograph graph cryptographically signed by the postgres community We do an app get update to update the system app get install postgres and postgres contribute for that particular version of postgres And then again clean up after ourselves That those are those are very similar to what we've seen before just a little more in-depth because we're adding a Debian repository from from the postgres community Here's a new command that we haven't seen before called copy and what that's doing is saying copy a file called start dot sh From my local directory and inject it into that container You call it start dot sh on in inside the container So that's a way of if you have configuration files or that sort of thing that you want to copy and have them be inside of the Container when the container starts up Yes, yes, there's a working directory command that you can also use if you want to change into a different working directory before the before the this command runs That you can use so so in this case we're saying hey just copy the start dot sh file over And then we're going to run run another command to remove this this policy that be File and then we're going to actually run that start dot sh command that we copied over Does that make sense now one of the things I want to explain is at each one of these steps here The way Docker works is it creates a cache of Each step so if you run this and you've already got the aboom to 12.04 image on your system And it's not going to download it again. It knows. Hey, I've got the latest copy I'm just going to use that and continue on then it adds a new layer by just changing the maintainer And then it adds a new layer for setting the environment variables Run and command. Okay, so run are things that get set up to create the image Command is actually what what command runs when you start the the container Yep, so as each one of these commands executes whether it's an ENV command or a run command Docker when you build that image it's going to go run that and then it's going to capture a snapshot of that and create a Cache layer of that so if you had if you had two different Docker files And there were only a couple of lines apart most of that's going to be cached And it's really just differences between those those at the end similar to how many people here use get And you do get commits and and you see just the differences it's same same sort of idea here It's just automatically creating an implicit commit between each of these commands Which is which is kind of cool, and if you want to you can go back and look at let's just do Docker history Let's just look at that my SQL 55 container again Drupal CI my SQL 5.5 and you can see You know what what what were those different commands that happened? What was the size of it? What was the? Over here you've got the the image ID. What's what what's what was that image? What was the the identifier of that image at that stage and you could even roll back and say hey I want to roll back to what this was you know four days ago at this particular stage in the in the process If you wanted to so you get these these layering effects that really help out with caching and reducing the amount of Bandwidth that's needed when you're copying around a bunch of machine images Does that make sense? You could Yep So you could try it. Oh, no that change broke everything Let me let me roll back you just do a docker tag and one of the previous image IDs And it would just roll back to that To that stage and then you go back and fix your docker file and do another docker build and rebuild the Rebuild the thing so let's let's let's show you what it looks like when you actually build an image from one of these docker files Let's go back to our Apache example. Just as just just as an example here, and if the internet works we should be able to do docker Build and the directory where that docker file is and it's going to go out there and Do those steps you see it says step zero Step one step two and you notice it it Creates a new a new image ID for each of those steps that it's going to cache So we'll let that run here for a second and then we'll run it again to show you you know How it's going to use those existing caches? I'll take it a second to do its update and install Apache 2 That's doing the setup and Just just by way of information one of the nice thing about containers is I'm not running Ubuntu on my laptop here I'm running Fedora on my laptop here But I'm running a containerized version of Apache that happened to come from Ubuntu So it makes it very easy to get applications, you know cross across operating systems across platforms Which is which is kind of fun? Okay, so that's that's that's out there and built if we did a Docker images And if I scroll up to the top you see now, I've got one without a name here Those created 17 seconds ago. That's that's that that one. I just created now. We should probably give it a better name, right? So let's do grab that image ID Copy that We'll do a docker tag Give it a name Now when I do docker images Hey, look It's got a name of J Smith HTTPD now that's that tag is just a human readable name You can you can put anything you want there. What really counts is the image ID I could create three or four tags that all pointed at the same image with different names if I wanted to That's just that's that's just a human readable name. Yes Yes That's correct two containers that are that are that are you know file of equivalent should be have the exactly the same Okay, so So that's that's that now. Let's that now. Let's go build that again and see what happens It's done. Why? Because it's got all those different layers cached and when you do Docker history On that image you can see what each of those Worry what are the each of those layers are that make that makes the makeup that docker image Make sense question over here So I was going to talk about that more when we get into security But yeah, we can talk about that now it's updated until you do a doctor a docker pole Oh, I need to repeat the question. Sorry for the recording So the recording was how long does this get cached if I do an app get update today? That doesn't mean it's going to be you know running tomorrow to get the update So it's it sits there in cash as long as you and tell you do a docker pole or a doctor Rebuild your docker container in your docker image if you if you're building it yourself In the case of the ones in docker in in Drupal CI we rebuild them every few days. So you see like These are a little older than that four weeks ago But typically every few days will do a rebuild or if we see that there's something that's that's updated in the upstream In the upstream image and then then you get to go yell upstream and say hey Why isn't your image updated in four weeks, you know, it's so We'll get into that a little bit more when we talk about security But but the basic idea is this is going to stay cached here until we either rebuild it and it does another app get And it says that sees there's a difference and so it'll you know, it'll change that or you know until you clear out the cash manually If you wanted to blow away that image it's a docker our MI for remove image and it's going to go Remove all the unused layers of that cash if it sees something is still using one of those layers It won't remove it because I'll say something's still using it and unless you force it to anyway That makes sense Okay, so that's That's a little little you know crash and burn demo. Nothing nothing went up in flames too bad so now let's talk about the Drupal CI infrastructure and how this plays into everything I Came in a little bit late to the Drupal to see I group and and what they were doing But I learned early on when I just jumped in and started helping out Is that they had two really big goals for for Drupal CI that kind of set it apart from the old test bot system? Goal number one was to be able to test a lot of different combinations of PHP Different versions of my SQL or Maria DB even point versions of PHP Maybe maybe you know the reason I got involved is because I wanted to test against Postgres I wanted to get you know Postgres support improved in Drupal 8 so it's like I'm gonna make it so we can at least test against Postgres Then we know when it's failing and when it's not then we can make it better So so testing across all these combinations and you kind of get this this this big huge matrix of all the different versions of PHP and all The different versions of the databases and and that sort of thing, right? One of the other core goals that that the Drupal CI team came up with was that they wanted to make it easier for Developers to be able to do their own local testing without setting up a whole a whole box or whole system To do Testing on on their own system. I'd like being able to test things right on my laptop I don't like having to depend on anybody else's infrastructure or the internet or whether Amazon's down today or those sorts of things I like maybe it's the Boy Scout in me, but I'd like to be prepared So so that that was one of the goals that came out of Drupal CI and Docker made both of these goals a whole lot easier So what we did in the Drupal CI community is came up with with kind of a layered Approach to our Docker images we created Docker images obviously for the different versions of PHP and the different versions of the databases But we did it in a layered fashion such that we have a base image that everything is based off of Beneath that there's a PHP base For for the PHP even for the different versions so we can you know instantiate different versions of PHP underneath that There's a web base and then that the individual versions web 5.3 5.4 5.5 5.6 web 7. Yes We are able to test Test things on PHP 7 believe it or not even before PHP 7 is quite out yet So that's that's exciting and then same thing with the databases. We have a database base Image and then we have minus go 55 Maria DB 55 Maria DB 10 Postgres 9.1 and Postgres 9.4 So we can actually go out and test across all those Different versions and if you don't believe me just yesterday I was sitting in the sprint room and tested a couple of things against both my SQL and Postgres and look I could go Schedule that test. Oh, I want to test against PHP 7 and my school It's there today it works mostly No, it really does work and it's and it's working surprisingly well So kind of fun stuff So let's dive in for a second. Again, let's get away from the slides and let's actually go dive in to Let's see. I'm gonna make that just a little bit bigger. So you guys can see is that better? So what I've got here is I've just checked out the Drupal CI test bot get repository So you can see you know kind of how we how we've done things in Drupal CI with with Docker If we go to the containers directory You'll see that there's a base directory a database directory and a web directory again. The base has all our base Docker images and so base base. We look at the Docker file there. It's pretty straightforward We'll walk through it quickly. We're gonna be based on the boon to trustee The maintainers Drupal CI are the ones that created this They've said an environment variable called Debian front-end for non-interactive. That's makes app get a whole lot happier Knowing it's not waiting for this there for somebody to press yes and acknowledge a bunch of things It's got a bunch of things commented out here as far as which mirrors to use and whatnot for a boon to we'll skip that for a second And then all it does is do an app quick an app get clean app get update app get install unzip vimtiny wget Doesn't app get auto-remove and an app get clean and that's that's all that base image does just the base You know things that we need we know we need for all the other layers So that's pretty straightforward Type today If we go to our Let's go to our PHP base for a second look at that docker file. It's not too much more Too much more confusing again this time We're pulling from Drupal CI base that previous docker file that we looked at as the base and then we're from there We're adding a second environment variable called home. Let's just set to slash root We go ahead and install a bunch of things that we need to compile PHP We that's lots of fun We do a get clone of Project called PHP and V that lets us set different, you know, which different versions of PHP which in PHP environments we want to build We go ahead and make a make a small hack to Make sure it's compiling with more than one core so that it works faster because faster is always better, right? And then we go ahead and build PHP we install a composer we installed rush Supervisor D we copy some scripts across that we're going to need for later and then we Start running this start dot sh that just brings that Brings that environment up and running Not too complicated, right? And then if we if we go down the rabbit hole one level deeper and go to our web base We'll see that it's it's based on the PHP base We add Apache a few other things here We remove the PHP 5 CLI version that was in there before because we're going to use this this this PHP and V version that we build ourselves We remove a couple other things here. We copy some some some configuration files over for our virtual hosts We we modify the Apache configuration to create our virtual host and set up, you know that we're using npm and Enable the that Drupal virtual host and that's that That layer right there finally last but not least We can go into our web directory what a web Let's look at five dot six And we can see that it's based on the on that web base that we saw we enable five dot six seven We're also looks like we're installing Mongo and APCU and upgrade curl and a few other little things that we need just to make things run nicely We copy up a bunch of configuration files over Dot I and I files for PHP and then we start that we run the start dot sh And what the start dot sh does is starts Apache With with the appropriate version of PHP running and make sure that that's up and running for us to actually be able to run our tests So again, it's kind of like this seven layer burrito dip, right? We get seven sis several different layers here But they each build on the on the layer before that and that in a nutshell is what the Drupal CI docifies look like now just for the fun of it. We'll jump back up here. We'll look at the database. It's just very very quickly Show you that there's no tricks up my sleeve. It's just really really easy Install my SQL server and that cat remove a few things clean up clean up But that's it. That's that's how terribly difficult it is to create a my SQL image again That makes sense Question You do you do Docker like if I if I did Docker build on this Docker file it would pull that DB base image if it didn't if it wasn't already on my system It would pull it from the from the repository from from the Docker hub No, no, the whole lamp stack is not on one. Is that is that your question? Okay? The lamp stack is not in one container We we have we typically have Apache and PHP running in one container And we have the database running in another container and then we then we connect those two containers together So Yes, this is saying that this this is our my SQL container and it depends on the database base container So anything that we added in the database base container would be in this container as well Sorry the image. I'm using the wrong terminology here the image Right So So so so let me let me take a step back and go back to this slide here to right here So this image, let's just pick the my SQL 5 5 container Or the image, excuse me. I'm messing up again the image the my SQL 5 5 image is just a single image But it's built on layers that were provided by the database base image And the base image Okay, so let me just show you how that that works. So if we go and say Docker history Drupal CI base we'll see that That latest version is Again, it's chopping off the first letter that was 7c 8c Right there, right? If we do Docker history Drupal CI DB base You'll see that 7c 8c that same image ID is here and That the database base just adds one layer on top of that Okay, and then if we were to look at Docker history Drupal CI my SQL 5.5 If we scroll up We should see right there 7c 8c that that that layer there that was the database base and then the my SQL images just added these layers on top of it Additional commands that were run now that creates one image. That's my SQL 5 5 You could create as many containers from that image as you want But that's just one image That's that clear now. Okay. Hopefully I start using the right language here It's hard. You're talking about images and containers and they get messed up in your head Okay How are we doing on time? Couple more minutes here, and let's go back to the slides and talk about just a couple couple other things here So I showed you I showed you kind of the the code and how it works You know Docker files aren't too terribly difficult to learn Troubleshooting them can be a little tricky at times. I think we've gotten pretty good at learning some interesting and neat ways to do that We're happy to happy to share those with you offline Just jump in the Drupal testing IRC channel and and and ask Jeremy or myself or Ricardo or one of the other guys and And we'd be able to help you that Docker versus vagrant. So so the question is is why when would you use vagrant? When would you use Docker? And when do you use both? again We're talking about application You know virtualization here rather than than virtual machine You know the machine or operating system type virtualization You would use you would use at least for me I use Docker when I want to have one application that's standalone that's self-contained and it's just that application What I use vagrant for is spinning up a new virtual machine to test a bunch of pieces of software and how they fit together And sometimes it may be I fire up vagrant to fire up a virtual machine that pulls in this Docker container in this Docker container and does things across across both of them one once you know Operating system level, you know, virtualization you're virtualizing the entire operating system and one you just are you're just virtualizing an application There is some overlap there and and it can be confusing and it can be tricky Especially what we'll talk about security here in a minute from a security standpoint You better understand what you know what layer you're doing your virtualization and how you're going to keep that up But does that help answer the question? Think think think think of think of Docker applications like the little Lego blocks and you're gonna stack those together typically use vagrant to To make it easy to reproduce You know a kit of those that you can plug together to build something a castle or a ship or an airplane or whatever Whatever you want to build out of Lego blocks It varies it really is Exactly Yeah, yeah boot if you're familiar with boot to Docker, it's just yeah for all intents and purposes. This is a figure Just with a different name. All right, let's give you some warnings here I don't want you to get too excited about Docker because it's not perfect. It's got some words. It's got some Some soft spots Try to warn you about some of those in the next few minutes before I wrap up here First of all if when you go and and pull an image from the from a registry if you don't specify a colon and what version you want Like you saw I went out and I pulled Like Ubuntu colon trusty or Ubuntu colon 12.4 if you don't specify that tag It's going to default to just pulling whatever the latest version is which is what we all want, right? And tell a new version of Ubuntu comes out and our application isn't ready for it and what happens My images don't build anymore Crud that's not what I wanted It also, you know from a from a security update standpoint may not be what you want Because typically you want reliability you want hey I built this image today and I built this image tomorrow And it's going to be the same except for maybe some security updates, right? But if it goes and pulls the latest version of Ubuntu, and it's a completely different version Suddenly you don't have consistency there and then you're wearing an okay to vote Do I have the security updates or don't I have the security updates and it gets confusing? so I Always recommend that you specify Which specifically which version of a of a you know of a particular image you want Now the flip side to that is that means you have to go and say hey when I want to upgrade to a newer version You have to go do that manually and rebuild your images and you know if you're using containers build off that image You probably want to restart those containers But that's you know that that that that that's that's security piece number one The next question more oh question before I give the big reveal here So if the image maintainer is not doing their job Then you're still out of luck you're right and that and that's that's exactly what I wanted to bring up with this next slide It could be it could be It's just a name for the image now in the case of the Fedora and the Ubuntu images on on the Docker hub They're they're they're actually official images from those communities But in general, what's the difference between these two commands up here on the on the slide? Is there is is there really any difference between these two? nope Docker is great in everything if you trust the internet Do you trust the internet? I can't tell you how many how many open source? How many open source projects out there as this did have something like this as their installation instruction? Just trust this shell script We'll get to that on the next slide Okay Docker Docker runs his route Anyway, let me talk about this is one more reason why you may want to run your own registry Just be careful with it again because there's no no authentication on the registries yet. Okay next slide Containers aren't bulletproof in fact Dan Walsh the guy at Red Hat to who worked on se linux and the big security guy there He's famous for saying containers don't actually contain It's not all that difficult to break out of a container Sorry, they don't contain then they kind of isolate somewhat, but if you're in an apartment building Can you punch through the wall and then get to the apartment next door? Yeah, it's possible So so you want to you typically want to use things like se linux expert Those those those types of security add-ons to the linux kernel to help provide more isolation between those Between those so yes, it is running in a you know in a container, but There's there's a number of ways to break out of containers, which which is unfortunate, but but it's still not bulletproof All right point three about security. Who's keeping your your your images up to date? Who's keeping your containers up to date? You could be pulling the latest images, but if you're not restarting your containers based on those newer images You may have an old version running out there Docker themselves did a study earlier this year, and they found that over 30% of the Docker images on the Docker hub on their registry over 30% had serious security problems whether it was heartbleed shell shock You name it So again It's nice that these things are out there and available for us to use but do your homework find out whether they're being kept up To date make sure you're keeping your stuff up to date Let's not make the internet a more dangerous place Just just because this makes it more convenient for us to do some work all right We've got a couple more minutes for questions comments complaints rotten tomatoes One here and then there if we can get you to step up to the mic for the questions That means I don't have to repeat repeat the question and it'll be better for the recording Thank you Yeah, it's regarding running docker in production, right so Versus vagrant regularly you won't probably use vagrant in production. I hope not So would you recommend it would you do it? And how do you for example keep your containers running, right? if you have a standard Apache installation then all the processes and Init systems are there to run it after reboot or whatever so right um I Used docker in production, but I use it in a limited fashion in production If you've got a lot of docker containers that you're running and you want orchestration around those There are open-source applications like Kubernetes that are that are specifically built to orchestrate Make sure that these things stay up and if if this container goes down start up another one And if this one can't be reached and started started up over here on another pod and those sorts of things There's there's orchestration pieces Short of that if you don't want to go to all that trouble Then I would say you manage it like you do any other application you monitor it And if you're if you're monitoring system tells you they hey Apache isn't it has gone down Then you then you go start start it back up again But yes, I hope you're not running vagrant in production because that's not not not the right month Yeah, hi, thank you very much really good presentation really enjoyed it. This is what I was looking for Particularly like what a local development environment that was because I have a Mac I think docker runs on a small v8 virtual machine. Does is that right? Now the thing that interests me is I use vagrant at the moment for local development What the biggest problem I've had is the performance Particularly things like if you're doing building Drupal views and so I was looking for docker to provide that Isolation that you get The VM gives you but but more lightweight is that does that sound like a reasonable approach or yeah I'm not the world's expert on on Mac site as a former Fedora project leader I'm kind of kind of biased towards Linux anyway, so I run Linux on my machine And I'm not it's been a long time since I've used it back But from everybody I've talked to that's kind of that's kind of say running say a docker application like your web developer site on in a docker Container on a Mac is potentially better performance than running that same site in a vagrant provision VM Is that would that be it really can be because you're not loading a whole nother operating system? Yeah on top finally, um, I spent you already know this tour is chit-matic It's a UI on the Mac for building docker not familiar. Okay. Sorry. That's all right Thanks very much super presentation. Thank you. Thank you So I was curious how you were bridging obviously you showed on d.0 a interface for I want this web version and this db version How you're taking that input and then instantiating those actual containers if you're generating a docker file on the fly that is It's it's not it's not an image that combines those two. It's just when we fire up a test bot. It says hey start this docker Container start this docker container connect the database here to the to the web one here And then starts fire off the tests on the on the web container Jeremy can give you more details on how that works He knows that system better than I do but Jeremy just raise your hand So anybody anybody has questions about the test system Jeremy's your guy and there's docker compose now Which makes that it's easier to reps that you know YAML file. It is Just a reminder for folks that don't know we're doing a docker tools throwdown Thing buff right after this in 130. I think that's what 345 So if you're interested in this if you have more questions about even getting set with Mac and stuff We've got people are a lot of people are already using this for local development like myself. So Cool. Thanks for thanks for reminding us about that All right, any other questions comments complaints. I think we've got about 30 seconds left Hi. Yeah You were building your whole lamstack in One image do you usually do that or build different images for different services or right? So I I build different images for different services. So in this case, we had two images We had one for the for the for Apache and PHP. We had one for my SQL or Postgres, whatever the database is I like to split things out separate service per image Cool Right remember that docker runs a single process. I'll pair it Jeremy here since he doesn't have a microphone So right now I will I will say it's bad practice To have like supervisor D or in it as your process that the doc that the doctor campaigner is running That's the whole I mean the whole reason of doing you know container or application containerization is because you want to run one single application Not because you want to run a hole in a knit system and a whole bunch of daemons behind it So Typically you want one container per application or one image per application per service Yeah, it does happen that way. All right with that I I live you leave you a sweet sweet dreams of walking on the beach That's sunset looking at gantry cranes Oh, yeah, they're they're right down there I didn't I didn't take this picture, but it was a picture of Have something similar. I think I think it was local here. I'm not I'm not positive, but Thank you