 I'm Sean this is Chris and we wanted to talk about something that we've been playing around with honestly in the past year or so that we're kind of excited about we like playing with different technologies and seeing how they can solve problems we have and so we kind of got on containers and Docker and we kind of wanted to share some of the experiments we've been running and we weren't sure what kind of level of familiarity with it like frankly a year ago we knew nothing about these things either and we're still not experts now but this will largely be tailored to the level of familiarity with it within the room so we kind of did a little poll before but I'll do it again because there's new people so containers and Docker in general if you're who would say you're like completely unfamiliar with them you've maybe heard the word but I've never used them before put up your hand please okay so that's the majority anybody actually using containers in production okay neither are we so that's okay and if you've maybe like kind of played with them done some you know done some experimenting on the side like you've at least run one Docker command of some kind okay a couple people over there okay so we're going to significantly tailor our presentation this morning to to that so please please bear with us so we'll spend a little bit more time talking about what containers are what problem do they solve why did we get so excited about them so is it fair to say many of you came here just hearing about this Docker container thing and you're like what's all the buzz about this okay fair enough that's that's basically how we got started wasn't us very long ago so hopefully one of the things we can demonstrate to you how easy it is to learn about this technology and become proficient with it and there's some great tutorials online and like pretty simple ones to you like the cost of entry is pretty low if you're going to invest some of your own time to learn this stuff so I'd highly recommend go like Docker.com has some great tutorials so we unfortunately can't won't show all of all of that but I'd recommend after this take some time to to go through those because they're they're pretty simple and pretty good and it's a fairly fairly shallow learning curve it doesn't take too too much to get the basics going so I think I'd kind of like to talk about what problem are we trying to solve why did we get interested in this one of the ones I know personally was I like to play and explore with different technologies right you're like oh Django I've never used Django before but I kind of like to see what it's about oh a new version or Nginx what's engine you know how can I use Nginx as a reverse proxy you know names so there's all these different different things out there and lots of them and oh Jenkins has a new pipeline feature in their thing I want to kind of play with that and so you kind of have these like discardable experiments right like I want to try some technology learn a couple things about it and then at the end of the day maybe I don't like it and throw it away so you want to be able to do that quickly right you don't and you want to be able to throw it away very easily at the end and so this was a problem personally I always had was by the time it takes you to download something you know install it on your computer configure a bunch of stuff and then you decide you don't like it and you try to install it but it leaves a bunch of stuff hanging around and it conflicts with stuff in the future and now your hard drive is a mess it was an impediment to learning I found so one of the big reasons I was interested in this is like oh it makes it really super easy to try new stuff so I was I think that was the first problem and I'll let Chris talk about kind of where he got into it so if you didn't see the keynote a lot of what he was talking about was separating the sort of what we do what we're trying to build from what it runs on right we spent a lot of time sort of configuring systems and servers and this keeps us from actually spending our time on the real like business value work that we do as developers so I'm gonna walk you through one of the examples of what got us interested in this and this isn't a deploying say like a website into production or services into production but this is more of a config management type problem so for those not familiar with that so when we build code usually we have source control and CI server so developers will check code in and we'll have a build server that will build the code run all the tests and then if you're using a continuous deployment pipeline you would actually then ideally deploy that into production once everything's passed so this is a sort of typical thing we've seen is this is like with Jenkins you'll have some Jenkins master machine that's got all of the configuration of what's gonna run and then you've got these different build agents and a lot of times we'll be spending a lot of time configuring little nitty gritty details on each of the machines right and I've heard them given little pet names like like they'll have these long complex serial numbers and somebody will just call it well that's the Zed machine you know that's the one that one just acts up all the time we'll just just reboot that one when it's not working we'll have this other machine that we affectionately call the R machine because it starts with the letter R and you know this one we put some little special packages on it they're kind of annoying to configure but you know what there's this this one project we barely every bill ever build it so we don't want to spend time configuring this on every one of our build agents and then we've got this you know extra machine and it's it's the one that's set up just right it's perfect it never fails and it's in theory the template we use to create new build machines so what happens is we spend a lot of time configuring these machines rather than figuring out how do we build our code efficiently and develop spend more time actually developing software this was a phrase come up by a guy at CERN it's this concept of cattle not pets so pets are things that we you know we give them names when they get sick we nurture them back to hell we know we take care of them we look at them fondly cattle are things with serial numbers we we can you know grow grow the number of cattle we have very quickly and what we want to get to a point is where basically if if our systems are getting sick or we want to add more systems that it's it's not a lot of extra worker configuration so Docker one of the benefits that Docker has that we found is it allows you to configure machines and so this was a tool that we this is one of the tools that we're going to talk about later in the presentation basically it's a tool that allows you to decouple the configuration of your build pipeline and the way you build things from the systems and servers that run on them so we want to spend basically no time configuring our build agents at all we just like to be able to tear them down and spend them up at will and Docker is one of the tools that allows us to do that yeah and environment parity in DevOps is a really big thing as a as a as a programmer you know you always run that well it works on my machine doesn't you know and we kind of make fun of that like well your machine we will we'll ship your machine to the customer that doesn't count but it works on my machine should mean something and I wish I could quote who said that but works on my machine should mean something we should have environment parity we should be confident about environment parity that the environment that I'm working with is the environment that Chris is working with is the environment that's on test is the environment that's on prod and the differences that do exist are very well-defined well documented configuration managed between the environments and I mean Chris and I were recently working on a project where it's like okay well we need to introduce like some really obscure old technologies like I think there was I can't remember there was some old stuff in there yeah there's some yeah and there's some old Perl scripts and other things like what we don't want to have to make every developer go through like all and getting Perl configured with an Oracle database and all this kind of stuff it was it was a complicated set of things you had to do and to then and this is for kind of like a side like a rather small project and so then to have to try and get an entire development team to all go through the steps you know precisely and not like anybody makes a mistake and we get it was a bit of a nightmare so this is one of those environment instances where you do it once the configuration management now you can have all your development team use the same environment and then you can think of one example we were playing around with an ETL technology called Talon and we're trying what we're trying to do is figure out a way so we can run our ETL jobs inside of a container and deploy them into a swarm and I'd spent a little bit of time it was like a few days to configure all these things and get them set up just right and it was coordinated with like a GitLab server and a bunch of things and then Sean finally he had come back from vacation I think it was he's like oh I want to catch up to where you are so he pulled the the Docker compose file and the configuration he had onto his computer and I think you were up and running in two minutes yeah and this was like a complex set of different servers and daemons like there wasn't just one thing it was like this is one of those things where if you wanted to deploy this on like an actual production server the old-fashioned way I mean we would have had to put an IT ticket and wait two weeks or four weeks and and it meant that we could have these reproducible environments very quickly so these are some of the challenges and problems that interested us in kind of experimenting and learning more about this so I'll give it maybe a brief introduction into what are containers because as we've been talking you might be thinking well they sound suspiciously like virtual machines that's not untrue it's not exactly untrue and Mitch who is standing here a few minutes ago talking about the work he said hashi corp he was the original author of vagrant and there's a lot of parallels in many ways vagrant does for virtual machines what docker does for containers so if in many ways he provides some virtual machines and containers provide similar thing in the sense that they give you reproducible environments but what are some of the challenges we have with virtual machines is one the images are very big because you actually have not only the kernel but the entire operating system all the entire file system there's a lot of other stuff that goes with it so you rarely see a virtual machine image that's less than a few gigabytes the other issue with virtual machines is they often take a long time to start up you're starting up the entire kernel every single time so the boot up time is often several minutes the other constraint with virtual machines is that because they're running on the hypervisor they have all the resources kind of pre-allocated to them so you don't necessarily get an efficient use of resources because you have to allocate each one for their peaky load the maximum amount of memory your CPU usage is going to use but if you've got a process running in one of those and you want to isolate all our processes 90% of the time it might not be doing that much and so you've kind of got this kind of inefficient resource utilization what containers have done is they've essentially run off some kind of old kernel technologies but kind of they've packaged together in a new way so instead of virtualizing all the hardware through a hypervisor you actually they all run on the same kernel so they share the kernel but they have complete file systems they have completely different file systems they have completely different process spaces they have completely different network stacks and so as far as the application running in a container is concerned it doesn't know about anything else and so you can have it can have its own dependencies it can have its own software and they don't interact maybe less so I know there's some non-technical people in the room big advantage the images of these things small and they spin up quickly that's good summary so the analogy is the analogy is back in the day with shipping you could find the shipping analogy there so let's say 200 years ago if you wanted to ship a good of some kind maybe you've got some food and then maybe you also have some you've got some you know oil containers you've got a piano and so all these different types of things you want to ship and they've got different forms right they physically look different they've got different sizes weight shapes and so you'd have to figure out well how do we stack how do we stack all these things on a ship or how do we stack all these things in a on a train and so you have this problem of you've got different forms that the goods come in and then you've got different forms of transportation and so you've got this end-by-end problem of you have to figure out how do we get these different things all across these different different modes of transportation and so the solution to that was the shipping container which is all the goods go into this one standardized box that's an international standard and then we design our modes of transportation such that they can all take this this the standardized box and so now your your truck your train your ship your rail all carry the all they need to know is they need to carry the box they don't care what's inside of it on top of that boxes don't interact without the box you can't ship the monkeys next to the bananas but what if you've got monkeys in one box and bananas in another box they don't interact and so when you put them in a when you ship them it doesn't matter so what does this mean for us in the software realm well before where you had to worry about like okay well I've got I said that the environment on my own machine the environment on production the environment on tests there's all these different environments how do we be sure that we've got environment parity between them and also let's not have to worry about interactions when we don't need to let's not have to worry about dependency problems and conflicts for things that have no need to really care about each other so going back to Chris's example with continuous integration you've got a Jenkins machine for example well now you have to install your build tools you have to install maybe some test frameworks and tools on there now you have to install maybe some performance testing and maybe some you know UI some third-party libraries well they might not necessarily need to know about each other for example your your build tools don't need to know or care about your maybe your automation frameworks your runtime automation framework so why all kind of put them on the same thing and have to worry about conflicts and dependencies like they that's that's try and separate things and have them very small and atomic as much as possible I'll come I'll show you I'll come back to all this so one of the interesting things about the way that Docker works is it's actually we don't just take an image and ship the whole image so you can actually go up on to Docker hub and like you can pull an engine X image or an Apache web server image what's actually doing is it stores the information in the container image in what's called a layered file system so what this allows you to do is we can add new information so you can take your base image you can start adding on your your your application and HTML files and all these different things so actually the additional things that you add are very very thin slice on top so what that's what you can see here is this is a base of going to image it's actually made up of multiple layers in the system so this is all what's the phrase item potent so the image itself is part of the container so there's a very there's a different sort of terminology thing here a container is a different thing from a container image so the container image is the read only part of the file system that you can ship and share with different things and once this tiny thin read write layer is once you deploy the image once you run the image this is the ephemeral pieces of information that are changed while the things running so do you remember in the keynote he talked about I think it was three different types of state ephemeral states sticky state and persistent state so containers a lot of time are ephemeral state they're state that you can launch do some activity and then tear it down so you're not persisting the data and if you go back to the original diagram here when I'm running a build process I pull my source code I compile it and then I deploy the compiled thing while I'm compiling it that process there's no reason that that's that that state needs to persist after I finished my build process so this is one of the real advantages of containers as I spin this thing up the actual read write portion of the container gets processed and then we throw it away lost our so like we have like lots and lots of different containers that are running so that's that's a very challenging problem we'll talk a little bit about it I don't know that we're going to go into that in detail but in terms of I mean like it running in a production environment how do you manage them all like that's what law would with Kubernetes and schedulers do yeah we'll do a little bit of a demo with registries so how do you actually have a central registry of what are all our containers might say internally with the company two separate problem one is the container images we've got a whole bunch of different images so that would be something you'd store in a container registry and the other one is then your your running containers themselves you've deployed them into a swarm you've got lots of different running ones that do talk and communicate that's so that's the technologies like Kubernetes and the first one is easier it's just a it's more or less a source like a similar like a source control problem soaring your images so you just named inversion okay the connection between the build process so sorry pointing at the screen okay so so we might have an image that describes how to build a piece of code so it's got all the dependencies to build the code in right when I actually build the code I then launch a container so I could have many different like deployed containers doing the sort of this is the ephemeral information the stuff that we wrote out when we're done so each one of these so like so say like multiple people checking code we may actually want to like independently build each of those revisions so we can actually show you that the build process will like launch one container to build the first revision of the code other containers build another one to build another one so each one of these will have different versions of that code in them and then when we're done it can deploy individual versions of the output we could probably get to doing yeah let's let's start doing some examples it'll start making sense yes you reuse the image yes so I think I think this will make more sense once we show some actual examples first talking please there's a lot of echo yes absolutely so that's a very good so the question was we do have to persist data at some point how do you do that we'll talk a little bit about that you can mount so the main solution is you can mount a volume an external volume into your container to allow you to persist data but one of the big principles that we try and follow is to have your when you're persisted data in one place so you have your your ephemeral processes that you can scale out as a separate problem to solve and where do you store and persist data but there is a mechanism to do that so this is com so docker is probably the prevalent technology container technology at the moment and the nice thing is is they've actually built the whole ecosystem around it so we won't kind of go through getting set up and installed but please go to the website and they've got it they get super easy to go in for developers and you can download on windows linux mac and they've got tutorials to get started and what i'm going to show you is hub dot docker dot com and so that we mentioned before about a container registry well this is kind of the github of the container world which means that people like open source if you create an image for something you can publish it to docker hub so there's all these publicly available images and docker actually maintains official images for lots of the really popular technologies and so for example here i'm going to say postgres so if you want to find a postgres image there's actually an official postgres image and it will actually give you a lot of information on on how to use it so you can go down here and you know read a lot of details about how to deploy the thing but fundamentally it's a pretty simple how to start an instant stalker run so this is actually a perfect example that we ran into so where when we're dealing with etl stuff so if you're i don't know if any of you are that do etl stuff but a big challenge is repeatable environments between dev test and prod and we were trying to solve this problem where well when i test something i don't want to have to go and get an it ticket and do some like complicated process to configure a new database server when really all i need to do is run some small unit test on a piece of code so really what i want to do is like spin up a database server very quickly and then tear it down so it's this is this is actually an example of like an ephemeral database it's not there for persistence it's this just there for my tests um we used a postgres docker image for this so you just run you run the postgres why don't we do it right now so i just uh they've got how to start a postgres instance i'll just copy this so there's this docker run command i'll give it some name i can refer to it later no we're we're going off script uh so i'll just take this and i will post or paste this into uh my command prompt and okay so the first thing it's doing is it said it could not find the image locally because i have not used this image before on this computer so it immediately goes to the public docker hub to say can i find this postgres image and then it says okay what are all the layers in this image in this layered file system and it begins to um it begins to download them uh download them here a bunch of them say already exist so this is the this is this layered file system we're talking about so a postgres image is not a postgres image it's actually some additional layers that have postgres sitting on top of some other um probably i'm guessing a bun two or something like that right so what you're actually saying here is they said already exists because we already had that on our system and now it's just downloading the incremental information that it needs to know to run postgres and so um there we go now um yeah now i have uh postgres running and i could just docker ps so ps to display the processes and i can see that um it doesn't format very well on the screen but i have one container running and it's off the image postgres and it's been running for 10 seconds and one of the one of the um problem we were trying to solve is we want to be able to spin up a database server with a specific schema right so it's not just we don't just want to we just want to like when we're running our tests be able to spin up a database server we want to spin up a database server that has all of the tdl scripts already run on it and then one of the things we can do is now we can create our own image that has an additional layer built on top of that that now has the um the ddl already configured so now somebody wants a database server that already is configured with these tables and and store procedures and all that they can spin that up without any extra um work so they don't have to figure out how to like pull the creation scripts and set it up with all the right configuration and we were doing this with oracle which had a lot of really annoying small little configurations we had to do so we so one of the things we're going to show you is creating a what's called a docker file that describes the um basically all of the commands that need to be run to configure a container image so the nice thing if you go on docker hub um and want to look and see how to get started you see at the very top here uh you'll see um for any for any image on docker hub so we're looking at postgres right now but they'll have uh they'll actually have what the docker files are and these are configuration managed on on github so uh what they'll do is you'll you'll commit to new docker file on to github and then it will trigger a build of a new image on docker hub and so we can actually look at a docker file here um to see what the docker file for postgres uh looks like and one of the principles that mitch talked about was declarative right we want to say we want to tell the system what to build not how to build it so the docker file is an example of telling what is the configuration of what's going on in here we don't want to tell it all the nitty gritty details under the hood of how to uh just ddl yeah but you could if you you could build that into i mean and and this is where separating the actual data from a container is you want to have a little bit of a separation too and we can talk about that after um i think for the sake of time we're going to start actually jumping into showing some examples right yes yes so you do need docker um if you're on windows there's a thing called docker terminal um windows 10 so before windows 10 it'll actually install a linux vm onto your computer and you need the docker daemon running um um we we've dealt with legacy systems and what we've found is that we've been able to wrap an entire legacy system into a container um then you could start breaking it apart and building it into smaller systems but um maybe that that's a maybe more complicated topic we can i'm more than happy to talk to you after this presentation so just very briefly there while we were talking i i stopped the postgres server and just to show you once you've got all the images pulled how quickly it is to start it up again so i'll just go i'll run the docker run command again there's a couple people that have flipped out i i really wish i could we tried before and we we couldn't yeah i we did try before we started and we weren't having any luck so it's the boot up time for images like on the order of seconds um that compared to like if you were to do a virtual machine so um i want to just show you quickly like other examples of other images you can get on here so like if you're looking at um so go back to docker hub and uh just examples of different technologies if you want to play with them um how easy and quickly it is so there's an nginx if you want to look at a nginx as a hosting server again it'll give you the how to use this image docker run and it will give you a tutorial how to run it um if you want python and we ran into this one uh recently okay yeah so i'm just showing some examples of different um official images so here's a image for python um again and how to get started so you don't have to necessarily install python on your local computer you can just pull an image for python um another one would be uh you've got an Apache server on here too uh so pretty much name any kind of technology and it exists as an image somewhere so now it now to try something it's just a one-liner more or less a one-line command to to try a completely new technology about licensing um Oracle doesn't use it on docker hub we're able to make our own but now we have to deal with things like um how many licenses i think there's actually we look for the Oracle licensing we want it to deploy a bunch of Oracle servers into a form and then i think they're they're licensing language with something around you can only have uh or the Oracle for us with like one one thing running per computer but you do need to because it doesn't make it for you you do need to like still manage like how many of these things are running oh this is just Oracle Linux sorry not that yeah let's move on okay so uh first thing so what we're going to do is we're gonna uh actually now we've put some time getting a bit of a background is go through a demonstration of building a very simple web application and deploying it in a container now you've got this nice box at the end and then this moves all the way from your local development environment into your production your production environment so the first thing we're going to do is we're going to build us on top of engine x just as you know pick a web server you could use Apache we're going to use we're going to use engine x so i'm actually going to go to docker hub and just look up what the engine x repository is or the images and look and see how do i actually run this thing and let's give this a shot to go doctor run and i want to expose a port so port 80 which is inside the container i actually just wanted to expose it as port 80 on my machine so you can do this port mapping so i'm just going to say port 80 should be mapped to port 80 and now just uh i'm going to run it in daemon mode and engine x okay so now if i just run to my ip there we go okay so now i've got a web server running um and now and that's that but a web server on its own is not useful right you want to actually have some of our own uh we're just going to do like a basic html file running on the web server to display an example but you could obviously make it more complicated uh so we'll just make a we'll just make a very simple web page and uh say uh so the worst web page well it's not even html but whatever it'll work um so now we need to get this web page into the container and what we want to do is really extend the image so images kind of have our containers kind of have this uh hierarchical inheritance based system so uh we can't just use engine x anymore we actually want to extend engine x to include our own our own source so i'm going to create a new file and and the convention is to call it a docker file um so i'm going to create a new file called docker file and there's this domain specific language that they use and and you can go online to look at the reference uh the first thing we need to specify is what image are we inheriting from so it could be ubuntu it could be so just like a very starting point um which has nothing much in it except for ubuntu or you could go from uh something that already has things installed so in this case engine x and so we do that by saying from engine x so it'll pull that image and then it'll start we can start adding things on top of it um some of the things we can do is we can actually run commands so if we need to install extra stuff like maybe an oracle driver or we need to um add python you can actually run and now you can just have access to any of your linux commands so like run app to get install you know and you can you can install whatever things you need to install in this case all we need is the web server so all the steps that would normally go into like setting up a like setting up a server you know like it's a pain in the it's really a pain to go through and have to type all these commands out all the time and then tell your friend to get them to do it in the same way it's basically more or less a script for how to build this repeatedly uh so in this case we don't actually need to install anything what we're going to do is just copy our one html file that we've made into the file system on the container so uh the declarative parameter is copy and we'll say there's public html index dot html and we want to copy that into uh there we go so we're just going to copy this index html for my local file system and when we build the container it'll now take the nginx base image and now it will copy this file in and now it's all packaged together and so we'll do an example of that we'll go docker build and uh i'll go dash t to give it a name so i can it's easier to refer to it i'll call this agile in the uh sort and now i'll just specify build it from the current directory and you can see what's on the screen it we already had the nginx image so from nginx and then the next command was copy so it copied uh my html file in and i just want to make sure i don't have it still okay so uh one interesting thing is if you don't specify your the name of your running instances it'll just pick something uh like a like this so i don't want to stop hungry Edison it'll randomly pick one for you okay so now let's uh build our run docker run docker run map port 80 to port 80 and i want to run what did i call it agile in the uh uh source and there you go so now we'll go back to our red page and uh there we go so there's an example of how we packaged our dependencies and our application together that doesn't look right you get the slides backed up okay so you don't obviously you don't want to like manually be building in the point you think all the time right so one of the presentations we're going to work here to talk about is actually doing this seamlessly inside of your continuous integration pipeline so what we're going to show you is using a tool called concourse that will build so so we're going to check in our configuration of our system the source code we'll show that we're going to pull it build it deploy it into a docker registry that will store so the image the image that he just built will we will actually store that inside of a registry so in your in your um i think one of the common ones is like nexus adws has a container registry system and then we will have a step that will actually deploy it into production so this is probably the least secure or a safe way of deploying software we're not going to be running any tests we're not going to be running any security checks but what we're going to demonstrate for you is we're going to be able to go from changing the file checking it in it's automatically going to build it it's automatically going to push the container image that gets built into a registry and then deploy it into production and it'll all happen seamlessly and so when we're done the first thing Sean's already done is you built this this engine exit and you deploy it uh we're going to have a i think i lost where i was so Sean so we're going to build a git lab repository so this is actually some place that we can store our source code we're going to show you how to persist that so one of the questions was um these these containers are ephemeral so we want to actually persist our source code our source control system so we're going to have some mounted volumes uh we're going to build concourse it's going to run inside of containers so concourse requires three separate containers to run um web is the container that your web browser talks to that's the shows you the cool ui interface it persists to a database and then it has the workers what's the workers of the individual build agents that run and build the source code um this is a docker registry that stores the images we're going to have that running inside of a container um this is swarm so this is a one way of deploying services so that they can be discovered and then the last thing finally is the thing Sean already built which is the engine x um web server that has the very simple uh sort of hello world example and then eventually ultimately we'll be able to demonstrate a full continuous deployment pipeline that'll connect to you from your web browser so we started here this is already done Sean's already built this he's already written the docker file that describes it now he's going to we need we need to persist our docker images the same way we persist so now he's going to create a registry uh image he's going to launch that and we're going to push the container image to build into the registry right yeah it's kind of like a source code repository exactly yeah so like our company we work for um has like a central nexus server um and it's got like namespaces and all sorts of different things that people can click all sorts of images right so we showed you docker hub which is the public one but if you're building your own if you're building an application you're not going to necessarily push your application to the public docker image repository so it's this is something that would be part of your own infrastructure internally um so what we're going to do is set up a large amount of infrastructure just for the purpose of the demo here to kind of demonstrate how easy it is to do but reality you'd have your own source control you'd have your own uh repository and these would just be things that you would run and maintain within your company and you wouldn't have to do every time but we're going to set all of this up as just a demonstration of how easy it is to use docker to spin up these things um just so we can uh build a deployment pipeline for you the wall shot is running getting the registry running i want to talk about one example i talked a bit about the example of creating the image that has your dvl on it so i what i've seen a lot of times is people um what's the definitive source of what your database is uh structured a lot of times i've seen it's actually the database itself if somebody wants to make a change they'll go make the change on the database server i think that's a really really really terrible idea um especially if we want to then do that in multiple places right so what i would do to suggest doing is having the dvl in source code you could create a build step that builds a container image pushes it to your container registry so now you've got a container image that has and you can make it version that your because the dvl of your database is going to be version so now you can have a container image that anyone can pull in your company that has the configuration of your database so that everyone doesn't have to go set up their own individual database with all that do it go to the registry have the name of that image have the right version pull it and run it and then they're set up going to be they don't want it you don't want them to look at their docker file you want them to actually pull the image itself but one of the things we could do with font course is you could have the so once you once you check the code it once you check the new dvl in you can have it auto build a new image and auto push it to the registry um my experience is not very mature it's um i mean i i give when i give microsoft a lot of credibility for for sprinting to catch up with the curve on this um on course does work with windows containers i haven't played around with it john you played a new experiment with the windows containers maybe you want to comment no so yeah so the question was can you run a windows container on linux and vice versa the answer is no because it's it's actually running on the kernel um so windows container run on windows linux containers linux uh however john uh discovered when he installed windows 10 because windows can run on a windows 10 yeah they actually have a like linux vm built into windows 10 now yeah um um yeah so this is the big advantage is you can figure one once you create the so this is complex configurations are great use cases because it might take a lot of work to get all of those those individual packages and things set up into your your image um but then it's there then you never have to do it again if i think i'm not answering this question yes like yes most of those are being scripted now we found like you can you can interact with them through the command line fewer and fewer pieces of software oracle included um absolutely require you to do it through the ui there are ways to script them yeah yeah yeah is it yeah there i'd be very surprised um how many pieces of software especially in the server environment absolutely require you to go through the ui to configure them yeah yeah this is a little bit like oracle's a great example of they're not embracing this technology well they're they're they're getting there they're getting there um starting to make it seem to do all that in the context um so we're just gonna move along with the demo so you saw at the end we have this like really complex set of containers that we're running so shan's creating what's called a docker from both files that allows you to spin out multiple containers and then have them uh talk to each other yeah so rarely we just have like one thing we're deploying right like the one web server it has to connect to a database it has to connect to um you know another service that's running for authentication so you end up with these stacks of different things that all need to be connected together certain ways so you saw i was doing all the docker run commands before with all these you know commands well that gets kind of tedious after a while so chris and i really like using what's called docker compose files where you can actually compose a bunch of containers and and declare how they run together as a as a stack so in this case what we're doing is basically building if you were to start a software company today and you need like while we need source control while we need we need to register you need all these things um what we're doing is really declaring the entire stack for one of those so you can see we've got um this is a docker compose file and it's a yaml format um so one service i want to run is a docker registry or my own internal docker registry and fortunately there's an image for that um so i can just specify what i want to call the service i can say what image i should pull from uh where it should use and then what ports i want to expose um git lab we need to have a place to put our source control if you don't have one already but in this demo we do so uh i want to run a service called git lab this is the image um i want to use and i'm specifying a specific version number um i can specify some environment variables we want to pass in in this case what the url uh is going to be for it exposing ports and getting to the question earlier around persisting data uh we want our source to control or we want our source to kind of stick around you know if the server shuts down it'd be bad if it was ephemeral so in this case we've actually defined externally mounted volumes that we've given names to so that these will stick around beyond the lifecycle of just the container itself um and so if everything goes according to plan i can i could call stop okay so the uh once i've defined my stack of different things i would like in a docker compose so i'm going rock docker compose up um and i've apparently formatted something wrong something does some typo okay mall programming see the typo oh this is it indenting indenting perfect so we just put up a registry and a git lab server and the git lab so the obviously obviously you don't want your source control server for the ephemeral right so we've mounted the persistence of the source control to volume on the system just taking a moment to for the actual git lab to get going okay so why don't you push your your image that you built to the registry oops one button no 40 minutes left okay no we're about halfway through everyone stand up get your blood moving and if if you're fine i'm sure some of you in this room are finding this going like way more basic than maybe what you're expecting i will not be offended if you need but you guys close the door i'm getting a lot of um noise from outside i can't hear the question maybe i'll come down to your room i'm not really that familiar with cloud foundry um but i suspect it is actually the um tooling we're using concord is built by the same people as cloud foundry um so you can push container cloud foundry yes the volume is just on its laptop um i mean you can mount the volumes externally one of the things we were we were doing is we were using aws and we had a container um form basically um and we mounted the volume rather than mounting it to like a be an individual vm like these containers have to run on a vm but what you can do is you can have a sort of cluster of vm that you can use a like load balancer to figure out where to deploy the container to you don't just want to mount the information like you don't want to mount on to like this individual vm i don't once again the vm is the same principle cattle not have i just want to be able to throw out those vm instances that the containers are running on so um there's um there's efs like amazon has the elastic file system so we can actually mount to like another external like a like a network drive for jankin i think so i haven't tried that that's a good question yeah i think i think you can that's the question was can you use the container as a node for jankin let's find out okay so yes okay so the first question was around security one of the security considerations is secrets so you actually i would say a absolute best practice is you don't want to store like like server passwords and and ssh keys like inside of your images you want to separate i think that's what the keynote talked about vault that's essentially what vault does like with hashi court produces one of their products is vault and it's secret management and how do you get secrets into containers there's a way to when you want the container to inject secrets into the launch container but you absolutely don't want your you don't want someone to just be able to pull your image and automatically be able to connect to your system that would be a security consideration um the other question was around why do you want to have your own internal hub for the images part of that is um you want you all things you're writing you want them to become publicly available yeah right yeah you might i mean lots of companies are even like totally open source so you might want it but generally generally yeah so there's there's quite a few technologies um like aws has has a container registry um you can just spin up your own so like there's a like what i think we're doing is there's an actual register container that is a registry that you can spin up nexus is the other product we talked about that's one of the ones that our other companies work we work for the even nexus so there's the interface to them is identical the push and pull is just the same as you do with doctor cups are there other technology like registry technologies that you know there probably is one i think are some of the ones yeah yeah so there's doctor's own there's nexus there's um yeah the one you mentioned aws is registry um the nice thing is the nice thing is is that the api is standardized so it's it's largely agnostic to which registry use so while we've been talking what i've done is uh now have gitlab running inside a container and now i've created a created a repository so we can actually put our source code somewhere i also have a docker container registry running on my local computer so now we have a place to push the images to you so if we go back to the diagram chris had earlier yeah we have we have gitlab with persisting the data to volumes and then we have the container registry running locally on the ocean what i would like to show is what i was typing there which was the stalker compose file i think the one thing i'll just point out is here's the gitlab server we're running now and okay yeah so here's the persistent volumes we've we've added to things so chris will kind of explain where we're at and then what the next steps are web server image we've got our registry that we've launched we've got gitlab that we've spun up we've done we showed you this where we've mounted a volume within the container into a folder on our machine and next what we're going to do is we're actually going to spin up concourse the concourse is this open source ci tool where everything in concourse is built on containers we're going to run it inside of a container you run all of this jobs inside of a container and basically it's built from these concepts all these all these concepts we've been talking about are our first class citizen inside of the concourse technology so one of the things we've seen is jaykin machines like we said they they tend to build up these like little individual configurations everything in concourse is configured inside of a container image and run inside of a container so there's no there's no state that sits around inside of these machines all of the configuration lives in your source code repository right so if you want to add more build agents in you don't have to configure a build agent very much at all all the configuration is held in a what we call the pipeline file that defines for the specific code repository what are all the things you need so if you add more developers you want to make things faster all you do is you just add more concourse nodes and it scales and it means that you can have per check-in you can have per check-in builds and you add more developers and it slows down you add more nodes and you still get per check-in check-in builds and they don't interact with each other because they're using containers so the three basic concepts and everything's built extremely generically so it's not built off of any specific source code technology so the first one is these concepts a task is a sort of basic operation that runs inside of your CI pipeline in itself is a script basically a script that runs inside of a container a resource is basically anything that has like versions you can check that you can like pull a version or push a version so the simplest one is your source code you can check for versions in your source code repository you can pull versions of your source code you can push new versions the there's a question i believe is a docker registry basically the same thing as a source code repository for images it is exactly as so our registry is another resource in this concept so there's a whole bunch of them this is not working oh right there's a whole bunch of them that are built in automatically so get being the most commonly used source code system you can create a time resource that will basically you can like pull versions of it so you're like oh is there a new is there a new version of time it's a way to trigger events you can say like trigger an event every hour that's three buckets cloud foundry was one we somebody asked a question about cloud foundry so you can push versions of your image into cloud foundry it will like deploy it into production and the way that concourse is built is you can actually configure and build your own resources so one our a lot of people in our company use tfs so we could in theory build a tfs resource that you could interact with and pull and check versions then the last one is jobs so jobs are these these types of actions that are triggered off of a resource change so a new version of someone's checked in a a new version of code into the source code repository a job is gets kicked off and it's basically the assemblage of a bunch of tasks so an example would be building your code or running your unit tests building a container image these types of things so when we're done we're going to have resources so these are reason to the two resources that we're that we have in our system we have a source code repository and the docker repo the docker repo being the registry that contains the images and these are the jobs um i thought that was funny so the two jobs that we're going to run is um take the docker um file build the docker image push it to the docker repository the second job we're going to run take the docker image and deploy it into production okay so shan's going to actually set up the concourse system he's going to add it to the docker the docker compose and then we're going to spin up our concourse system right so what we need is those three the db the web the worker those containers that make concourse we need to deploy those three containers and so we're going to add it to our docker compose file and what i will do here is show you i'll show you what i'm doing i'm going to uh cheat somewhat for the sake of time um which is uh there's just some environment variables that need to be set up for uh concourse to know um but for the sake of simplicity we're gonna bypass that so that's actually going to be three separate concourse containers and this is just the way um concourse so concourse itself um the developers have built it all containerized itself so they've created it essentially like three it's like like microservices right you've got three concourse services that interact with each other you've got the web server that our web browser so when we see that that ci pipeline that beautiful ci pipeline that's all running on the web server you've got a database that persists certain things um and then you've got the workers individual workers are their own individual container and those are the ones that you can scale out so you can spin up n an n worker so that's kind of that like microservice architecture where you keep things small it's it becomes easier and easier to deploy things in own containers instead of having one thing that does everything and they just connect like so in the one of the things in the in the docker compose file is he's he's told the containers how to talk to each other so there's essentially like a little bridge network then all these containers run in and a little dns server that runs in so you can like name the individual containers and then tell them how to talk to the other containers so concourse is uh now running um and this is the page you get when you when you first see uh first see concourse and says no pipelines configured um so we do need to kind of tell concourse what what the pipelines are and there's a command line tool that you can download to to do that so i'm just going to download it in the directory i want um and then we will go through and we will create our our first pipeline so the pipelines the build pipelines themselves are code as well right we want to have everything configured in code and check into our source code repository so when you when you pull a version of your repository of your source code it doesn't just have all the source code it has the information now because it's got the docker file that will describe the thing that's going to run you can have docker files that describe half the thing that will build your source code and you can have a pipeline file that describes how to orchestrate that so has anyone ever been in a situation where uh you had to fix a bug on something that was built like like four years ago and you go find the source code thing but and you and you pull it and you try and build it then you find out that like the environment has changed like the the development environment has all changed and it's different and now I have to go hunt down like what versions of different things were installed on equal machines um all the ideal is if when you pull a version of your source tree everything in that source code describes everything that is needed to reproduce the build that you originally had and this includes the configuration of the build servers as well so what I'm doing now is um I've used this uh you can see here I've used this fly CLI tool to actually interface with the concourse server so the first thing I did was I just logged into it and I declared a target so now I can I called it light but I can refer to this concourse server as light from now on so the next thing I want to do is actually define what our pipeline looks like so that image Chris showed you before of um how how information flows from github there's sort of our githlab into and then build it to now deploy it we want to define what that looks like in a declarative form so uh concourse uses a yaml format and we want to create one of these pipeline yaml files and so first thing we want to do is declare our jobs so remember the jobs are things that respond to changes in resources and can push changes to resources so the change the resource we're going to respond to is the changes to our source control and it's going to build our docker image and then push to the docker registry and Sean does not know the soft memory he's got a cheat sheet which is okay so trigger this the this trigger flag what that's telling you is when a new version um appears in the source resource it will automatically kick off the build so this is optional you you don't have to do it you can you can have jobs where you have to you can go into the web interface and manually kick the jobs on now I've been I think we've been kind of being a little bit hard on Jenkins um they've they've been moving this direction as well so they've been getting to the point where you can totally have their pipelines configured as well in code so this is this is the this is this is a lot of what he was talking about in the keynote this morning it's getting towards declarative systems we want to say what we want the infrastructure to look like we don't want to spend time like actually setting it up ourselves okay so in this pipeline file that I somewhat cheated on to interest of time so here's the jobs right first thing we want to do is just one job which is build our docker image from our source and then push it to our repository and so that's what this does and now we have resources our resources one is our source in in git lab and how to access that which branch to be on um and then we don't want our pipe so we don't want to check in the username and password of our git lab server into our source code repository right we want those we want to have those all somewhere else in our in our bulk and then the next one the the second resource is the actual docker repository going to push to you so now that I've created this pipeline file we can go back and use our command line interface actually the one thing I need to do I do need the credential file so I'm just going to copy over the the one that we created before that has username and password on it and now so now I'll run our fly and we'll target our light concourse database and we'll set pipeline and call it mean pipeline and so this next this next step is he is going to so he's injecting this last last part that says load bars from this is this is the command that allows us to inject secrets into our concourse pipeline there you go for the second time um well have you have you checked the code into the repository yet um I don't believe I have yet okay let's do that I've nothing to build yet so um there's add on uh source okay get add all right okay yeah sir um but oops I'm just going to check do our first um check in of the configuration of our docker file that will then build the so the fog was according to plan it was according to plan uh we should see this start build have you pressed play yet um for the second time why don't we just pick the build up this way all right so we're building so now it's building so remember when he went docker build originally and it pulled the engine x image and then it built the image with our index dot html um it's now doing that for you can see it's it's pulling from docker hub uh it's downloaded it and it's going to build that image for us and of course we're over conference wi-fi so this is going to be incredibly vast yeah so this might take a few minutes because the wi-fi is is slower but uh the one interest the one neat thing about this is you do get like pretty much out of the box per developer um builds because if you get a check in because everything runs in containers they don't interact with each other so you can create n number of containers for each of your builds so every developer they each get their own builds for each of those check-ins um so i don't think we're going to get um just uh time constraints yeah i think we've got enough questions that i don't think we'll get to uh pushing to production and powerpoint just crash uh but if i go back to here the next um yeah i think we'll uh there you go so the next step from here would be okay now we push the image to the docker repository now we want to deploy it to uh production so we would add a new a new task for the for the deployment step and in this case for the demo what we want to do is we have a docker um a docker cluster running on the computer and so now we would just run this update man and say update to version 2.0 and then it would pull the image and then suddenly you've got that you've got the new version and uh you can do things with like all these schedulers and with the different schedules out there as you can set up different deployments options so like a rolling deploy so it doesn't tear down everything and have all services not working it'll actually um deploy kind of the new version uh bit by bit um and actually you have some waiting periods make sure that you can check to make sure that things are operating properly before it tears everything down so talking to the mic so it gets recorded so you get um you get kind of like no downtime zero downtime deployments um now clearly you wouldn't want every check in to go straight to deployment so what kind of things may you want to add to your pipeline in between the tests yeah right right and so i think jezz humble did a talk on this i don't think talked about the last year he talked about building these pipelines where you want to have um various stages of um testing go throughout and the purpose of the test is to prove to yourself that you can't deploy it um and if anything goes wrong along the way then uh that pipeline would would fail and that would be your your safety check yeah so check check marks there i mean one thing they do is yeah secure so you can actually build your security checks into and have that as a step by step in this pipeline here yes so that was kind of i think we'll leave a couple minutes for questions people have been sitting for a long time we didn't get to going all the way to production but frankly that's kind of a bit of a trivial step you just create a new job you say this is the one command i want you to run to push it into production and uh you're pretty much done at that at that point the one interesting thing is i'm going to pull up concourse there's some really good tutorials i can't remember the people that made them but i found them on on github they have like a step by step guide to use in concourse and it builds you can you can follow along and and build your own pipeline and build your own tasks uh deploy them on your machine like you've seen it's very easy to just run all of this stuff internally within your own machine you don't have to like actually get servers requisitioned and everything so you can play around and get to know these technologies on your own um and there's they do have a slack channel um with people who are like very helpful and this is an open source thing but Pivotal maintains it um and so there's a lot of great people who help out with it so here is actually the real active uh production pipeline for concourse itself so concourse is actually used to run its own production pipeline so you can see kind of the uh complexity with which you can build some of these build some of these things um and uh yeah so we deployed a lot of a lot of stuff this morning um and what's that yeah so that's a good so actually a concourse has a good the question was concourse versus Jenkins um and Jenkins is adding a whole lot in terms of pipelines and containers so I think the whole this is more of demonstration of where like the whole CI community is headed to you're going to see more convergence from Jenkins and all these other ones to support declarative configuration managed pipeline support tasks in containers um that's where we're headed concourse I think is leading the way in that regard concourse does have a good page I put up here that actually explains concourse versus Jenkins concourse versus team city or what why why did they bother creating concourse in the first place and one of the things like you mentioned why concourse exists is the Jenkins you've got all these plugins um but with each plugin you install that's also a bunch of configure you have to install the plugin you have to configure the plugin and if you want to add another agent to your cluster now you have to go and find your list of agents or your plugins and install them on that and on that as well so concourse is um philosophy behind it is don't do any of that stuff and the trade-offs you get are it's kind of infinitely scalable it's it is less mature of a product right and one of the things I talked about was like the different resources it's like that would be an example of like plugins right these different resources of it supports get out of the box but it doesn't support tfs no and no one out there has written one for tfs um so you know the first person that does it I would really hope that they open source that and push it up to uh to github or to docker hub and github right so um I I personally hope that this takes off and then we see more of a community around it and the more people that start building these things on their own um we can start leveraging these things but this has a lot of potential to do the kinds of things you're talking about that that are already built into jankins it's not there yet any other questions so um so the performance of the build or the performance of the execution um so the the execution of your actual application right um okay so um concourse doesn't really have anything to do with that the execution of your application um because it's just it's building it but then once you deploy it so there's a lot of um uh good monitoring tools out there like container based monitoring tools it's kind of like this an exploding industry at the moment um I couldn't tell you any off the top of my head um but the idea is okay you deploy it to production and now you've got you instrument your application somehow and collect those but you can have a thousand uh you can have a thousand containers potentially all running exactly the same uh exactly the same image um in peak load and so you need to monitor all of those things potentially and then if traffic uh you know your the load is low it it auto scales down to you know a handful of um identical containers so the whole space around monitoring um is uh a complicated it's growing really fast um it's an important one for sure um I don't know if I answer you we can talk afterwards if you want to yeah yeah the easy way is um they've got um shaw 256 hashes so like you oh what's I would not look at the image I would look at the the docker file right and and so you're not you're not manually usually manually deploying a docker image what you do is you go and edit the docker file and then that gets built and then built into the image so a lot of it a lot of the um docker hub images you'll see are backed by github repositories and you can go and look at the actual docker file that was used to build that image and they'll be versioned together so you if you have a specific version you have two versions of a of a docker image on on docker hub you can go back and then dip the docker files that were used to build those docker images that would be my recommendation right it's it's just a code because it's just a text file that tells you how to how to build that docker right so it'd be like dipping to docker images would like kind of be comparable to dipping to like executables you wouldn't really do that you'd look at the source code because if I did not maintain the docker file any source control tool how do you maintain the docker file in case if i haven't maintained uh not maintained so this is this is kind of come back to what is your single source of truth right we're talking about the database example is the single source of truth the actual database itself or the scripts that generate the database to me you want to have your single source of truth is code everything is code the server is ephemeral you can regenerate it from the code so I would I would really really strongly recommend not doing what you're describing so is it uh can we maintain both the docker file and the docker image in nexus nexus isn't really a source code this afterwards why if we kind of maybe need a diagram to to explain some of that so um the same microservice thing it's a separation of concerns right so you're going to have a source code repository and an image repository and then you're going to have like something like concourse that's going to hold for one build and then push to the other beginning of the session you started with the number of villages of the encodes so what if I just have one machine that has multiple images within that machine that will be running finally so that will be I might we have a lot of lots of build yeah so and uh if I want to define the dependency of this that's not going to be possible I just said really only build separately and I'm just running them within one machine itself yeah so um I'm just thinking that if I want to change this type of the way right now my server is running and this way if I want to change which government will be able to help me out in that case then say for is there a java machine that's not running in the same machine that it was built before it's also running and maybe I can and need to ensure that the machine should be uh decent enough to handle all those things yes that's a good so I think if I understand the question correctly it's um I have one machine that's running all my builds and so I and and they can all these things are running in parallel and I need to make sure my machine is powerful enough to run all those things right so this is um one of the concepts uh I think we kind of concept of scale scale rather than scale up so the the sort of historical mainframe approach to doing this would be you would just make like some honking huge machine and I think that actually Dave Thomas might have been advocating for this earlier in the week is buy a like spend $50,000 and give it like a huge amount of resources the other approach is because all these things can run independently and they don't interact and because setting up a new machine this is one of the big advantages of concourse is setting up a new machine is trivial I can and then there's tools in things like AWS that can autoscale it depending on how much is being done rather than having one huge machine let's just add machines as I need them and tear them down as I need them so those are the two approaches yeah it's and it's a trade-off right it's a it's a trade-off and having to deal with like uh many build agents and and scaling it adds complexity system I acknowledge that right so it that's definitely a trade-off and like I said one of our keynotes earlier the week was saying that that hardware is getting better and better and better and getting cheaper and cheaper and he made some quote about having like a terabyte of non-biotel memory for the next two years which I would love to talk to him to get a reference on that maybe some of these problems that that concourse is trying to solve might go away I that's possible um I think one of the use cases is if you've got let's say 150 developers all working on a product and you want to give them person like every commit they get the feedback on did their own individual commit um work or not I mean that's in agile is a lot hold my dev ops natural everything's about fast feedback groups right like get minutes down to seconds seconds down to microseconds get the feedback faster if I can be told within um a second that my personal thing that I did is broke something and it's preventing it going all the way to production that's like makes things much more efficient and much more effective um so if we can provide the infrastructure that easily cheaply that enables to do that it's it's a good cost it's a good investment so if I've got 150 developers during the day I can easily spin up lots of um nodes on amazon that do that and then like a whole bunch of build agents like 150 build agents everybody gets their own and then at the end of the day uh tear all of them down because everybody's gone home for the day and or or autoscale them autoscale them to the um to the amount of work that you need but because they're all identical all the agents are identical they don't require configuration it's very easy to to elastically scale um log into your build machine you like remote into it and do things to it that's a spell because that indicates that the configuration of that machine what's the single source of truth as soon as you start doing manual configurations to a machine itself it becomes a pet it ceases to be handled right so having this usually something that happens when you have one really large machine that you're configuring people end up doing like a lot of individual like this thing's going to exist for like years probably right that one machine you're you're you're the mindset so this is kind of a change in a mindset away from we're going to have this one machine that exists for years and i'll just like incrementally make changes to it to the single source of truth about configuration lives in code and the machine itself is a family that makes sense all right so we're out of time um we will be around so um come up and ask us questions that i want to be respectful people's time i think that there's actually like maybe 20 minute talks and stuff starting so people want to go i will thank you for hearing us ramble for an hour and a thanks