 grandition of my bio. Thank you so much. Yeah, so as you mentioned, I do come from a theater background, so this performance space here seems very accurate. I feel like I should be singing Hamilton really with all the rope up here. Maybe if we have time and instead of questions I'll just do some Hamilton songs or something. Great, so hi everyone. My name is Chloe Condon. I'm a developer evangelist at Codefresh. Today I'll be talking about modern CI CD with containers. Just a quick show of hands. Who in the space is working with Docker containers? Awesome. So many hands, yay. Wonderful. So containers have, as you all know, have changed the entire scope of automation. So one of the kind of implications that you should stop is that you should stop using the staging server. And I'll go into the air quotes in just a minute. I know this is a really bold statement, but with Codefresh I absolutely mean this. And the intention that I want to kind of give you in this talk is a new perception of how to view the staging server. Don't worry, I'm not telling you to get rid of it completely. That would be crazy. But for some time we've been implementing CI CD using the staging server and the staging server has served as a very meaningful part of our pipelines. But what I want to show you today is, given the latest technologies that we have, we can actually rethink our pipeline and we can rethink the way that we're implementing continuous integration and continuous delivery. And we can, in fact, make it a lot faster and a lot more streamlined. So real quick, before we dive in, it's kind of, can we see? Okay, cool. So a little bit about me. As you mentioned before, my name is Chloe. I'm a developer evangelistic Codefresh. I'm a blogger of all things container, Docker, diversity related. If you're looking at me and thinking she looks so familiar, there's a blog article called What It's Like to be a Woman at a Tech Conference where I'm standing in front of a lot of groups of men going like this. That is me. And if you haven't read that article, I would highly recommend checking it out either as an ally or a woman in tech to give you kind of an insight of what it's like to be at these conferences for people like myself. So I will go ahead and get started here. I'm just so excited. There's so many people in this room. Like, if I was doing a musical right now and I got this kind of attendance, it would be amazing. Cool. So a little bit about Codefresh. Before I get started, we are a Docker native CI CD. We're a platform that basically automates the process of building, testing, deploying specifically for Docker users. Very much built with Docker users in mind. We've really seen the way that containers have disrupted the space and we wanted to build a product around that that would really help people with very helpful UI, very fast builds. So here's a little mini agenda of what I'm going to be covering today. First, I'm going to talk about common CI CD implementations and the challenges that we run into with those implementations. And then finally, I'll kind of chat about containers and their impact on traditional CI CD. So this should probably be a review for most people here. In traditional Git flow, you create a branch and you work on changes. And when you submit those changes, the only thing that the person doing the code review knows is maybe a past unit test. But this doesn't tell them that the changes are functional. You have to wait until they get into staging to run a deeper level of testing, integration, performance, security, etc. So why can't we do all of that testing until we get to staging? Because we don't actually have our application running until we get to staging, right? So this model has serious drawbacks. The point being that when it hits staging in the traditional way of continuous integration, it's hitting a much broader set of tests. And that's where we start running not only unit tests, we run UI tests, linear performance, in some cases, manual tests to our code. And if we look at this on a sort of slightly different timeline here, this is basically the same graph but visually a little different. You'll see we're working on our feature branch, we're making a pull request, but it's staging. That's where most of the heavy lifting is taking place here. And on this timeline, you'll also see sort of on that bottom part, it's much more costly for us to fix those issues the later on that we're fixing them. And of course, these numbers on the bottom are approximated. So if we find some issues that we're working on, let's say, on our feature branch, it's much easier to incorporate these changes and to make these fixes earlier in the process. The later we find these issues, the more expensive, the more costly it is. And of course, as a developer, it's very frustrating to get feedback and the usual mentality is, oh, you know, ship it now, fix it later. It would be great to be able to implement that feedback immediately. So another aspect of the traditional way that we develop and we implement with continuous integration is that sort of handshake that we have between development and deployment. And a typical handshake will be something like a label or a stamp in the codebase that can be test reports, test results. Sometimes it can be some proposed changes that we've made to the database. And also that I think under that same umbrella sort of being the list of known issues, you know, communicating that as part of the release notes. So what are the challenges? We've touched on a few, but I'm going to review a couple that we usually see. So firstly, we're all familiar with someone broke staging. And this is really frustrating, right? It holds everyone back. It holds back the people who broke the staging server, and they have to go back and fix it. But it's also very frustrating for people. We it creates a lot of bottlenecks, right? So if we're waiting for staging, and it's totally been screwed up, that puts everybody behind. I'm frustrating for the whole team. And it's likely to happen when we implement implement a process in our code, where it's being tested for UI, it's being tested for integration for the very, very first time. And in this case, as a developer, as I mentioned before, the first time that people are seeing my feature up against the main branch is at staging. So this scenario is incredibly frustrating. If I'm given feedback, I'm unable to implement it immediately. Since the attitude is always, let's just go, let's move it along, we'll fix the next iteration. And I don't know when these changes will be added. If I'll have to reprioritize them, I'd, I'd much rather be able to implement that feedback now, versus later, an example that I always like to give is as an actress. If a director gave me feedback, I would want to do that in the very next version of the performance that I did, which is kind of a staging server to stage get Oh, thank you. I, I will say a quick side note, very big difference between theater and tech has been people do not laugh at my jokes sometimes. So I encourage you to not and laugh in this presentation. Thank you. And lastly, we have those frictions, which which are, this is a really great segue into containers, I think. There's frictions that we have between development environments, staging environments, production environments. And these issues really stem from not necessarily, you know, differences in the code base itself, but things like environmental variables. And that really, that that's really something that when we get to that staging server, if we're seeing it for the first time, it shouldn't be the first time. We have some flows in the process that are very time consuming, to find out why these issues are happening. So eventually, we find out it's not about the code. It's, it's some variable that we forgot to include that we didn't have in each environment. But what if there was a way that there was like some sort of container that we could, hmm, seems like an interesting concept. So to summarize, before I talk a little bit about containers, this is one of my favorite little cartoons here. The main issue is that code changes and pull requests are being tested for much more extensively, and for the very first time in staging and not earlier in the life cycle. So there's really no room for feedback. When the product owners and customers communicate, there's that sort of common joke that I have up here of, you know, what the customer had in mind, what the developer did, how the salesperson described it. Whenever I look at this cartoon, I'm very confused by, by how the programmer wrote it, like what I would, I would expect more from a programmer. But the typical, you know, it's let's push whatever we have, let's do it in the next iteration may not be the most healthy way to, to think about approaching, you know, updating these changes. So and then finally, there's the friction that we have between environments. Containers, so I was really excited to see so many people raise their hand when they said they're working with containers, because I usually don't get that many hands except at DockerCon, which is a very different scenario, obviously. How many of you are using Kubernetes by chance? All right, we got some adopters in here. So containers, as I mentioned earlier, have fundamentally changed the way that we do a lot of things as developers. And it's really reshaped the way that we build software. And it makes the operation part much easier for us. And it really helps us rethink the way we did things before. So I'll start with sort of the very basic impact of CI and CD in our pipelines, then I'll get a little bit more complex. So the very first and basic thing is that we can run every step of our CI inside a container, which is awesome. And as you know, when we work with containers, each time we run a new container, it's a fresh new instance of that image. So we almost are completely eliminating any chance of new builds being impacted by previous builds. So you're isolating builds, you're giving a fresh container each time that we run build and compile the code. And the chances of running it one time having it work and then running it the next time and having it suddenly explode, very unlikely. So the second aspect is that unit test itself, or is the unit itself that we move from one stage to the other. So we can actually hand it off to operations, deploy it into production. It's a much more reliable and self contained unit. So we know our Docker image past our unit tests, our integration tests, our security tests. The likelihood of it all of a sudden not working the same in production, much less frequent. Or just moving our code from one stage to the other. And that's why Docker images are just such a much more complete way of describing our application and describing our code. It's not only the code, but it's everything else we need to run it. I really like to think of it like Blue Apron. Does everyone know what Blue Apron is? If you listen to podcasts, I'm sure you know what Blue Apron is. So kind of this idea of not only having all the ingredients that you need, but also the directions for it in a box. I feel like it's sort of the way that when I first started learning about containers, I could really visualize that. And I really feel the same way about doing CI CD with containers. It's going to be with the unlikely exception of someone deciding to get Blue Apron and just going freestyle on it, which I'm not really sure why you do that. It's going to be basically the same meal every single time. And then any kind of variables that you would need. Maybe they like forget to include something. I don't know, this metaphor is going off the rails a little bit, but it's everything that you need plus the directions of how to do it. That's how I like to think of it. So zooming out a little bit from the Docker image itself. One of the strong drivers of images and containers is the adoption of microservices. So containers have been built from day one to support microservices and it has all the ways that we can define the linkages between them, whether we're using Docker compose, Kubernetes, mesos. They all have slightly different ways of defining how the application is working, but kind of the same idea here, right? So containers actually allow us to define an application much easier with more than one container microservice, which ultimately kind of allows us to clone our staging environment easily. And we can create a staging like environment much more simply. So this is where the air quotes come in. Stop using the staging server. So what if there was a way that we could have sort of this pre-staging environment? It would be a very interesting concept, which containers would would make very easy to do. So the reason I'm saying staging like environment and not staging environment is because we have a rule that our staging environment has to be as identical as possible to what we have in production. And that's because there's certain things that we still want to use our staging environment for. So we may want to do kind of an, let's say I want to do a test on my retail site for Black Friday and I want to make sure that I can handle a hundred thousand extra users on Black Friday. Of course I'm going to want to build a staging server that is pretty much as scalable as my production environment. And I can test that on my staging environment. But for other things, so for other testing types like UI, integration, we don't really need the scalability that we have in production. We just need to have all of the pieces of the application so we can work and test our features. So microservice, microservices really allow us to clone much earlier in the life cycle with a staging like environment. So by using kind of these one-off environments like this we have better test isolation. We can test changes one at a time and then reiterating on what I mentioned before, all those frustrations of not being able to implement that feedback that I'm given much earlier, I can implement it then. So I can spin up a pre-staging environment, give it to my PN, give it to a customer. It doesn't have to be someone super technical. They can take a look at my feature and go, oh, that is what I wanted. That isn't what I wanted. Be able to change that ahead of time. So by the time I'm getting to staging, it's clean, it's good to go. They can go ahead and I feel comfortable with that being out in the wild, so to speak. So when it goes to staging, my feature works as I intended it to. There's nothing really holding me back from implementing them. So I can share the environment and I can start running a much broader set of tests earlier in the life cycle. So you may be asking, well, we can't really have an environment exactly like a staging environment. How do we test for performance earlier? So the answer is we can run performance tests on those one off environments and then track the performance over time. At least that's what we've been doing over at CodeFresh. So if we see over time that performance is slowing down, not going up, that can be a way to trigger a reason to go back and revisit what you did and tackle that much earlier in the life cycle as well. So if we go back and look at our staging environment and we see this set of tests, we see that everything can be shifted to the left. And this is the part of the presentation where I'm very tempted to sing Beyonce's to the left. I feel like I should just have it in this presentation. So there's really no reason for us to stick to the way that we've implemented CI and have that staging environment be the first time that we're testing our code for these things. The more we shift to the left, the faster we identify these issues, the less bottlenecks we have in production, the more streamlined our pipeline is and the faster we can push our code changes into production. I really do need to add a Beyonce gift here or something. It's silly that I don't have that. And then last but not least, when we look at our Docker image here, it's a great revised way of looking at what our deliverables are. We're handing off a Docker image that's much more self-contained. And if an issue is found in that Docker image versus a branch, if I rerun the same image with all the other microservices, it's much easier to reproduce and to tackle it. So this is the same get flow that we had before. We have the feature branch, we have master, we have staging. But if we're working on the feature branch, there's really nothing holding us back from provisioning a node and then doing it in automated fashion with Docker compose up. There's a lot of other great open source projects out there. And if you're unfamiliar with Docker compose, it's kind of a way of zooming out from a single microservice and lets you define more than one microservice, the volumes, all the networks between them. It's an abstract way of defining an application and it allows you to get instances of the application much earlier and on demand. So Docker compose, Docker swarm, really great ways of scaling your application. Kubernetes is also really a great way to run containers. They all have slightly different ways of describing your application. And a lot of these technologies allow you to describe how your application is running much earlier in the life cycle. It'll allow you to get a running environment of a feature branch with everything else around it. So I can send a link to my team, I can get feedback, I can incorporate that feedback much earlier. And we can do that in an automatic way. So not only as a developer can I spin it up, share it with the team, we can embed it into our continuous integration script. Can spin up the environment, run our integration tests or unit tests of other services and then shut down the environment. So we can kind of run in parallel the integration tests, the unit tests with any of the other services that we have. And then we'll still have kind of our last check, like our that final kind of like sanity check at staging. But the likelihood of finding issues, as I mentioned before, is much smaller since we've done all these checks ahead of time. So next, since I mentioned some kind of docker container technologies and orchestrations, you can go and implement it all yourself. Maybe some of you have. Maybe some of you are currently working on something kind of similar to this. You can implement it using these technologies. You'll need the scripts to provision the nodes. Of course, the docker daemon and then all the clusters if you're if you're working with Kubernetes to be able to monitor those. But I'm going to show you kind of how when you build a product, this can be implemented. So you can think of it sort of as a layer on top of all of this and show you kind of what I mean by that staging pre-staging environment and kind of how we think of it as at CodeFresh. And hopefully that kind of inspires you to either build your own tools or use our tool if you want to do that. But it'll give you basically what I'll do a rundown of is kind of giving you the ability to simultaneously run complex tests and then run them on every build, not just as part of the staging environment. So I'm quickly going to switch my display here. If you want to try this out for yourself, this is the GitHub repo that I'm going to be using and just real quick to break down what I'm going to do here. So I'm going to build my image. I'm going to create a composition. So that's kind of like Docker compose. Being able to kind of describe how I want to connect. It's just a very simple Node.js application with the Mongo database. So I'm going to define how I want those two to connect, spin up a pre-staging environment, add the tests. I probably won't get to deploying Kubernetes today, but I'm happy to walk through that afterwards with anyone who wants to see that and then sort of automate the whole thing. So let me switch my display here. To open my display preferences. I know this is super exciting you guys with this here. Everyone see that OK? Cool. Alrighty. So the first thing I'm going to do is I, as I mentioned before, this is just a Node.js application with the Mongo database. So add my repository in here. Great. So I have a Docker file in my app. So I'm going to go ahead and use the Docker file. And the path to my Docker file is at the root of my repository. So I'm good to go here. And here I'm just confirming that I have the correct Docker file. As you know, some applications can have more than one Docker file, but this looks good to me. Cool. So I'm going to go ahead and build this. So the first time this will probably take a minute or two with the Wi-Fi. But any subsequent builds are going to be a lot faster. And that's really kind of the power of Docker, right? Is anytime we're building the image, it's that whole copy-on-write system that we only need any additional new layers that we've added. So while this is building, I'll just kind of show around here. So this is where all my builds are. I can click in here if I want. And I can see all the commits from my team here. So right now we're building that Docker image after we've cloned the main repository. All of my images are living here, as well as the metadata for all the images. I'm going to make a composition in a moment. So that's going to be sort of that Docker-compose idea, all of the directions that I want of how to connect my database with my application. And then environment. So this is sort of where those pre-staging environments would live, that I can share with my team, that I can make changes and updates on, and run all of my tests up against. So let's see how this build is doing. Cool. So we really use compositions for two ways here. The first one would be to spin up a pre-staging environment and to be able to share that. The second would be to run unit tests and integration tests. So the whole idea there is to spin up the composition, run the unit tests and integration tests, and then shut it down. And we can do that in two ways. We can use it in our particular application. We can use the UI in our app or you can kind of build your own YAML around that to define exactly how you want those tests to be run against it. And I can show you that in a sec. Cool. So this built. So now I'm going to make a composition and I'm going to call it demo chat. You can see all the quirky names that I've named this before. Great. So you can use a file in your repository. That would be something like Docker Compose. You can use a template. But I actually just to kind of show maybe people who aren't working as closely with Docker Compose type things right now, I'll just sort of break it down here. So first I'm going to add my application, which is demo chat. And as I add all these things, you can see it's making my recipe here. So I've got demo chat. It's defining the image. And I also need the whole reason I'm creating this composition is to run it with my database. So I'm going to add Mongo as well. Yay, Mongo. And then demo chat needs to run on port 5000. So I will put that in here. And you can see all of the things that I've typed in. It's really spelling this out for me. See if I can find a Save button because everything's tinier than usual. Well, do I want to save? Yes, cool. Cool. So now I'm going to go to my images. So this is my image as well as all the metadata for my image here. I've got everything from the SHA, the branch, last commit. I'm hosting this on the code fresh registry, but you can you can host this on any registry. But what I'm going to do here is there's this little launch icon. So this is how we create our pre-staging environments here. So I'm going to select demo chat. I can run as a standalone, but of course I'm not going to do that because that wouldn't make any sense. So I'm running my composition. I really like to, like I mentioned before, I really kind of like to think of this as a rehearsal for my code, much like I wouldn't want to come and talk in front of all you lovely people without practicing this talk once or twice. I also kind of feel that kind of ownership around my features and different code that I'm writing. I really want it to be in the best possible state before I put it out into the universe be that to a PM or be that to a customer. And I can actually see this link right here. So this link can be accessible by anybody inside or outside of the organization. And if I click that, yay, live demos. So this is just a spun up version of my application. And I can share this with anyone. And this was a heck of a lot easier than waiting for the staging server. I know I don't know what your Slack channel is like, but I've known I've been at companies before. I was like, who's on staging? Who's taking staging? Staging broken. So it really eliminates that aspect of it. But this is just like a much more kind of quickly spin up, quickly bring down kind of way of showing that. So the environment just lives here now. I can remove it. I can spin it up again after I've made a change. Maybe I want to change the color of the apostrophe in there or something. And then similarly, so I did it just for the sake of viewing my feature. But I can also run unit tests and integration tests against that versus doing that at staging. So if I, for example, want to do MPM test, I would run it against my composition here, which is demo chat. Demo chat is the composition that I made. And then I can go ahead and build that. And it will just run it against that pre-staging environment. So being able to kind of test for these things ahead of time really saves a lot of time for not only my team, but also it just is going to be a much sturdier, much more reliable version by the time it gets to staging. So let's see. So probably build pretty quickly here. I'm at 10.19. I'm happy to take any questions now. Or if anybody wants to talk afterwards, I'm absolutely happy to talk about things afterwards. But otherwise, I am running to the end of my time here. How do you describe the state of the code post after you've run all these tests in your staging request? How do I? I'm sorry, in your pull request? In my pull request. So usually we have sort of a way, at least within our application to show that it's either passed or failed. So that way we know by the time it's going to be deployed that we have it marked. And that's either with the badge and the repo or, I don't know if I'm answering your question. So you have some conventions to communicate the fact that you've done all of this work before you've ever created that pull request? Yes, yeah. And kind of a way to visually see that would be this, of course, is mine. But I can show you we use code fresh on code fresh, of course. So I can show you kind of what we're doing within here. And I can, because of course it's a little different when it's just my demo version versus an actual team using it. But here you'll see that, you know, oh, this did not work here. This is passed. And anything that we're going to let go through to deployment, of course, is going to have to pass on. Do you have any good solutions for automatically provisioning these staging environments based on? Thank you. So far so good. Do you have any good solutions for automatically provisioning these environments? Like say, I make a new feature branch and I push it for the first time in this flow you showed, you went into the code fresh console and created your composition, et cetera. What I really like is for my engineers to just get a notification that says, oh, it looks like you made a new branch. Here's your environment. Yeah, no, that's really I we definitely have ways kind of on the back end that we allow people to do that in our application. So something that I didn't really dive into is, so you can work with a template. You can work with your Docker Compose file. You can do all those things. But we have built something that's kind of like a code fresh YAML where you can really customize that yourself. And then of course, if you build your own tools, there's definitely ways to kind of like automate that whole process. But yeah, and we've seen a lot of companies kind of use that feature to they'll work with a Slack bot or something that'll say like, hey, here's your here's your new environment, like good to go. But that would probably be my suggestion on how to do it because I just like having the ease of a bot telling me versus me going in there. Hey, Chloe, thank you so much for speaking with us today. So you talked a little bit about infrastructure as code. Can you elaborate a little bit as time allows for how you code fresh or some of the particulars around what works and what doesn't in terms of code changes in the Docker files in additional, I know you use a different pipeline, but some people use Jenkins pipeline. Once you start putting that in with your code and you treat it as code, what are the pros and cons from a development perspective? Sure, kind of like I want to make sure that I understand the question. So more from a Docker perspective versus kind of the code in general. If you're going to turn infrastructure into code, yeah, you let developers kind of take ownership over some of the release pipeline mechanics. Yeah, so I mean personally kind of seeing how people use our product and seeing how people have really talked about how Docker has changed the way that they've done continuous integration and continuous deployment. I think it's really up to the team to define a set of ways in which it'll be a process. I mean, I can't tell you how many times we've talked to people who really didn't have a process in place on how to get everything. They're like, oh, well, you know, Jim does it this way and Tammy does it this way. And we kind of, you know, oh, we do pre-staging environments, but only like this part of our organization does it. Personally, we find that when people are kind of able to, especially with Docker, it's so easy to spin up those pre-staging environments. And it's so easy to, it's much more kind of ephemeral than the way that we've done, you know, CI and CD in the past. So yeah, I think, you know, it really, I think it really speaks to a strong team. I mean, personally, if you kind of take a look here at, I guess I'm in the wrong one here, but kind of the way that we're setting up our builds here, I think one of the really great aspects of being able to have kind of that visualization of it is as a team, I'm really able to know what's going on. And I'm really able to know, like, OK, great, like this is all in one place for me visually. But also if I need to go back and see, yeah. Hi, thanks for speaking today. So you talked a lot about how, like Docker can help create staging environments. How similar, I guess, do you recommend or do people in practice actually make their staging environment and their production? For example, like, does this could the next step to this just be like deployed to production or, you know, the things that I'm thinking of are, like, in production, you don't have a situation where you can just wipe it clean and deploy anything. So like, how how does this? How do you bridge the gap between like this and production? Production, yeah, sure. So we do have ways to kind of automate that process as well. Like, personally, we have a lot of integrations with Kubernetes and here I can kind of step into my own application here. See, so kind of as I mentioned before, there's a lot of ways to customize kind of with that recipe script that I mentioned before. For example, we have a lot of people who here, there's a good kind of visualization within our application. So there's kind of two ways to do it. You can do kind of a code fresh YAML build, and that's going to be very similar to kind of literally writing out like here's all the things that I want to test. Here's all the way that I want to run my application and here's how I want to deploy it. So we have kind of UI here to do that. We have like a deploy script that you can write within here. And let's see, I'll hop over to usually in this demo, I kind of show how we do that within our application. But for example, with Kubernetes, you can define your service as well as having Mongo in here. These are just going to be like running, this is all running within Google Container Engine. But we have this set up within our own application to or within code fresh to have that be very automated. So it's definitely possible to do that. And if you're building your own tool, I would say, I mean, it obviously depends on who you're using to do your deployment with. But it's it's it's a tricky thing to build. But it is possible to have that all be kind of streamlined. So here, when I do have the Kubernetes deploy script set up, I just come in here and it's all set up for me. I just select the cluster, the names, based on what branch I want, and then it's good to go. So I guess a little bit along those similar lines. Obviously, this is a demo app. So it was a simple. But when you get into the more branched out microservices and you have multiple services and things like that, you may be a developer that's you're on your own feature branch for your one microservice. And you want to be building your environment based off of your local feature branch. But you basically want the default versions of everything else in the system. How have you guys chat or have you guys solved that problem by having, you know, the ability to be able to easily compose saying I want the master of this, master of this, master of this, and then my branch of this? Yeah, I mean, that's pretty easy to do when you kind of get into the customizations of things. We see people kind of using ours specifically for a lot of different reasons, but that really comes into kind of the beauty and the magic of those compose files. You can really define any kind of different volumes that you need between them. And then if you want certain, like particular versions, I guess like a good example here, like kind of a visual version of it is I'm given like pretty much every version of every commit that I have based on the different images that I've built. So that's pretty customizable in that way. But yeah, yeah, I can also show you after. Well, thank you. Thank you, Chloe, for the great session. And of course, my introduction services are available as a service. Thank you. I'm just going to have you follow me around. So, well, yeah, thanks.