 Thank you everyone for being here. I know last couple of days were scary with the weather and the changes. So a round of applause for the organizers. They did a really good job of keeping us updated and organizing the whole event. So today, we are going to talk about integrating infrastructure as code into a continuous delivery pipeline. What are the various considerations, any best practices that are there, and the patterns that we have used in the past? But before we get started, we'll introduce ourselves. My name is Adarsh Shah. And I've been in the industry for a long time, doing a lot of various things, but focused on DevOps and cloud computing lately. And I'm Matt. As we said, working in DevOps, doing the DevOps. Love DevOps days, first time doing it in Baltimore. So glad you all could have me here. And we both work for Contino. Contino enables enterprises to accelerate innovation through adoption of DevOps and cloud computing. If you have any questions about the presentation or Contino, Matt and I will be around after the talk today, tomorrow. So feel free to talk to us. So the first thing we're talking about is infrastructure as code. And it pretty much describes itself in the name. There isn't much more to it there. But it begs the question, why do we want to do infrastructure as code? And the general idea here is that we want to apply the same practices we apply to deliver our software, not the changes we make to our infrastructure. And if you're not using infrastructure as code, there are certain challenges. First one being configuration issue. You might have a configuration drift. Because if you're managing your infrastructure manually, you don't have automation. It becomes very difficult to keep your configuration in sync. Repeatability, human error, all of those issues. Time to complete, because if you're doing things manually, it takes longer to complete. So all of those issues come in when those are the challenges when you're not using infrastructure as code. The other thing is we mentioned was continuous delivery. And we could think of that almost as the umbrella term for all of the practices that we've developed to test our software and get it to our users. The book was written by Jezz Hubbell and Dave Farley a number of years ago. This was the first mass introduction outside of people already doing it. And the general idea here is that we want to get those changes to our users quickly, safely, and especially at a steady pace. But that's really high level to impact this a little bit. I think this is really the main underlying principle here is that we want to work in single pieces, not batches. All the reasons up there are pretty self-explanatory. But you might see this and think it's funny. Like, this is a manufacturing example. They're building widgets or something. But we can think of our code and our commits in the same way. Each commit is a piece of product that is going out to our users. And we want to make sure that that piece of product is ready to go and there's no defects in it as quickly as possible. We do that by getting it out to our users as quickly as possible, not letting them pile up in the middle of our system, as you see in those boxes there on the slide on the left. So now that we talked about infrastructure as code, talked about continuous delivery, we want to look at various considerations and best practices when you want to integrate infrastructure as code into a continuous delivery pipeline. And we'll also be talking about some of the best practices with infrastructure as code and continuous delivery on its own as well. So let's get started. OK. So the first thing to mention is source control. This slide might look familiar. We basically pretty much clipped it from 10 deploys per day. If you're not familiar with that presentation, definitely would recommend to go watch it. But all squad haven't talked about basically everyone knows where to look. If we put all of these things in code and it's shared and it's available to everybody, it's no longer a secret what's running or what somebody's thinking because that code is available. Anybody can check it out. They can run it and make sure it works. And so as I just mentioned, we want all of these things in source control, the things we're talking about today infrastructure as code. So there is those actual infrastructure definitions themselves, but any configuration management, any test code, CI, CD, job definitions, we really want all of these things in code so they can be source controlled and that they're accessible to everybody so they can check them out, submit issues, make suggest changes if they need to. We'd also say that we want to modularize our code. This will create a well-defined interface in the sense that if I'm an infrastructure person and I'm now creating an infrastructure module, somebody can go and consume that. They don't need to ask me to go ahead and do it. I understand what this infrastructure should be because somebody's defined like reasonable defaults, maybe some non-negotiable things in it. And all of these things combined, we just want to make sure that we point out like that enables collaboration. There's this thing now that both sides, the person consuming it and the person publishing can work around and collaborate on. So the last thing to mention is that your infrastructure process right now, there may be somebody doing it manually. You might have a document there. The chances at any given time that document's 100% up to date, it's usually low. When you're writing things as code and it has to be executed, especially a lot of these tools that are in declarative format, that code serves very specifically as documentation. You know exactly what the infrastructure is because it's declared in this code. One final thing, so we're also talking about the continuous delivery part. If you're putting your IC code into source control now it's triggering off some type of automated CD job. You don't want to break the workflow, the tool you're using. So this is an example of like a TF plan. So Terraform has this workflow where you do a plan and then apply and it tells you what it's going to do ahead of time. So you still want to make sure you're using that workflow. So at some point during your process, still do the TF plan. You should still have somebody check the plan and make sure it's good to go before you apply. So like for your software, you should be thinking about automated tests for your infrastructure as code as well. So here's a test pyramid that shows if you go from bottom to up, the costs, the maintenance, the brittleness and duration of the test go higher. So it's a typical test pyramid that you see with how explaining it for your software testing. But at the bottom, you'll see it starts with static analysis. If you're using Terraform, you can use something like Terraform validate or TF lint. And there are other static analysis tools with all of these other systems. So you should be doing that and you should be running that as often as possible, even integrating with your IDE. So you know if there are any syntax errors, any linting errors as soon as possible. The key thing is getting fast feedback. And as you run this, because these tests are fast, you can run it as often as you want. So after static analysis, as you go up, there's unit testing, like for your application, there are cases where it makes sense to run unit tests for your infrastructure as code. One example I would give is, let's say you have bash scripts and you have some logic in your bash scripts, you can use BACS as one of the frameworks to test out your logic in bash script. You can even stub and mark your dependencies. So it is a truly unit test. You're not calling your dependencies. So that's unit test for you. If you go up, it's integration tests. Integration tests are where you actually spin up your resources, you configure them and then test it. This is valuable because, let's say you're spinning up instances on AWS and you can spin up an fMRAIL environment and runs your tests using InSpec or GAUS that validates that whatever you're expecting your code to do, it does that. But you have less of those integration tests because you should be covering most of them as part of either unit testing. After that, if you go up top, is running a smoke test. The best way to test if your infrastructure works and it's configured correctly is to run a dummy app. So let's take an example of setting up a Kubernetes cluster. So if you're setting up Kubernetes cluster, the best way to test if that works or not is run a dummy app on it in a container and then run some smoke tests. Use something like Selenium or one of the other frameworks to just do a smoke test to verify that you can actually run an app in that cluster. Yeah, so there's a number of, security is a huge thing for all of us, right? Especially with infrastructure. We need to make sure all the ports aren't open or et cetera. Right now, a lot of the time, the way we do this is through handoffs. Like somebody will set up an infrastructure, security team checks it. And we want to break out of this pattern. We want to be able to do these checks again to take this piece of product or commit or code change and know that it's free of defects as soon as possible. So we want to take these same things that are being done maybe by a person right now and build them into our system so that they can happen automatically on every single commit. And these are some of the examples up there. And outside of the checks, we actually want to do as part of our continuous delivery process. There's also security considerations you need for the CDE process itself. So we would definitely say make sure that you have some type of consideration for your secrets management. If you just have like a single Jenkins master instance with all the credentials in the world in it, that's probably not the best way to go. So you want some type of solution there to make sure that you're doing this in a secure way. And if you're building some type of artifact, it's always good if you can sign and verify it so that when you do pull that artifact into your production system, however you're gonna run it, you know for sure that this is the thing that you checked and that it's free of any defects. So like for your security, you should be also thinking about compliance. I'm sure a lot of you work in an industry that is heavy compliance like finance, healthcare and all of the other industries that have compliance like SOCs, PII, HIPAA, PCI, all of these compliance requirements are there and traditionally compliance teams kind of have paperwork. They do these checks manually and they check those boxes with compliance is code. You can actually automate all of that. And by doing that, you can run those tests or your compliance checks early on in your process so that you can verify if you're compliant or not with your infrastructure, even with your apps. And when you do that, you find those issue early on and fix it so that you're not waiting till the end to find out whether you're compliant or not. So it gives you faster feedback. There are certain frameworks available like you can use inspect for that and also HashiCop Sentinel that came out last year that allows policy as code. And I'm gonna give an example here. So if you can see the code there, it's self-explanatory but if you see what this is showing is if you are updating some critical data, it's only allowed if you are part of a certain team and if you have multi-factor authentication enabled. So at the bottom, if you see the last three lines that says that is multi-factor authentication enabled and are you part of the SysOps group and if you are then only you're allowed to change that critical part of data. So there are compliance requirements. I think PCI has where in certain scenarios you need multi-factor authentication to do an action and something like this helps you to automate it and if you integrate it as part of your continuous delivery pipeline, you can run it sooner in your pipeline and get feedback. So we talked about various considerations. We talked about security, compliance, we talked about testing, we talked about source control but now let's look at some of the patterns for provisioning your infrastructure. The ones that we'll be talking about today are up there immutable VMs where you have immutable images. If you don't know what immutable is it's basically you can get rid of it. It's immutable as the word says it's against running long-lived VMs. So we'll be talking about that. We'll be talking about containerized services. So if you are using containers and using container management platform, how do you actually provision and some of the patterns that you can use? And the last is base image and app pool. We'll get into that when we discuss it. So yeah, the first example is using immutable virtual machines and this will be the most in-depth example. So there's a lot of repetition between the three and it's just to show in some different contexts how these things all work together kind of in the same way. As Adarsh mentioned, if you're not as familiar with concept of immutability pretty much it means that we're not gonna change config on running servers. If we wanna update a server, we're gonna bake a new image and then we're gonna replace the old image with a new one. And one of the patterns like he mentioned this is a good practice for is that it doesn't keep long-lived servers, it helps prevent configuration drift. A lot of the things that he mentioned and one of the first consideration slides. But yeah, let's jump right into it. So we don't wanna get too in-depth here but there's some general CI process that you don't wanna prescribe exactly how to do this. But to the extent, if you have any code that can be unit tested, you would unit test it here. But if you're using declarative format type infrastructure as code like Terraform or CloudFormation or whatever it is, more likely there's really gonna be some type of static analysis. If it being declarative, there's not much unit testing going on with those things. But at that point, once we then have a commit that's being merged into master and we have a potential release, we want to pack it into an image and you could use a tool like Packard to do that. Also, I might as well throw out right now. We're using some very specific tools up here to help demonstrate this. We're not necessarily saying you have to use these tools but it helps with the explanation. But at whatever point we do pack an image, it's in some type of staging, a WS account or something. We wanna test it so we're gonna deploy it to an environment. So then we can do some of those slower tests that you saw in a Darsh's pyramid, integration test, security checks. But to be specific, this is at the VM level. We're checking this VM, so you wanna keep it obviously limited to those things. If you're checking something that depends on a load balancer in your production environment, it's not gonna be too helpful to do at this point. But whenever you have those checks that are part of your system pass and to your extent that you wanna do, make sure that it's free of defects, we now have something that we think is releaseable that we can deliver to our users and we will publish it to that artifact repo if we're using AWS as an example and this is an AMI, you have your AMI repository and AWS account and now it's available to be consumed by your infrastructure as code. So the same pipeline would go through a similar process in running your Terraform code that I had previously checked through your CI process. So again, we wanna deploy this to an environment so we can run some checks on it. In this case, it's the opposite of the one I mentioned before. The Packer example was related to the VM. In this case, it is related to whatever this module is and I totally forgot to explain what the module is. So it's set on the prior slide, maybe you read it, if not, I'll catch you up on that right now. So we're talking about like a complex app here, like something that's multi-tier, it maybe has something to catch cluster like Reddit. So you need to build images as part of this infrastructure module to deploy that cluster to your AWS account in this example. But we also need to deploy our application VMs too. So there's two packing of images here. The first one in the infrastructure module is about building that cache cluster. Maybe you have a specific requirement for how you build your Redis cluster and that's what it's doing in the bottom there. But same idea, now once that's done, we can say that this commit is published, we're ready to deploy it, deliver it to our users. There is no artifact in Terraform so you're probably pushing a tag here or some other way of tracking this change and saying that it's a release. And then at that point, I could consume that image that was built in the prior process. So then there you go, you have your Redis cluster. And then yeah, it deploys AWS. So top part, we're gonna talk about the application a little bit. And we have this one shaded out purple because this talk isn't really about your application CI process, it's not really about your application testing at all. It's about how do these two things interface. So whatever process there is for building your application and checking that it's ready, you're gonna do that. But at this point now, we are talking about immutable VMs. So this change now has to be baked into a new VM image. We could use Packer again to do this and we wanna follow a similar pattern to check to make sure it's defect-free that we did for the Redis cluster. So same thing here now. We have that box that just says app test that's extremely broad, but basically what we're saying is if you have any higher order tests that are end-to-end smoke tests, it would probably happen the same time you're doing these new infrastructure type checks that we're describing now. Same as before, once that image is done, we're gonna publish it and then we need to be able to trigger off our Terraform steps again so it can go through that same process, pull that new image and deploy it to AWS. Cool, so now let's talk about containerized services. I'm sure a lot of you use containers and if not looking to move to using containers. So what we'll be talking about here is the infrastructure module to actually set up a container management system. And then it's also fully decoupled, like for the VM, immutable VM option. You had a decoupled pipeline for your apps and for your infrastructure. Similarly for containerized services, we'll show the same. And last here, its apps are deployed using the container management system tool set. So whatever tool set is available with your container management system, you'll be using that. So yeah, so similar to what Matt talked about in your immutable VM option, you have a CI process, you run your cheap tests, the validation, your syntax validation and your unit test if there are. After that comes the testing and validation phase. And you see here, it's a little different diagram here. We're not using Packer here because I'm assuming that you're using one of the pre-baked images that are already available. But you use that and then you bring up an ephemeral environment. You bring up a temporary environment and then run all your integration testing, your compliance and security checks. This part is really critical from what we see. Traditionally organizations have separate compliance and security teams that get involved later on in the cycle and then they find out issues right before going to production. But by doing this early on, you actually find those issues sooner rather than later. After that, you set a form and you can spin up your resources on AWS and set up your Kubernetes cluster. Once that's done, your infrastructure is ready. Your container management system is set up. And then looking at the app side at the top, again, it's grayed out because our focus is not the app testing and app CI process, but you have some kind of CI process there. After that, you build up your Docker image, run some tests around that, and then think about security here as well. So you might wanna scan your images using Clare or there are other scanning tools available. They check for any vulnerabilities that are there in your images and a lot of other checks to validate that you don't have a security issue with your image. After that, you might wanna use something like Notary, which actually certifies your image and validates it when you are actually using it to make sure that it is the image that you published. And then after that, you can push that image and then that image will be consumed and deployed to your container management system. So as you can see, in your continuous delivery pipeline, you're thinking about all the various aspects that we talked about earlier and at various stages and trying to move those practices to the left as much as possible so you get early feedback. So the last example is not immutable, we're the first two are. Docker containers force you to work in an immutable fashion. This one we may not recommend, but this is a perfectly legitimate way to do it. You probably should be more sophisticated because of that last line there in that if you're able to update your applications independent of your images and your virtual machines, more likely than not, you're gonna end up with long live VMs. So you need some type of process to make sure you're killing those things off to make sure that your configuration is in sync with what it actually should be. But another thing to point out here too is our infrastructure module. We're packing an image like in the first example, but this is the base image for application where in the first example, we were packing some supporting infrastructure and then we built our app image as part of our application process. So that's a big difference here and we'll just go through this really quickly and point out the differences. So again, as I mentioned, we're still packing that image in the bottom part, but now this is our application base image and another one of the consequences of this, we're deploying this environment to test. We have to use Adarsh's dummy app example here now too, but this is literally just to test this server pretty much because it's not tightly coupled to our application, which may or may not be ideal. But the benefit of this is that you can update your apps really, really fast. And if we jump up to the top part, you see we have a little Gradle logo there. I wasn't sure what would be a better example, but for years, and I think they still do Netflix packages like all their stuff into Debian packages, they used to use Gradle, they built their own tool eventually. They do work in an immutable fashion. They have a tool called Spinnaker, which is an alternative way to do immutable infrastructure. But in this case, since we're not doing that, we just pack that Debian image or RPM or whatever it ends up being, doing the same checks that again, we can't necessarily do directly with the image that we built before. We lose a little bit of the certainty there. But once we've done that check to our level of success, whatever we say is our definition of done, it can get pushed off to some type of repository. It can be something as lightweight as S3. It could be a full blown Debian repository type thing. And once that's actually done, we need some way to tell our servers that there's a new app available. So pull this thing down, bootstrap yourself and get it running. So I've seen console used to do this in a really nice way. If you're already running console, you can have like a KV there that keeps track of the latest version of the application. The application servers monitor that and then they update themselves based on that change. So we talked about the various patterns. We talked about various considerations when you actually integrate your infrastructure as code part of your, into your continuous delivery pipeline. But all of that is not just for technology benefits. There are definitely technology benefits that you get out of doing that. But it's also about the people part and the process part. So as you integrate your infrastructure as code as part of your continuous delivery pipeline, you are actually affecting the process that goes on in your organization for putting the code into production. It also affects people, right? Your teams, how they are structured. You should start thinking about how to better collaborate. So as you can see the points over there, it actually enables your team to interact better because now instead of having that security team that is a silo and does things in a siloed way, this actually brings them all together and we'll be showing some diagrams after this to show how that changes once you start integrating your infrastructure as code as part of your continuous delivery pipeline. So yeah, your infrared, your security, compliance, QA, all of those teams work together and then it improves the process like I mentioned earlier and that helps in getting faster feedback. Yeah, so the traditional model and not using infrastructure as code and maybe you are using infrastructure as code and you're not doing it in a way that is continuous delivery. So you could even be using Terraform and publishing infrastructure changes and handing that off then to your compliance team, your security team, somebody who's responsible for confirming that infrastructure works if it's some type of QA team that's QAing your infrastructure. But the thing to point out here though is that it's an inspection. That person is saying this thing is done. They're handing it off to another team who's then inspecting it and if there's a defect in it, it's already too late. That person is pretty much done with the code at this point because they've handed it over the wall, they've handed it off to this other team who will then decide whether or not it can be published and deployed to production. So as you can see, that was the traditional model but now after you integrate your infrastructure as code to your continuous delivery pipeline, you can see that the roles of those teams change. They are actually more enablers and they get involved early on. So the previous diagram here, you can see that they're all involved towards the end right before going to production. Here you can see that they're involved early on. They build the right tools, they automate things and they include it in your continuous delivery pipeline so that the dev team, the infrastructure team who is writing the code can actually validate their changes and they can make sure that there are no security or compliance issues, there is automation so you validate the code that you write. So let me just summarize here what we talked about today. We talked about what is infrastructure as code, we talked about what is continuous delivery and how does a typical continuous delivery pipeline should look, what are the various considerations and best practices that you should be thinking about when you're doing that. And you might be in a process where you are already using infrastructure as code, you already have continuous delivery pipeline but maybe it's not integrated. So you should be thinking about all of these things that are mentioned here, the bullet points when you're doing that. If you have some parts of these missing, if you already have a pipeline and integrated, you should be thinking about those missing pieces as well. And last but not the least, it's not just about technology, it's also about people and process part. So that's our presentation. Thank you everyone for listening and if you have any questions, we'll be around today and tomorrow. So we'll be happy to chat further about any of the things that we mentioned today. Thank you. Thank you.