 This is getting started with testing of an infrastructure, so I'm not a developer. I'm a sysadmin that's just trying to do things better. I work in a mixed environment, and this journey started because I wanted to manage my window boxes. So this is just my journey into what I've learned. This is still a work in progress. I'm still trying to get this workflow adopted by my shop. But here are some of the concepts that I'm needing to teach my teammates about. So first, what is it and why is test-driven infrastructure worth the effort? Basically, test-driven infrastructure applies agile development principles and practices to infrastructure, bringing along with it similar benefits of minimizing risk, building confidence in the code and process, and trying to keep efforts on task and outcome-focused. With a verifiable behavior where we are checking our results against our requirements, we have an increase in quality, making it easier to improve upon, giving us confidence, which leads us to safety fastening because we're applying good practices and processes, which allow us to speed things up and enable us to be more adaptable, letting us approve upon our design. Of course, these new efficiencies also reduce IT costs. Current problems with system deployment and maintenance. Sometimes it's just a lack of process. Every engineer, tech, this admin is implementing solutions however they see fit, whatever they think is best. This can be chaotic and definitely difficult to repeat, can't verify what you need is actually being achieved, and is just not scalable. Or there's manual processes where things are being documented, there is some structure, but this is still prone to error and inconsistency, and of course documentation becomes outdated, and does introducing risks and deteriorating trust in your process. Or just poor code. You have started automating, but the automation is not well implemented. You can have a bunch of scripts that are difficult to understand and become unmentainable. Configuration management can help address these issues. Configuration management is translating your infrastructure into code. Configuration management ensures the systems are in a known good and trusted state. This allows visibility into your config, and they are trackable as you use version control for your configuration. And it also provides reliability as your configuration management will ensure the state of your systems and making your environment more predictable. This increases productivity, resulting in fewer outages, less firefighting, and less firefighting. Now that there's more, with configuration management, also now a more efficient way to do change management, giving more confidence to deploy, letting you scale faster. An example of configuration management workflow. This is Chef, but the idea is similar with configuration management systems. So at the top we have a repo where we're checking in our configs. In the blue below that we have our workstation where we're implementing and making our changes to our configurations and related attributes. And below that is our configuration management server, which is responsible for deploying these configurations and enforcing them with policies. And to the right is the node that it's responsible for. Some of the popular options you may refer to as Chef is the puppet. The newer guys already include Salt and Ansible. At the moment I've been starting with Chef, because I work in a mixed environment and they've had a good history working with Windows. They have a large community, a lot of out-of-the-box tools, and they've been continuing to innovate on their workflows. So an example of what a configuration management file may look like here is very simple. Just an action to make sure a certain package is removed. Below that we've added an attribute to be more specific to a version of that package we need to guarantee is removed. So the configuration management has a major shortcoming. It does not prevent that infrastructure code. That is poorly created and maintained. Incomplete code including things like leaving comments with to-dos or sloppy quick fixes can be unclear, leading to a code that can have unintended side effects and could be catastrophic, now reducing confidence in your processing code again. So configuration management on its own is not a complete solution. So we have test-driven and behavior-driven development to help us out. Test-driven development is a framework for managing and involving environments that facilitates the creation of highly reliable and maintainable code. It helps us maintain scope because we have tests that are written first against specific requirements. These tests, and we code against those tests. These tests provide us with faster feedback and help us reduce risk and build stress in our process. Behavior-driven development is an extension of test-driven development. This facilitates the collaboration between stakeholders and developers by describing requirements as system behaviors. So business values are prioritized because the requirements are correlated to business outcomes. Test-describing tests are written to describe the system's expected outcome. And communication is improved as documentation becomes more readable up-to-date and is readable for experts across different domains that can be outside of the team. So an example of such a test is here in the service spec. This allows us to a more readable test that is validates that our server is in its correct state. At the top we have it, like 480 should be listening. And below that we are ensuring that we have the right release distribution is actually deployed. So Mnemonic for remembering test-driven development is the red-green refactor. Just red, we're just starting with a test in a sales state. This test is also the beginning of our documentation. Green, we're writing just enough code to suffice the test. From there we can refactor cleaning up the code, change how we are implementing the solution, but we cannot change the behavior as we need to always pass that test. And we rinse and repeat as necessary. Continuous integration is a practice from extreme programming, a type of agile development to help speed up the process. We're basically going to automate test-driven development. So continuous integration is a practice of short development cycles with automated testing and automated code integration. This allows us to detect issues sooner with more immediate feedback because of the automated testing we are able to fail fast. And then changes become even faster to implement because they are incremental and verified along the way. So two notable types of testing as it relates to test-driven infrastructure is unit testing to verify the individual components that you're working on, independent of changes being made to the greater system by other members. And then you have your integration test that when you test, you're testing your changes as they play with the entire, with the rest of the infrastructure. So a simple example of a continuous integration workflow. You've made some changes, you commit those changes. The configuration, the continuous integration server fetches those changes and begins a build process to test and verify whether or not the changes suffice for the test. From there, the continuous integration server can send out feedback to the team to let us know the results. Popular options of continuous integration software includes Jenkins. It's probably one of the, if not the most popular software out there. That's what I'm using at the moment. I started with Travis CI mainly because they are a hosted, they do free hosting of this for open source projects and integrate with GitHub. Ms. Go, which is a tool by ThoughtWorks that looks pretty interesting to me and is probably next on my to-do list to investigate. But there's code review. Just because we're moving towards automating all the things doesn't mean code review should be overlooked. And as you see here, we have one character who's asking for his family to check his code and asking him to highlight anything that looks stupid. And he proceeds to highlight his face. So code review. Code review is just peer review of the source code. This helps improve software quality, ensuring if other people can read your code, it helps ensure that it's readable, which means it can be maintainable. It also helps with knowledge sharing of the source code. And new team members can observe different approaches and solutions to problems. Tool-assisted options include Garrett, Board of View, Fabricator. My examples are just to give you an idea what's out there. There's tons of available tools. As far as for code review tools, they act as gatekeepers for your commit. But they can assist with comments and discussions, audit and tracking, and help visualize this to actual test-driven infrastructure. So with test-driven infrastructure, we have documentation. This is the first step. Usually we start with documenting what our requirements are, including the components, versions, how to actually install, and then we move to writing tests, which will describe these requirements and ultimately become the documentation. And then we script. We script towards these tests, and we're not trying to do anything extra. We just need to make these tests pass. And another important component is being able to audit and track the code, which is where version control comes in to play. And the process is continuous where we're automating testing and changes, and if they pass, we automatically integrate them. So a workflow that I've been, that I'm the way I'm currently using it right now, there's no exact way to do this. Some people don't even do code review. So there's no 100% way to do it, but this is what I'm experimenting with at the moment. So I have my configurations in my get. I do a fork. And going the wrong direction. Do a fork. Code, code just enough to... The coding includes writing the test. So I write the test and I write the code to pass that test. I commit, and my commit will trigger a get hook that will call out to my configuration, my continuous integration server Jenkins in this case. It will push the job out to the appropriate place with the appropriate tools to run the test and send them up in the test environment, which can be a container, VM, Amazon instance. You can pretty much talk to anything you want for your test. This does not have to be your actual production-like environment just to do your initial test of your change. And then we get notified whether or not it passed. You can go to the IRC email, carrier paging, whatever your team is using at that time. And if it passes, I have Jenkins issue a pull request. And then from there, it is ready for code review, and if it passes code review, can be integrated. So continuous delivery. Continuous delivery gets your code actually to your staging environment. The practice says continuous delivery of changing. The continuous delivery is the practice of delivering every change to a production-like environment. This increases the ability to adapt as we are gaining more feedback faster. This also increases reliability as every incremental change can be observed and lowers risk as a result and is a path towards getting to production faster. So what that would look like is here we have just passed code review and the pull request is being committed. Then we call out to our continuous integration server again, pushes out our test, but this time it makes sure it spins up our test. Now integrate it with the rest of the master branch and everything else that we have that's been deployed and spins it up in a production-like environment. From there, again, we get our feedback. And if it passes this time, we actually push out to your configuration management server that is responsible for the staging environment. Continuous deployment is taking this to the next step. This is actually automating the process all the way through your staging environment to production. This provides us with a quicker return on investment because we are getting to production more immediately and we have faster client feedback. So if you break something, you should know pretty fast. So continuous deployment workflow is just, like I said, going through from our dead environment to the different staging environments, QA, whatever is necessary for your team. And then automates it out to production, which is the key difference for clarity. Continuous delivery can automate you all the way up just before getting to production. Continuous deployment is automating that final step. As my research, I found few people that are actually able to achieve 100% automated workflow like this. But options include plug-ins for Jenkins, like workflows, pipelines, Chef having native tool called Delivery Truck. Go has this idea built in from the very beginning, hence why I kind of want to investigate them. Pipelines are just a series of stages to provide you visibility and feedback in the workflow. So challenges. This all sounds great that I want to implement with my team, but why this won't work? It's a cultural shift. I have to get everybody on board to change their workflow. That's also a time investment. That does cost money. And it requires full adoption. So if people are not writing tests or they're writing too many or writing tests without the proper follow-through, it's not going to work. Or even if they're trying to, but they're not writing good tests. Either they don't know what to test for or don't understand what they're testing for or overlooking what needs to be tested. Or just writing things that will pass anyway because they're testing trivial code or their tests are trivial. So in conclusion, test-driven infrastructure improves reliability because we have a test-first process. And those tests help produce current and readable documentation because of automation. We have an increase in speed and we have an increase in productivity because our tests are helping us to maintain scope. And we are more adaptable. We're doing less firefighting. We're able to get feedback faster. And all this helps reduce the cost. So for more sources and information, I provided several links. I forgot to mention the slides will be available. I actually got a link at the very end for my slide deck. And thank you. I just do want to do a shout-out to Ned Harris of Chef I.O. because I picked his brain a lot for getting started. I had a lot of trouble getting some of these concepts and terminologies defined. So yeah. And you can follow. I'll tweet out the link for my slides in a little bit. Also, just the QR code will take you a link to my page where I'll be putting up more information about this talk. There's already a link to the slide deck there as well. I know this was a little fast. I got a little short on time. So I'm more than willing to take questions later on out in the hallway or whenever. But I think I'm about out of time. Thank you.