 Good afternoon everybody. I'll go ahead and get this started right at the top of the hour as planned. Good afternoon. My name is Lance Albertson. I'm the director of the Oregon State University Open Source Lab. And today I'm going to give you a brief introduction to Test Kitchen and Inspec. Before I get started, how many of you have never heard of Test Kitchen or Inspec before you read the stuff today? Okay. How many of you are currently using both? Okay. So great. That gives me an idea. So first off, I'm going to go a little bit of an overview of what we're going to talk about today and then I'll dive into Test Kitchen itself, Inspec afterwards, and then we can open up some questions from there. A little bit about me. I've been at the Open Source Lab since about 2007. I'm mostly a Sysmin by trade. I manage about 10, 7 to 10 undergraduate students at any time doing Sysmin type tasks. I've been active in the open source community over the years quite a bit. So I know pretty well a lot of the projects. What we do at the Open Source Lab, you can think of us as like a co-location hosting company for an open source project, but we're a bit more than that. On top of co-location hosting, we also have our own private clouds, whether it's on Open Sack or Gennetti. We also have our popular FTP mirror that people download a lot of our stuff from. And we also do some manage hosting for some projects that don't want to deal with the Sysmin tasks of managing stuff. The other half of it is that we're an experiential learning environment for a bunch of undergraduate students. They get hands-on experience doing production type activities. They work closely with all of the open source projects, the setup, manage, troubleshoot any services that we set up for them. And we also do our best to try to teach them new and current DevOps practices and technologies. And many of our alumni have been well known through the community. Some of our more notable ones are the co-founders of CoreOS. We're both OSL students. Right now I'm currently the only full-timer and I have about eight undergraduate students that are working for me. So I'm going to be talking about how you can make your life a little bit easier if you're working on developing infrastructure code. So one thing is that developing and testing your infrastructure code, whether it's Ansible, Puppet, Chef, you name it, can be difficult. And if you do something wrong in production, you're in big trouble. Also, how do you easily deal with issues that may happen with a platform you're using? Maybe they upgraded a couple of packages and you don't know what's causing it. How are you going to be able to replicate that and then fix it? There's so many ways you've got to work around that. So what are some ideal ways of working around this? Well, really what you need is a tool that can spin up that test environment that can mimic production in some sense. And then after you spin that up and get that environment configured, set up like you want for production, you need a tool that tests and verifies that state is exactly what you expected to be. So that as you make changes, you can see what's going on and what's happening. And in theory, you would like to use those same tests actually on the production system to make sure that it's correct as well. And getting all of this together really adds another layer of testing and ease to production deployments that you really need to have in this day and age. So the two tools I'm going to be talking about are both products from Chef Incorporated. Well, actually first I should step back. Test Kitchen is actually a community created project that's maintained by Chef. Test Kitchen is used to create self-contained environments. It executes configuration management tools, and then it can run a test suite and verify. Initially, when Test Kitchen was created, it primarily used server spec. And as things went along, server spec kind of had its limitations on doing infrastructure integration testing. And so Chef created something called InSpec. And it's an open source framework for testing and auditing your applications and infrastructure. If you look at the syntax for it, it's very close to what server spec or RSpec is. So if you're in the Ruby world, it wouldn't be that hard to understand. And the nice thing about InSpec is that it works completely independent at any other tools that you want to be able to use. You don't have to install InSpec on a server to test it. You can have it on your workstation or your bastion host and just run InSpec and run the profiles that you want to run on it. And as I mentioned, you can also use this on production system for auto purposes. Say you have some mandate from your employer that you need to have PCI spec, or you need to make sure these things are done. You can continually run it to do that. But that's kind of a synopsis between the two tools. How we use Test Kitchen at the lab is really kind of why I wanted to talk about this, because this tool for us has been really amazing. And it really works well for us because we use Chef quite a bit for configuration management, which is a first class citizen in both of these tools, but it can work on many other things. So what we primarily do is in Chef you have cookbooks, and inside of those cookbooks you have recipes. And those recipes can do things like set up a daemon, install packages, and so forth. And so what we do is we use Test Kitchen to deploy actual virtual machines and use the same code, Chef code, that we use in production to test things. And then as we run through that, we can write tests on that to make sure that the system as it's configured is exactly the way we want it. And this is really great if you start getting into complex systems. If you make one minor change, you may not think it's going to affect something else. It'd be important to know that. The other thing is, once you have something deployed and you want to make a minor change, you like to be able to ensure that it's going to work the way it wants. And so you can make a minor change, add, adjust the tests as needed, and then you can run it and make sure that the full suite works exactly as you intended. Some other things that we've been using it quite a bit for is testing major software upgrades. Excuse me. So we have OpenStack and Chef in our environment, which are both very complicated systems. And OpenStack has a six-month release cycle, and doing in-place upgrades is a real pain in the rear. There's a lot of moving parts, there's a lot of things going on. And I've developed a way in our test kitchen environment that we can test this. So I can have one branch that's using the current code base, I can have a new branch that has the next version, and I can make sure everything works. And then if I really want to, I can test, I could spin everything up in the old environment, switch my branch, run my converge on test kitchen, and I can see what happens in theory on a semi-production environment. So I can catch a lot of these major issues, maybe not all the nuances things that you don't quite catch in a real production environment, but most of them. And we've actually gotten to the point where we've used test kitchen in a multi-node environment. So I can have a controller node, I can have a compute node that acts as part of the cluster and make sure that they can talk to each other, the firewalls are set up properly, we can run commands as we want between them. And the same thing with Ceph, we could do the same thing with there, we could have a node that spins up that tests that it can mount the Ceph FS file system, for example. And the other major thing is if we're dealing with operating system platform version upgrades, so for example, Cento S8 is coming out weeks from now probably. We're currently on seven, once eight comes out we can start using test kitchen and inspect to ensure that all the changes that happen between seven and eight, we can deal with it and we know ahead of time that we have everything sane. So test kitchen is YAML driven, it works on a variety of platforms. It primarily, I'll be showing vagrant here, but there's other things you can use it with, it actually works with Windows as well. I don't have much experience with it, but I know it works there. And it's really, really awesome. So the basic workflow is, is that you configure it in the YAML file. Initially it was the .kitchen YAML, but now they've created just the .kitchen YAML. You can set up VMs or containers or EC2 instances, you can connect it to, they have various drivers that allow you to do that. And then after it spins up that in VM, it installs the configuration management tool, whatever that might be. It copies any files to that node, usually the files that you have that define what you want to do, and then execute and converge that config management tool to the state that you want. And then after that it runs the test suite. Inspect is what I'm talking about today, but it also has support for server spec and baths if you have experience with that. And then after everything happens and everything works, if the test passes then it destroys the VM or the container or the EC2 instances and you're done. This tool can be integrated into your CEI environment. A lot of the upstream community cookbooks have this integrated end using the Docker side of it with Travis CI or Circle CI and it works really, really well. Installing test kitchen and inspect, there's basically two ways. The easy way and the semi-hard way. The easy way is you can download the binary packages that Chef Incorporated actually has. And that's what I would recommend. Chef Workstation or Chef DK, they recently changed their licensing, so the latest versions require you to accept the license. You can read through what that all means. If you don't want to deal with that, you can just install it with the RubyGEN environment and just bypass that completely. I recommend using something like RVM or something similar and then you can just run gem install, all this, and then it will just be there. And you don't have to worry about the ULA because the ULA doesn't apply to gems or anything like that. And also in our case, if you want to make sure you have Vagrant or whatever hypervisor you want to have set up on your system, make sure that's configured already. So here's some common commands you might see. You're in a new repository. You want to initialize it. You can just run kitchen in it. If you want to see the list of suites that you want to have, you can run list. And here you're seeing that there's one called default. That's the main suite that you have. And then you have two platforms on it. And so it kind of branches out from there. It's using the Vagrant driver. By default, it's going to use Chef Solo. Here it's not actually using InSpec. It's using probably server spec in this case. And it's using SSH to connect to it. And you can run a variety of commands by default. You can converge, which basically spins up the VM. It runs a configuration management tool, and then it's done. If you want to log into the machine easily and just poke around and like you're doing some development saying, okay, this is working the way I want it, you can just log into that. There's also the verify command that you can run to run the actual test suite and then destroy it to destroy it. If you want to do all of those steps, of course, excluding log in, you can just do tests instead. So test as converge, verify, destroy automatically. So configuring it is pretty simple. In this case, you want to define a driver. In this case, I'm defining the Vagrant driver. The provisioner is what you need to have defined. Here I'm using Chef Solo. And then what platforms you want to have. Kitchen already maintains several Vagrant boxes that are intended for this. So for example, putting these in here, it'll automatically pull it from that repository. So you can list as many operating systems as you want, or you can configure how you want to pull that from exactly if you want to as well. And the suite is where you actually get to this integration side of things. So in Chef World, we typically do it by recipe. In puppet, it might be per module or manifest or however you want to do it. And each suite can have its own specific configuration as well if you have that. And I'll show some examples of that later. But this is a very, very basic configuration of the YAML of how you're going to do it. And this actually shows on the previous slide what it looks like when you run Kitchen in it and List. Now the first example I'm going to give is just a simple Chef with Kitchen. Here I'm installing, I'm ensuring that I'm installing Chef 14, which doesn't have a ULIS, so I don't have to accept the license every time I run it. And I'm running it on two platforms. And I have a simple recipe that all it's doing is installing the Git package. Also, I have all of these examples in a Git repository on GitHub, and I have a URL at the very end if you want to check that out. So when you actually run it, this is what it's going to look like. And here I'm showing you how you can actually run individual tests. So this first one is running Create, and then it'll eventually run Converge. Now I sped this up a little bit so that this isn't real-time how fast it's going, but you can see here now we're converging. It's installing the Chef binary, and it converged it, installed Git, and then I verified that Git was there and it destroyed it. This one is running Test, so it's going to run all of the suites. And you don't have to run any other command. This is the only command it's running, and it's doing it on the CentOS 7.1. So you can kind of see how differently this is working. It's doing the same thing, installing Chef, it runs Chef, it installs the package, and then Inspect runs. And here it's doing iterating through every... Yeah, this is just doing one suite. Now another nice driver that's included with the kitchen is one integrated with Docker. It's actually called Dockin. I don't remember why, because the original Kitchen Docker didn't work very well. This is very integrated in with Chef. There's probably ways of making this work with other platforms, but I wanted to at least show that. The nice thing about this is that once you have all the images downloaded, it's really fast. The problem is that if you're running recipes or code that doesn't like being inside of a container, this doesn't work very well. So here there's some upstream images that are already pre-created, and it's doing the same thing, but in Docker. So here I'm doing Chef Docker, and it's actually going to do both suites. It's going to do a bun two first, and then it's going to do CentOS. So one thing you might wonder is, do I have to run them all serially? No. In my next slide, I'll actually show you running it concurrently. And you want to be careful with how many concurrent ones that you do, depending on the machine where you're running it. Any questions thus far? Okay, so there's that. There's that example. This one uses a concurrency of two, and I'm enabled to color, so you can actually see it running concurrently at the same time. Now at the lab, we actually integrate all of our test kitchen instances through our OpenStack cluster, so nothing runs on the local host. So there's times I can spin up 18 VMs all at once to test a really big suite of things in our infrastructure and get a really quick feedback on stuff. Now we get to the more interesting part of the talk, and that is making kitchen work with other configuration management tools. Like I mentioned before, test kitchen really... Chef is a first class citizen, but community members have extended kitchen to include other platforms. So here's an example with Puppet, where I've defined the provisioner as Puppet Apply. I describe where the various paths are in this particular suite. I'm pretty sure you can define this up higher, and so that you can have individual manifests for each suite, and that's the only thing you define. And as an example here, you can say, for example, I don't want the Chef RPM or anything to be installed because we're doing Puppet. And so this full example is also in that repository, and this works a little bit differently, but it works. And I'm not sure... When I ran it the first time, I noticed it seems like it worked a little bit differently than I expected, but it seems to do exactly what it's supposed to do. So here it has its own method for installing Puppet. Basically, it installs the repository for it, and then it installs it, and then it runs Puppet itself, and then it runs the verification from InSpec. And then the same thing happens with CentOS as well. It works the same way. So this is a great example of using test kitchen and InSpec to test your Puppet manifests and so forth. So you can just create a kitchen YAML file in your repository for it, and you can start testing all of your infrastructure right there. What if you're using Ansible? Well, there's another one for that. So here's an example that uses Ansible. You define it as an Ansible playbook. You have some other attributes that you want to have. And if you look at the upstream documentation for this particular driver, or this provisioner, I should say, it'll explain all the different options that you need to worry about. For my example, those are the ones that I use to kind of make it work the way I want it. And so I'm still doing the same thing. I'm installing Git. And so here's, oh, of course, this worked the other day. I'll skip it. But it basically goes through, let me see if I can refresh this, because it did it the other. Let's see here. Let me add it one more time. This was doing it before. Nothing like having it break on you. So here we go. It's spinning up Ansible. It's spinning up the VM. It'll install Ansible. I've noticed this one takes a little bit longer. So there might be some adjustments that could be made to how Ansible gets installed with this particular driver, or this provisioner, I should say, or I could have changed it in a different way to make it work. But it works. It runs the manifest, and you can test it and verify that it works the way you want it. Any questions? The test results? They just get sent to standard out. So in theory, if it fails, the command returns like an exit value. And so if you have it in your CI system, that would show a failure. But if you have it in your CI, you would basically just look at the output there. I think there's some different ways with kitchen, at least. You can change the output. But I'll talk more about InSpec later about the test results, because you can do quite a bit with that. So here's the example going with, oh yeah, this is with CentOS. I did these images in GIF, so I can't exactly pause them. So I wish I could. Here we go. We run it, installs it, and it verifies it's good. All right. Now we're getting into an interesting one. You can actually use kitchen to test your terraform. This particular example is just doing a local docker. But I'll show you briefly what it looks like if you do it in a more complicated way. But basically, if you want to make sure that your terraform configuration works exactly the way you want it, you can do that. You can run InSpec tests to do that. It works in a very interesting way with how Test Kitchen works. So here's an example of doing it. It runs really fast because it's docker, and it's just spinning something up, creating it, testing it, making sure that it's there, and then destroying it. When you do more complicated things, it works a lot. It takes a lot longer. Now, what I'm interested in terraform is actually multi-node testing. So Test Kitchen currently does not have a concept of if you need to have a service that has multiple nodes and testing and interacting between each other. I've had multiple discussions with the upstream authors, and they basically all kind of agree it's kind of outside the scope of Test Kitchen. The intent is you do one small test on a single node, and that's all you need to do. But I have a need as a system administrator that I need to find a way of testing my more complicated systems. And so one way that I've been working around it is actually using Kitchen Terraform and tying it with our OpenStack provider. You could easily tie it with AWS or whatever else that you would want to do. But what we've done is we've done it to mimic a OpenStack cluster. We actually run an OpenStack cluster inside of an OpenStack cluster. It's funny as that sounds. And then another one is simulating a Ceph cluster as well. And when we do that, we actually have to spin up a ChefZero node because Terraform expects a Chef server, which is its own thing. And we don't want to set that up every time. So there's this in-memory only Chef server that we can use. And so we spin that up, and then we configure our nodes in Terraform to talk back to that. And I spin up three storage nodes that are acting just like a regular Ceph cluster, but they're VMs. I spin up another node that's set up as a Ceph FF client. And then I can write tests that either test all of the nodes, the same test. I can test individual ones. I can run commands on them. I can make sure everything is working the way I want it, which is great if I go in between one version to another to make sure everything is working the way it's supposed to do. So that's been a really awesome, amazing way. I couldn't get the GIF to compile in enough time because it was so big. But here's I'm speeding it up quite a bit. But this is spinning up all of those. This is doing the Ceph FF, the Ceph version here. So I'm spinning up the node that runs Chef zero and then this is installing the first node. It's going to install Ceph. It's hard to say those back and forth. Ceph and sets up all the OSDs and it goes on to the next one and keeps on going and going and going. This took me a while to figure out. And once I got it going, I was so happy. It made testing our upgrade so much easier. And when I did the upgrade on our production stuff cluster, I didn't have any issues because I could actually simulate issues potentially like, okay, what happens if I do this one upgrade here and I don't do the other one? How does this work? And then one more node after this. And the nice thing is is when it destroys everything, since it's Terraform, it just destroys everything in the background as well. And this assumes you have Terraform installed on the machine. So shortly here it should run the integration test. It'll probably go by so quickly that you couldn't see. But here you see you're seeing it going through each node doing its tests and now it's destroying everything. So that's one advanced example of using Test Kitchen in a multi-node environment. Now let's move on to Inspec. So Inspec is an open source framework for testing and auditing your infrastructure and applications. It works with testing your system and making sure it's in the state you want it. It can display violations and it creates a report. The report can be done in a variety of ways. It can be in JSON. It can be in HTML. It can just be the standard out. It can just show you only the errors that comes out and so forth. It's created by Chef and it works completely independent of Chef. You don't need Chef to run Inspec. Inspec does its own thing. And as I mentioned, Inspec foreign later have the new Yula. It runs remotely and you don't have to install an agent or anything on the remote hosts, which is great. It runs everything remotely. And the other part is that there's some communities around that on creating a collection of tests that can be managed and done with the community. And this integrates really well with Test Kitchen, which is what you were seeing in the previous slides. But you don't need to use Test Kitchen to use Inspec. So some basics. You create a profile, which you don't have to do when you do it in Test Kitchen because it just does it magically. And a profile can be just a complex set of tests. So you want to say, my application X, I expect these types of things to happen all the time. And so you would write individual tests. You would set some of the tests at some kind of a score. Like, if this really fails, this is really bad. If this fails, it's not so bad to make it more of a warning. You can also set dependencies. And you can point to public repositories or private repositories. You can say, well, this actually needs to have this baseline for our company that we want to have and make sure that this node has that all configured and everything. The next part is actually adding your tests. So there's over 80 resources that you can use that are useful to create and construct tests. So several of them are just going through files. So there's actually one specific for Apache Comp. There's one for any files. Or you just want to be able to see specifics about your Docker. Or you want to see how your FS tab is configured or files. Or the really useful one is just running a command and checking the output of that command and making sure it's right. All the actual tests that you do are written in Ruby in an RSpec syntax. So it's pretty readable. Adding resources is pretty easy. You can include it yourself if you want. Or you can extend it. Actually, I added the IP6 tables resource because they had IP tables, but they didn't have IP6 tables for IPv6. And that was easily added and got that incorporated in a release version, which I think is actually included here. And then after that, you target your system. So you run the tests on the infrastructure however you like, whether it's locally or remotely into the cloud or however you want. You can run it apart of Test Kitchen as well if you want to use the same tests. Or Chef actually has a product called Automate, which can integrate all of this. So it can run, basically it runs InSpec with a set of profiles continually and reports back to a web application to give you nice, pretty generated reports. So you can see what's going on. You can click buttons to make it go do and fix things if you want to. I haven't had experience doing that. They've open sourced that product, but that's also under the new ULM. But that's something cool you can look into quite a bit. So here's a very basic test. We have this any file off to the left. It's very simple. And then on the right, we have a very simple test. So here we're describing the path to the any file. And then here we're setting if you read the documentation for the any resource it describes, here's how you do it. So for just a simple port or for the syntax up at the top, you just put the name of the variable and the key on the right side. So here you can say it should be equal exactly to this. There's also a compare, evaluate or match, which means you can do a full regular expression if you want to have that fun, which we use quite a bit. You can also do values of less than, greater than, all that fun kind of stuff if you're expecting various things. If you're getting into a section and part of the any file, you can just, there's a syntax for doing it where you just do the section dot the name of the setting and return that. So here's what it actually looks like when you run it locally on the system. This just runs, I had a file locally straight there and I just ran it. And this is what the output looks like. By default, it shows it in a nice colored output. If it's all green, it means it's good. If there's red, that means something failed. The inspect executable has a variety of commands. The most useful one you're probably going to run into is the exec command. And as I mentioned, there's a variety of ways if you can do it. You can point it to a specific test that you want to run. You can point it to a profile. You can point it to a remote SSH repository or Chef has a supermarket where people can publish them where you can grab it from there as well. If you look at the man page or the documentation, there's a whole bunch of other settings you can do. There's even settings for setting a bastion host and a way you want to log into a bastion host and then from there you can run the command outward. They wrote a really awesome library to manage SSH connections and with Windows it uses WinRM. And it's very stable from what I understand as well. So this works really well on Windows as well. Here's a little bit more drawn out example of a target. So here I can use the target URI method. So I say I'm going to SSH as this username to this host. Since I'm not logging in as root, I need to enable root. This assumes that sudo doesn't require a password. But if you do require a password, there's another parameter you can give to do that. Here I'm giving it to a path to an SSH key that I have and then a path to the test of the profile and it runs it. I'm not going to do a deep dive on profiles, but I wanted to at least show you what it looks like. It has its own metadata structure. So you have a YAML file that describes it and the dependencies that are there and other various information. Controls are actually where the individual tests are. You can have different controls for each type of test and each control can have its own setting that you want. Libraries are anything that you've kind of wanted to extend. So if you wrote your own type of resource, you would put it there and it would get locally sourced. And then there's a files directory. If there's any files that you need to use to have the profile access while you're running it. As I mentioned, there's a community behind a lot of these profiles. There's one dev-sec.io. They provide a lot of baseline tests for your typical environment, whether it's an operating system application or specific components that you want to test. And the cool thing is they also provide remediation examples for Chef Ansible or Docker. Yeah, it was the Docker I was supposed to put. I think I might have had something puppet. Oh, it should be saying puppet there. I made a mistake. And it's really, really, really useful. You can contribute. It's all in a GitHub and you can contribute to it. So here's an example that I've spun up a standard vagrant VM and I'm running the baseline, the Linux baseline test on it. And I'm kind of slowly going through showing you the output of all the various areas that have gone through that's happened. So this has a whole bunch of tests that are community driven. And the default VM that comes from this particular box that I use has all of these issues. At the very end you'll see how many tests to have and where things are set. And I guess I didn't show it. But there's a lot of failures. So the great thing is, that's awesome. I have all these failures. How do I fix it? In this case, on the dev-sec one, I can actually go to their website and I can download specific examples. So they have Ansible, they have Chef, and they have Puppet. And you can run that and then rerun the test and see how it looks. And that's basically what I have this example as here. So in this example, I'm logging in to the system. Let's see here. Oh, this is actually showing the same thing. So this is all the errors. Any questions while this is running? So now I'm actually logging into the system and I'm running a script that basically will run the Chef remediation for the Linux baseline. Say that again. It runs commands using standard Unix utilities through Ruby library, I think called train. Or Windows types of commands that use the standard commands to kind of get all that information. It's inspect based. So it's all written in. So there's Ruby code and inspect that does that. And it doesn't run Ruby on the remote node, but it runs standard Unix commands or Windows commands to get the information it needs. And then based on the output of that, it uses Ruby on the local host to determine what's going on. It's all part of it. Yeah. So, and I don't know if this got locked up or what, but basically here it would be running through all of the various changes and it doesn't look like this example is working the way I want it and it's not showing the end. But the end result is that all the tests pass except two that were marked as skipped as their minor issues. So that's one example. Coming back to here, another example, if you're not wanting to use the upstream one and you've written your own one, this is a great example of, hey, I wrote something and something changed. I want to rerun it and make sure it's fixed. So in summary, Test Kitchen is an amazing tool. I love it for developing our infrastructure code. It has a wide range of drivers. Even though Chef is a first class citizen, it has some extensibility in other providers. Inspec by itself is also an amazing tool. You can make sure that your infrastructure is working exactly the way you want it. And if you want, you can tie it in a shop automate and have even more amazing reports. And when you tie them together, it's an amazing set of tools. So with that, I'll open up for any questions. I think, oh, did I just go over. Are there any questions? All right. Thank you for coming out.