 Okay, okay, I think we can start or you want the one minute sure Okay, so welcome everyone. Let me introduce Vasil Kai Gerudo will be giving you talk about as even development for servers infrastructure So for is yours? Hello guys welcome everyone. Thank you for coming to to my talk before We actually start I will Speak about myself and while I am doing it if you have your laptop connected to the Wi-Fi you can clone the repository You see the URL on the screen to be able to follow The talk on your own laptop It's not it's optional. It's not mandatory. I will be showing I will try at least to show you some Some code on my own laptop But you might find it useful To be able to follow yourself So I'm started with the Unix world in 2003 it was a free BSD CD my friend handed me over it saying that it's really something different and difference Different from what I got used to It really was was because before that I was using only Windows Killed my PC a couple of times while trying to install free BSD format at everything I started to work in IT area in 2005 and In 2012 I joined Red Hat as a junior technical support engineer and From late 2015 I'm working again at Red Hat as a senior CIS admin Responsible for central CI Infrastructures I have never worked as a quality assurance engineer and no close to nothing about Testing software. Hopefully it will not be a problem because we will be speaking about testing infrastructures anyway We are going to speak about test driven development in general like what it is and What TDD stands for And then I will show you the server spec tool Which I personally use in my day-to-day work to make sure that my servers are set up correctly I will quickly quickly go through getting started steps and I will speak about different components of server spec and then will Demonstrate you the end result of what So what's TDD This abbreviation stands for test driven development and Wikipedia gives us a very nice definition of test driven development It's basically a process when you Get the requirement first translate them to the test case Create tests based on these test cases and then only then after that after you have your tests ready you Do the implementation the actual development work? so This first then the actual code for example, we can have a requirement that the user password length should be eight Characters or more but not more than five hundred twelve characters and among other things a test case might look like Check that the password variable is set and check that the password variable length is more than more or equal to nine or less than or equal to five hundred twelve and then we implement some codes to ensure this and To work on it for if for infrastructures though the definition will be slightly different it's Again, it will be a process where you get together a requirements, but instead of Doing some software changes you change your infrastructure to be compliant to to the test cases and I'm going to show you how to do it now We might have a requirement for a server to have a user nobody with a spin no login set as a login shell for this user and A test case will look like we check that user nobody exists and we check that this user has spin no login set as a login shell Implementation is easy. It's only one shell command change shell for user nobody Testing software is easy in my opinion, and I'm not saying that it's easy to do it What I'm trying to say is that there are all kind of Tooling around software testing. You can have parallel unit J units whatever unit We have p-test for Python testing. We have our spec for Ruby code testing We have all kind of integrated test suits Very cool stuff For infrastructures Until like recently I will say they're running those grep commands to see if a user actually exists and what login shell he has Or you might want to try to switch to this user and see if you're successful or you're getting an error or you can get What is called useful useless use of cat and grep award and run this ridiculously long when liner to ensure that everything is set up correctly or You might do it in the right way and you might use this Set of commands pipe together and eco pass or eco fail depending on the result if you want to script it Downside of this is that it's quite complex in my opinion to Understand and support like not sure how many people in this in this room knows about know about get and comment Quite some people okay So yeah, anyway, this is not How a proper testing should be done, it's not kind of unified way because Some platforms for example like this get and comment and I would love to see some kind of obstruction layer On top of the shell commands. This is what Ansible does by the way. It's just Rapper around usual shell commands you're using There is this abstraction layer and it's called server spec. It's written in Robbie by Japanese guy his name is go sookie me a shit. I hope got his name, right? It's basically our spec tests for servers and What server spec does it translates the R spec tests that you write to a shell commands And it uses as a sage winner M docker API Interfaces like that to actually check the actual state of of your machines The official site which contains all the documentation server spec the torque and As you can see on the slide on the screen it our Test case with the user nobody looks really simple like it's if you read it out loud It's almost plain English. So we describe the user nobody which should exist and which should have login shell It's been no login. It's pretty awesome To get started you need Robbie, of course you might want to have Robbie version 2.3 because this is what I used to Prepare for this presentation to run the demo It might work on all the Robbie versions as well, but I'm not sure about it After Rob installation, by the way, I would recommend you to use some virtual environments for for installing right like rbm or rvm because it takes care about all the dependencies and It basically installs you very You get a very consistent ruby installation and you can maintain different versions of Robbie for for your own profit So after you have ruby installed you have to install the server spec gem create a working directory for For where you will hold your tests and run this nifty command server spec in it which will ask you some questions and Will generate you some boilerplate The main parts of server spec test suite are the rakefile and The spec helper I will speak about those parts later a little bit what I want to speak now what I want to describe now is the Disadvantages of the default boilerplate Server spec was initially designed to test only one single machine the Guy who wrote it initially he didn't keep in mind that we might have a cluster of machines that are set up In exactly the same way and that some code sharing should should be applied with the default setup you basically create one subdirectory in the spec directory and it's One-side directory per machine and if you have like ten Ten servers that are set up in exactly the same way you will have ten sub directories and ten Tests test files that contain exactly the same code this is not quite the good approach because there is this dry or Don't repeat yourself principle that cannot be followed in this situation Good thing is that since its aspect it's ruby you can modify your rakefile and and kind of create some rules Describe your infrastructure using ruby code I want to show you how it looks like so just give me a second right, so this is pretty long and But it doesn't do Anything special so what it does is it goes through all the sub directories all the Files and directories in the spec directory it sees if what it processes right now is a directory or not and if it encounters a directory it Adds it upends it to the roles list so each sub directory in a spec directory will represent a role name like database web server Capsule satellite whatnot After we have our roles List generated we iterate through each the each of the roles directory and find the files that Have a special naming convention so the name of the files should conform to fully qualified domain name underscore spec dot RB and Based on that we tell server spec Where it should go What machine it should log into? to run our tests and Then based on the host list we generate the set of break tasks Which are basically represent? the testing actions themselves, so it's really easy and if you know any any Programming language you will be able to understand this rake file in no time So this is how I use it and I was working on implementing like this Assumes that you have a file system layout that conforms to your infrastructure but since it's Ruby you can use YAML file for example to Create a list of roles and then the list of roles will contain a list of servers attached to it or you can even talk to satellite for example and Based on the information that you fetch from satellite you can Apply some roles to the server so your Rake file will be a little bit more complex in this case and some advanced programming skills are needed But the possibility is there which is pretty awesome in my opinion I'm going to commit the YAML file support to The same git repository that you saw the start of the presentation the Second part of the like we spoke about the rake file and what it allows you to do the second part of the server spec is spec helper where you specify the Metadata username to log into the machine password or SSH key You modify SSH settings there stuff like that It also The support for shared tests shared testing is also Implemented in the spec helper and I'm going to show you what exactly it does. It's really simple so these two lines they're basically do some again file processing and They fetch all the files in the shared directory and Just run a require on it, which is basically import in Python or include in C Something like that. This is how you can reuse the shared code on on your machines. I Will not speak about Formatters too much The only thing that I will say that there are three built-in formatters in Server spec documentation HTML and JSON Documentation is what you see in the console output that I will show you in a couple of minutes HTML just generates an HTML file with the results I will also also show it to you how it looks like and json generates a json formatted file Which you then can import to some database or you can feed it to some script that will process the json and Generate some statistics for you Let's get to the practical part of this presentation. So how how you can implement How you can employ a server spec to During the ansible role development The steps are pretty easy. You get the requirements for the role. You translate the requirements to the server spec tests You run the test on a clean machine to make sure that they are failing then you implement your ansible tasks and You run the test again to see if You got it right in the ansible Let me Actually show you what I mean. Can you hear me? It's fine Yeah, so because I was told that the microphone is not good here People might have troubles here in me so we will be creating an ansible role that does Apache installation on a machine very easy like it will Install the htpd package It will make sure that htpd is running and it will create a very simple index HTML file so those are our requirements and Those requirements are translated in this really simple set of tests So we create a packages list and three of them for each package we ensure that the package is installed then we ensure that the htpd is running and then we test that index HTML file has certain permissions and owned by user nobody and group nobody and We match its content to some string Right, so I have this Vagrant box Vagrant virtual machine prepared. It already has the Packages installed Because it's the it would take some time otherwise, which we don't have And I'm going to run this test again this machine To see the list of all the Can you actually see it? The very last line on the console Okay, so to See what machines are you going to test when you run a rake? spec command you can run rake dash capital T and You see that we have only one Only one test here which Test this demo machine, so we are going to run it and see what happens So as we can see we have ten Tasks or examples. This is the R spec This is how our spec calls the test tasks So we have ten examples and seven failures As I said that we have all the packages installed three of them and all other tasks were failed and right here we see what exactly is wrong with with our machine service is not running and Something is wrong with index.html And of course there is nothing listening on the port 80 because htpd is not running So we go ahead and develop a role. I already have it The developer will just show you what it does. So we have this sensible role that installs stuff makes sure that index.html has correct permissions owner and group and It makes sure that htpd is running So let's run the ansible Playbook and see if That tests will be green this time. So we have the index.html generated and htpd is running And now we are running our tests and bam. It's green I mentioned that Server spec generates html files with the reports Here is how it looks like. Excuse me. Unfortunately, I cannot Quite make it bigger, but believe me. It just describes the All the tests that you have and their results Shortly, I will show you how it looks like with the with the failed test So this is the initial development of an ansible role and you can see That it doesn't quite look good because the syntax highlighting is a little bit off in Vim And it's because I'm using old kind of old YAML notation for it So I want to refactor it and I want to Look at a little bit more YAML friendly, let's say and what I also want to do is I want to ensure that service not only started but it also Enabled so after I reboot the virtual machine the htpd is still running. I Have this htpd refactored role here which already looks a little bit more readable in my opinion and It has this Enabled keyword set as well What we are missing is that we are missing the test implementation for htpd to be enabled and To audit we just Edit the test file that I showed you before and Just add another test here that htpd service should be Enabled Let's run the test and See what results we will get so as expected because we didn't have Ansible task to actually enable htpd Before that one test fails the one that we have just added and we see what exactly fails that htpd service should be enabled and it's not and In the report It looks like this so you can see this huge red Box here which shows you what what test has failed precisely now After we run our modified Roll let me just edit the the playbook so it does its job and After that hopefully the tests will pass. So yes, we have only one changed item after the Ansible run Which is exactly the enabled state of htpd service. So let's just make sure that We did everything correctly Yes, so every time you want to change something in your Existing Ansible role or you want to migrate part of Your automation tasks from puppet to Ansible or vice versa or you want to migrate from Ansible to chef or salt stock or anything you can Write the test first Describe your environment in the most suitable way for you Then do the migration and run the tests after you migrated your automation code to another another automation suit or You just refactored it this might save you sometime during the support process or Troubleshooting process That's it hopefully This stock was useful for you. Do you have any questions anyone? Please so yeah the Server spec do have this capability and it's just the question was If server spec shows you the actual state when a test is fail So for example, there was a failed test that showed that File doesn't exist the B file test has failed and it was not clear from the output What is the actual state if it doesn't exist or it's a directory or stuff like that? And the question was if server spec can show you this Yes, it can and for some measures it actually shows you what is what's actually wrong In this case the file was not existent and that's why you saw nothing the actual state was like no state basically, but if The index HTML will be a director instead then you will see that expected VAR www.html index HTML to be a file but got director instead. So yeah Sometimes it doesn't show it but most of the time when the actual state When the object that you are trying to test is there, but it has some State that you don't expect you will see what it is There was someone to my right with a question Yes, so the question is have I been looking to other solutions instead of server spec? The answer is I didn't know that other solutions do exist. Yes So the question was there are actually couple of questions I heard in in this monologue the first question was are there any real real world examples for server spec Yes, they are for example, I use it currently in redhead we have a huge satellite deployment with seven or eight capsules and All the capsules should be set up In the same way, so I have a test suit that test those capsules are the same and it actually Was quite helpful because those capsules were set up by different people and initially More than half of them were set up completely differently and those tests Yes, that's true. Sometimes you just need to set up stuff and then write the ansible automation for it And then there was another question or actually concern that When you use puppet or something that runs an agent on a machine It will ensure the state of this machine anyway Is it right? Yeah Yes, yes, and it's completely fine because If you I have this task for example right now currently to migrate Some subset of puppet manifests from the puppet to ansible because it only they are only one-time tasks that belong to provisioning process not to configuration management process and I wrote tests that ensure That I don't forget to migrate all the manifest I have to migrate and I don't lose anything in the middle you know and more on this you There might be three people involved in the process one guy wrote ansible playbooks. The other guy has this task to Switch part of part of ansible roles to puppet and there is a third guy the actual user of a system those tests kind of Ensure that during the During this migration you don't forget to Control something because this is what happens during Refactoring actually when I was preparing to this presentation and when I was creating when I was refactoring this HTPD role from badly formatted to the good formatted I lost the Part that ensure ownership of a file and The test started to fail and I saw it immediately and so ah, yeah This line was deleted while I was copying and pasting text around so It was useful in for this present for sake of this presentation The main idea is not to lose stuff on your way from one Thing to another thing. Yes So the question is how do I actually ensure that I have all the tests in place? Well, that's where test driven development process kicks in you just have to adhere to the process Yeah, in this case you again It's the same you get the requirements and you ensure that you gathered all the requirements first Then you write your tests based on those requirements And then you implement the other thing that you want to migrate to right so it's There is there is a possibility to Forget to write a test for something But this just means that you don't follow the test driven development process. That's it So the question is why would I use puppet and ansible at the same time because ansible is the is meant to be a provisioning tool and puppet meant to be a configuration management tool and in my like See sudden picture of world. It's completely two different things and different tooling exists to Do these things right? Yeah, I agree. It's arguable, but It's just my personal preference. Let's say So yeah, the comment from the audience was that why Would we use to Two different tools to do to control one machine basically and the suggestion was to have some Jenkins job that will run Ansible periodically And it will ensure the state of the machine instead of running ansible to provision it and then set up puppet on top of this machine and Let it run each half an hour each 15 minutes My comment on this would be that both approaches have Their right to be implemented and again it's just a matter of Adopted practices in each separate sysadmin team We are doing it like this We have ansible playbooks that we are running when we are setting something up initially and then we have puppet Which runs periodically on the machine and ensures that the stuff is set up In a way we expect it to be Sorry, sorry, sorry, there was a question from there and he was earlier than you excuse me. I'm not quite good the question Like okay, so the question is There are two Yeah, there are two approaches in writing tests Test driven like what I have shown you and behavior driven I cannot answer this because I don't know what is behavior driven tests And as I said at the start of the presentation no close to nothing about software testing So I might you know miss the Some concepts some basic concepts Yeah, so the question was In the situation when we use ansible to provision a machine and then Run a puppet periodically on it. How do we actually ensure that? Puppet is actually doing kids job and that it's actually not dead that it's running that it's there Yeah, so The concern is that with ansible you get immediate results that something is wrong. You can see changed or failed Tasks and with puppet. It's not quite Clear because it's a demon that is running on a machine and you don't quite see it, right? So my answer to this is there is monitoring for the for it which ensures that you have a puppet process in place and that We kind of work this around by having Manifest in puppet we change a special file every time and we have a monitoring check for Which checks this file and if it's m time is Larger than half an hour we see an alert this means that the puppet is not doing his job anymore So the question basically is if you can If there is an easy way to match Existing Ansible roles and existing Ansible inventory We use the roles in server spec The answer is there is no easy way for it. You have to know Ruby and kind of code this up I didn't see actually. This is a good question. I was trying to look for some something that is ready but I Had impression that either people don't know about server spec too much. This is why I speak about it right now in the first place Or people just cannot find a use case for it Which is obviously not true taking that there are at least some people here in this in this audience But yeah, there is no easy way or ready to use tools Okay, anything anything else? Any other questions comments? jokes Beers I will have today Okay, thank you very much guys. Thank you for coming Bye