 So next talk we have for test-driven development with Ennisville by Pooja and Info about her is like Pooja is an application developer in Roadrunner and has been developing software in Ruby on Rails and Python Django based web app Okay. Hello everyone Thanks for attending the talk and I in turn I'll just try and do justice to all the caffeine that you have inside you not put you back to the sleep Little bit about myself. I'm currently working with Roadrunner This is a pretty new startup six minutes old based out of Bangalore It is into hyper-local delivery. I don't know how many of you have heard about it Prior to that I was working with ThoughtWorks for last three years Mostly engaged into developing web apps based on Ruby on Rails and Python Django Also, I've been doing on and off DevOps work and today Presenting on the same how to test drive your infrastructure because I think mostly Most of the agile conferences here and there a lot of lot of talks we receive on how to what are the best practices around application code or What are the why do we need test-driven development or why do we need continuous delivery and all that focuses around your app code But what about your infrastructure code? So so that is what I'll be presenting today That's my Twitter handle. I would be happy to receive any feedback or opinions on this All right, so so again to start with test-driven development How many of you have been using this on a regular basis? Okay, okay Just to just to brief about the concept. It's a simple concept, right? So it's for developers where the feedback cycle is faster So before any upcoming feature you just start Writing tests for that feature you test the required behavior from that feature You you run the test and see those tests failing then you then you implement the feature and see it pass So as to know what your feature is exactly supposed to do What are the benefits straightforward, right? So receive your failing feedback faster and not late into production or any other pre-live environments To to to give you a clean code to give to improve your overall design And and and I think so this is just my personal opinion Along with all these benefits It has also some in a way helped us in the continuous delivery process of application development Right so so to define or to just explain the regular flow path of how app code gets deployed to the production That's a developer pushing his code from his local machine to some version control system and from there, there's a CI server sitting and pulling this code on and off and and this CI server enforces some practice Running the test around the around the artifact that has that it has just pulled If those tests fail right that artifact is rejected and not pushed to production But when it passes it goes through series of servers and environments and finally push to production Right, this is how it goes Right, okay But when it comes to designing infrastructure The usual life cycle begins with bootstrapping a server manually, right? I want to develop an application which is on Ruby on Rails and which needs engine X server and which needs My SQL as the database all right create create a server spin up all this Install all these things on the server and get my thing running on some environment some environment I'm not calling it production yet and then image the same box into multiple servers and then built up what is then Ocounter's production environment Why why it isn't so right traditionally Infrastructure has been lagged far behind all this development practices in terms of CI in terms of code quality in terms of in terms of testing per se But but I think thanks to community when it when the realization hit them there was evolution of the provisioning tools like Ansible Chef puppet and And I am to say here that it did make the life simpler rather than just going and installing things manually and then Then thinking about what did I install and what what I need next if I want to just duplicate the environment This gives an automation script where you have where you have a proper set of knowledge that what I need on my server So and civil and all this provisioning tools made the life simpler, but but after after this was introduced What is the flow path of infrastructure change? I'm not saying the this bootstrapping So it's now your server has been bootstrapped and now you want to add some change to the production environment How does that infrastructure change is deployed to production? It goes somewhat like this So a developer just sits on his machine pushes the code to the to the version control system and from there it goes to production many of Many of you will disagree saying that no we we do Test it manually on some some pre-live environments because we don't want to screw up the production environment right we do but then But then how many times? We are confident about a simple change and there exists a syntactical error and unable to start your server right How and and this is just a smaller picture sitting having one developer and one production server Think of this and scale it to multiple servers having in a distributed environment where there are multiple dev sitting Can you enforce each and everybody to just test the their infrastructure change? No, right? There needs to be some tooling that enforces certain practices in this deployment process However first I would like to give you the demo of how things breaks on production. So this is a static page that I have served on an engine x server and This is what I've deployed on a vagrant. I've automated it wine and civil script So now to give you the Code sample this is my bootstraps script bootstrapping script where I have an engine x server and I'm copying certain The configs here being there is one server listening on port 80 All I have to change now is to deploy another application on another port Right, so I have already done this. I'm just gonna check out another branch All right, and then I'm gonna apply the Ansible playbook again on Just a prior question. How many of you have worked in with Ansible before? For others just a quick quick introduction Ansible is is nothing it just gets set of instructions and installs whatever You are asking to install on a particular environment. That's also so to show you here So when when I say the task those are just the simple tasks that will run on on a particular host Those are the tasks that I'm gonna run on on my vagrant machine right now after applying this Sorry, this is just a dynamic IP thing Right, so so it ran set of tasks you see the engine x copy config has been changed Which means there is some change in engine x config which also triggered the engine x restart So it restarted the engine x and the output shows that everything went through which means now my environment should be up And I should be able to have two applications one deployed Which was earlier on port 80 and the other one which I just just added I deployed it on port 3000 All of this is port forwarded to the host machine. I'm just gonna refresh the previous application and Now it's no more there I'm testing whether the new one got deployed I'm port forwarding it to a 9000 and There's something else. There's no application deployed there, right? So so when you see here basically there was something else running on my port 3000 But none of the script told me so right so so consider this scenario and and changing Or adding something on a new port is just a simplistic example, but it did make your engine x stop, right? So so now consider coming back to my presentation Now consider there were tests around this And and I had run those tests on my Local environment and and that would have given me a failure saying no this particular change is not ready to go on Production and something will go wrong Let's let's just say something and the test is not giving you feedback that what it is What is exactly going wrong on the server? So so would that help a little bit? right So so I started thinking in those direction. I had spent a lot of time Developing application code and that's why when I moved to infrastructure. I started thinking in the same lines. So So I started Reading about can we can I write test for my infrastructure code and there I Learned about this term test-driven infrastructure. So So Okay, so when you there are five principles basically that define the infrastructure testing and those five principles being One to document all your system requirements have a high level system requirements Written somewhere so that you know what behavior is expected out of your system Once you know what the what is the system is expected to be like then you write test around that behavior, right? So do I have so for example, right? So I know that in a production my production environment will look like there will be two app servers and There will be a there will be a mySQL server running So basic behavior will be both the app server should be up my SQL server should be up And all this server should be able to talk to To themselves among themselves, right? So this is the simple system behavior that I'm setting up Run those tests see them failing and then you implement actual infrastructure code to bring those environments up to bring those servers up And then see them having those behavior Being implemented and when you see that you run your test and you see them passing Also, obviously so you version your infrastructure code because Like always right it gives you easy collaboration The most important point is to do this continuously, right? Because If you compare the frequency of application code being deployed versus infrastructure code being deployed You would say there's no need right because I probably deploy an infrastructure change once in two months Now consider the probability of application code breaking your in breaking your production environment versus your infrastructure code breaking your environment The percentage is probably high so so I would say that there's need of doing this continuously so that you enforce the practices Which will going forward which going forward will all obviously give a safer environment For this development as well Okay, so so now when I read about this this all principles and everything in theory it looked fine What about implementing them so I started writing tests around my and civil script This is my and civil playbook I'm just copying the engine X configuration from a template file to the on the remote host So how do I test this so I started thinking on can I have? Test around whether the file is being copied or whether the file contains proper route whether the file contains Application being deployed on so and so port right and that's how I wrote my test So my test look like this particular thing should be a file This file should contain so and so text so that when whenever someone changed this particular line It will fail What if someone adds something new it won't right so which means then you have to test drive your Infrastructure where you have to add a test before you add a file Does that make sense to me it did but not much because I'm not testing the higher level behavior I'm not Testing if this file contains blah blah blah will give me certain assurance that if it changes if my project route changes It will show me but will it give me error like what I did Will it give me error when I started something on a new port and there was existing something this won't right so So I started doing research around this and There's a there's a big Google group discussion around How to whether we should test drive your Ansible or not? Why specifically around Ansible all of us know that it's a very declarative language Unlike chef and puppet which claims themselves in as a very declarative one Ansible has just gone beyond limits With with puppet I have worked with puppet I haven't worked with chef But with puppet there is some aptly there is some logic in your code with Ansible It's plain simple. So when I'm testing the the way I did before I'm doing nothing but testing Ansible modules and I don't want to test because I trust the community So when I read those Google group discussion something which really caught my eyes was this line I don't want to test what what it is being implemented But I want to test the outcome of this right if I do a particular thing What it will result into this is something that I want to test and and and so These questions were right that can I ping this particular website is my server up? Are these servers able to talk to each other and not? this right, so So in the same thing I also came across server spec Have you heard of this? Okay, so server spec is a Ruby library which gives you a way to test the system Behavior it's a simple our spec matches, but not for your application code where you just save one expect one is equal to one It gives you matches like expect a particular port is listening expect a particular Expect a particular serve when I do a curl a particular output is received and and this this way So so I started writing server spec for my implementation so to just Show you a bit of code So this is how the sample spec file would look like it just test whether port 80 is listening or not and and When I do a curl on that what is the output and when I added the new When I added the new app being deployed on port 3000 I also added this test so so from the implement from the demo Before this you would remember that it failed right the port three thousand was listening But there was something else running on it. So now when I run my specs on my local machine All right those failed So so if you look at this further It will tell me that the unit test that I wrote is fine that particular thing is file And it has this particular route, but when I say is port 80 listening. No When I do local local os 180, it's failing but port 3000 is listening So in a way my test start telling me that something changed that you made is exactly not Is exactly failing because there is something else running on port 3000 right? So if you read read the failure Well, and if you understand how server spec behaves and what what exact behavior are you expecting out of your? Environment I think those can give you a proper feedback in a way rather than having no test around your infrastructure so So I started my talk with test driven development where where a test is written first for a feature then it's developed and deployed to Production which looks like this For infrastructure, I would say the sequence won't matter as long as everything is tested So I personally was fine if these two swapped If if I develop the code first and then I test that the behavior is still intact and then The easiest thing or the main thing for me to get the code deployed to the production environment somewhat look like this a Person changing some infrastructure code and pushing it to the put to the version control Was pulled by CI see I running Swarva spec on a vagrant machine right so so this vagrant machine was introduced just for solely for the purpose of Testing it testing it and being it production like right so so at the end what I'm expecting of my production server should be Exactly replicated on this vagrant box If it fails it fails You don't have to push this to to the production because you don't want your production to to just Go down And if it passes It gives a fair amount of confidence that whatever I did Was at least right because it was tested if it failed maybe it was some something due to the hardware failure Maybe it was because of a network failure But but a developer can see that I did it and I know that my code doesn't break As a developer you would not want anything less do you so why vagrant? Why not run it against any environment? Why not run it against you at your staging right as we do now manually? Test it against a running environment and and see if it passes it push it to the production Do you really want to mess up your environment and then revert your changes and apply it again? If you have something virtualization at your hand, why not use it? It does give CI to enforce certain practices around around your deployment and delivery process Where where it ensures that the the set of tests are run against some some server and it knows that it will pass More than that it also gives liberty to any developer Just spin up a vagrant box on his local machine and see his changes are working fine Why to wait for CI to tell you that right? So so I think as long as you get the feedback faster in cycle I am fine with with vagrant being spend up by a CI server or any any local developer writing his code there Little bit to talk about server spec as I said earlier It's it's a simple Ruby library and having our spec like matches If you haven't looked at it, I would really suggest to to have a look at it. It's worth looking at even if you have not even Automated your servers using any provisioning tools It's not too late to just try testing it its behavior because if it fails it will it will let you know right away that it's not working so I did all of this and And and all my work was just just screwed by these two terms kitchen ansible Which does exactly same what I told so I did everything manually and then this particular gem does everything what I told It it has it takes an ansible playbook as input It runs that ansible playbook on on a pro on a platform it can be vagrant It can be Docker anything and then it then it runs set of test on that particular platform which is what I just did right and It's a yamal file Right, so it it everything to configure here is just in three blocks One is you give you a provisioner right? So that's the name of the playbook that you want to apply those are the host where you want to apply it That's my provider. So it's a vagrant box It's an open to and and set of configurations around it which you want to just Which are just optional and then it takes a block where you just give name of your test file Test file is nothing, but it could be any any test that you want it could be a Server spec it could be a yamal test. It could be anything. It just takes the block and run that test file and and it just Yeah, so I think kitchen ansible was something which just Which just made everything easy like a breeze To talk about why should we do the test driven infrastructure, right? What are the benefits of doing this? Pretty much all the previous slides cover this but just to just to sum it up It will give you readily verify behavior, which means I do some change on my local machine I want to verify it right away. I don't want to apply it to any environment and see that so that that gives this is Available at hand you can fail fast and not worry about it You can improve your overall architecture and the major one. You can always always Do refactoring without worrying what will break? Awesome. So so everything is done. I've set up my server. I've written ansible script. I've tested it and everything works fine So I have four nodes now and all of them are being tested What if one of the node goes down not because of any of this it goes down because there was a network failure It goes down because there was low disk space So it's always suggested to have some sort of net some sort of system monitoring around this There are a lot of tools available at hand. There's a Monet there is ingenious. Sorry Nagios. There is a New Relic for that matter, but but yeah every team ensures that this is in place and and They should Taking the opportunity of this. I would like to just just share the knowledge which I came across while doing this presentation This is a new tool that has been introduced a couple of days ago. It's called auto They claim it as being a successor of vagrant So it not only knows how to develop your application. It also knows how to deploy it Around this they've developed with your Okay, so they've developed with Dependencies being your first-class feature So you just you just name that my apple my code or the application that I'm developing is a Ruby application And it it installs all sort of dependencies that it needs. So it's it's having both a look at it's very new But but it's stable. I am I'm giving it a try right now All right. Thanks I'm going to provision your vagrant or Docker and then run set of tests around it So it's not going to market its actual implementation on a vagrant box on a virtual environment Right. So if you're doing it locally You may or may not but for CI I would suggest Not doing it is is better because then you have your one server running always like production But there's no test. There's no application test around it So you always have that vagrant environment up and run set of tests around it But you but if it goes down, nothing is to harm, right? Right No, you just you just either you revert your changes and apply it again or you just Yeah, or you recreate but I think if reverting works, then you don't have to recreate right not by the system Hi Yeah, yeah, it's a good question that that was continuously on my mind when when I was working on it So so I think as I as I get the example having unit test around your infrastructure code Does not really Make help you having reaching the ultimate goal Testing the actual behavior. I could test that particular template file contains this I could test that my particular server is being installed I could test my particular package is being installed but but But that would just mean I'm testing my playbook and it will exactly look my Look like as my infrastructure provision file Versus if you if you have a system level test, which means system level test will test the system behavior It will only come in place when you have the system running So you have to develop first you have to deploy it on an environment and then test the system behavior Okay, so I'm not going against unit test. I'm just saying it doesn't make it It is not relevant in case of infrastructure Development From your experience, what do you see are the limitations of test-driven infrastructure, okay? As I said, right? So I I did not go for testing with puppet But I did with Ansible and my test looked exactly like the playbook To me when I was writing test first, I felt like I'm writing the playbook itself One is that two is infrastructure is about spinning up a server So You won't be able to test a lot of things unless it's implemented on a particular environment That is that is the biggest limitation because you can you can test that a particular Thing has been written, but you cannot implement it. It has been installed unless it is actually installed or you have to mock it So so those are the limitations that I faced Yeah, hi, I work on puppets. Oh, could you just give me a brief on what's a different between puppet and Ansible? I mean In terms of writing test cases as well as in terms of structure of the code How different is that? Okay, so I Did not Work on testing the puppet thing. I was only looking at developing it to me it one. It's a Ruby file It's not a YAML, right? So so there are functions there are Passing of variables around with Ansible. It's a YAML file. So there's a hierarchy defined there That's the bigger one in terms of how it looks in terms of implementation also I think it's pretty much same. It's works on roles and host and everything. So the implementation doesn't change mostly Also since it's function in the puppet it gives you liberty to write your logic there with Ansible There are no function. So you have to use existing modules or you can create your custom Would you how would you that? Test drive a cloud infrastructure such as a AWS which uses a simply email service a Q service and S3 which are you know which have been Which have been a spin up from from the API From the from the Amazon Applied okay, okay, so so to understand your question better. How do you test in case of? Something like AWS where you we just pin up from the from from the AWS portal and don't write script for it Okay Yeah, I never came across that scenario But but yeah, I I would just think that whatever is expected out of your end system should be Should be replicated somewhere and tested. So if you have to have some something with AWS if you are spinning up with with AWS users Know your system requirements first and and have that system be here Replicated in a vagrant environment or in any virtual environment and test it first and then use create your production environment So just to add to the limitations part So one of the challenges that I see is that yet fine runs fine on vagrant and Docker But you go to AWS or easy to instances you have another layer called routing the boxes are different You could still be running Docker, but then you have another element of routing and all that so you don't actually Get the same environment in production that you can do it There are quite quite a large number of differences between what you do on your system Yeah, I agree with that And but but I think somewhere it has to start right So why not we we start doing testing whatever is present at hand and then Then actually talking about the limitation in open forums where then it the weather in the community can take it forward Or we ourselves can take it forward to to write something around it to to make it production like and to have something Available which can be exactly replicated as production thing Other questions Nothing. Okay. One question. I want to ask. Yeah, I've been using vagrant for this Serving like I Python and Apache five so like if I use it auto have to write code on Ruby or Python Sorry, your application is in Python. Yeah application is Python, but a vagrant allows a Ruby only, right? Writing a vagrant file Yeah, yeah, yeah, so auto has auto file where you just So it's it's again very declarative. It's something like Ansible. So you just say my app type is Python Right. So so have a look at it. It's Ruby, but it's it's again much declarative than what vagrant was