 So, thank you for joining, quite a packed house. I wasn't expecting that. So let me introduce myself. As I was said, my name is Robert Zaheddin Trek. I'm currently working at Red Hat as a Senior Technical Support Engineer for Ansible Products. But before I switched sides, I was for about five years a DevOps engineer working on implementation of DevOps practices in various companies. And was one of the people responsible for architecting how those practices will be implemented, which tools will be used, and how the overall workflows will be used. And during that time, I heavily relied on Ansible to provide me necessary tooling mainly for automation. But as I would like to show in this talk, it can be used for much more. So let's take a look what I would like to talk about today, next 50 minutes. So, first of all, since this is an entry level of beginner talk, I would like to describe what is a DevOps workflow, what I think about it. What I mean when I say we want to scale a DevOps workflow, where can we actually put Ansible, where can we fit it inside this workflow, what it can do for us. And some wise about why to consider it, why to use especially Ansible. So let's begin with some basics about what is a DevOps workflow based on. So it's a course based on DevOps principles and DevOps concepts, which are for me based on four pillars. First one is culture. So in DevOps culture, we want to bring people and processes together. We also have a pillar of automation, which creates for us the natural fabric for all the DevOps practices that we are going to do. And then we need measurements because in DevOps we have this notion of continuous improvement and measurements allow us to see if our changes make sense and if we are moving in the right direction and we need to adjust. And lastly, it's about sharing because we would like to enable collaboration and feedback from other people that were not initially involved in designing DevOps workflows. So in case we are going to scale beyond single project or single team, we want to hear back if what we designed initially made sense for broader audience. So what is the DevOps workflow about? Well, essentially it's about efficiently producing useful outcomes because we want to somehow deliver changes from development and production. We want to continuously improve this method or this workflow of introducing and delivering changes from one cycle to the next. We also think in those workflows about how we can enable new opportunities for not only providing value, but also for providing experimentation. It's definitely not just about velocity. DevOps workflow is not just about how many diplos you can do per day. It's essentially, or it can be essentially thought of as a framework in how we are going to implement DevOps concepts in your organization. And there is of course no one size fits all. So DevOps workflows can be implemented in various ways by various tools and it's also very dependent on how your organization is structured and what best fits for your use case. So this is usually how people depict DevOps workflows. It's an infinite cycle that goes from planning and implementing software changes or code changes through building, releasing, deploying, operating and in production and then going back through monitoring and introducing patches or enhancements. So what drives DevOps workflows for us? So first and foremost it's the need and the want to eliminate repetitive tasks and time singles. This is something that is going to change over time as you progress with your DevOps activities because you will be reducing friction and enabling more frequent iterations. So suddenly what is repetitive task will not be for example, just introducing the same change to hundreds servers but will continuously for example change to how can you enhance your continuous deployment to much set of project, not just a single project that you've started with. So third thing that drives DevOps workflows is automation. We want to apply automation wherever it's possible. So usually when we think about DevOps it's also coupled with the notion of the idea that everything can be written as a code. So the automation usually comes in form of continuous integration and deployment pipelines, infrastructure as a code, for example. So let's take a look on some very, very simple DevOps workflow how it could look like. So as a first step we have we are writing some new code based on our version control flow. So for example you are implementing a Git flow or you are implementing something much more simpler. You introduce the change, version in your control then your CI tool kicks in, detects the changes, pulls them, tries to start the build pipeline that also triggers any required testing environments to be provisioned and the builds to be deployed on them so we can start testing. You do all the testing that you need and report back the results. Once this happened you maybe have some improvement process so for example you are using pull requests so somebody has to approve your code changes and then merge them. Or maybe you are so far ahead that you have automated mergers so your code will be merged to some production branch for example. This again kicks in the CI which again starts pipeline but this time it will be pipeline for example to production. And after we are in production then we have to continuously monitor and manage the desired state of the deployed software. So when you look at this example workflow something that is very nice about it is that most of this is CI driven or Git driven and which allows us to heavily rely on automation which tells us that the majority of the scaling we want to do in the workflow will be based on the automation toolbox that we will choose. So how we can scale our continuous integration deployment software and pipelines how we can scale our infrastructure as a code tools as well. One thing that we need when we want to scale we need our tools to keep up with the increased demand. So when our infrastructure grows or our organization needs grows then we need to make sure that the tools can keep up with them. And then we also when we try to scale this workflow we want to enable collaboration and reusability of something that we already created so mainly the automation fabric. And lastly we would like to enable soft service to others so the tools that we pick again then provide access to the automation and to the overall infrastructure as a soft service to others so we don't have to be bothered by creating for example tickets and having somebody manually creating for us or hitting some button to deploy our software. So once we know what is DevOps workflow so let's take a look on what Ansible Tower and Ansible Engine can give us. So basically you can think about Ansible Engine as a university automation language which can scale through different parts of your organization not only the IT. It provides you a human readable format so you write your automation in YAMU files and it comes with nice built-in features that will allow you to create this collaboration and reusability. It will allow you to collaborate and reuse the automation. But that's not all you probably will need so Ansible Tower comes in handy when you want to glue together different parts of the workflow that you will create because it gives you a transparent way how to execute the automation defined in Ansible. It also provides a solution how to scale which is that something that we want to scale our tools with increasing demands and it also allows us to create the soft service. So some building blocks of Ansible. The first thing that you will see when you start working with Ansible is that there is a great set of modules which are the basic units of code that are executed that will be making the changes on the desired targets. Those modules are used in something called tasks which are the smallest unit of work. Examples can be starting and stopping a service or removing something from a load balancer. We can bundle tasks in something called roles which are shareable collections of tasks. And when I say shareable then we have because we have tools called Ansible Galaxy and molecule that allows us to share the roles with others and molecule is a testing framework that allows us to create a testable roles so we can trust our automation. Examples of roles can be installation and configures to whole web server, deploying an application. So you will basically create all the steps and define all the steps that are required to deploy your application and you can put them in a role. Playbooks are the high level description of desired outcomes so playbooks can be seen as an ordered collection of tasks which means that you can put all the tasks in the playbook as well or you can use all the roles that you created and put them in a playbook. Playbook also requires that you give it an inventory which is the collection of hosts that you want to run your automation against. And inventory is the list of targets. So some examples which can be seen. This is an example of tasks. This 19 lines of code does one thing and that it deploys EC2 instance in a desired region with some instance type image and we can also define how many of them we want to deploy. As you can see the module is this little part here that's called EC2 which is something that comes with Ansible you don't have to write it. You just provide your interface and you can fill in all the necessary variables or parameters and that's all basically. The things on the right side is our definition of variables so I'm not putting in aesthetic values but I'm allowing these values to be later overridden. This is how an actual role looks like. So this is the role that has the task from previous slide. All the tasks necessary for deployment and provisioning of the EC2 instance are in one directory. I have two tasks list here just because I would like to do some pre-flight checks which are again some tasks that will just execute different modules to let me know if I already deployed the same instance with the same parameters so I don't do the job twice. You can also define some default variables which will be inputted in the task if you don't provide them later. This is the definition of molecule which is the testing framework that I mentioned before. You can, this way, you can define a test that will verify that the basic behavior of your role is working and you don't have any bugs. Some example how it can look like without going into much detail is you define a test sequence which is the set of steps that molecule comes with and it will check some basic errors, syntax errors. It will execute your playbook and you can run all that stuff in Docker containers. So moving forward, this is an example of playbook which is using the same role as we saw before by including it in this little part here. We also, as I said, have to provide a list of hosts that allows us to target or deploy the tasks to different targets. In this particular instance, I'm running it against local hosts just because the EC2 module is not actually running against AWS instance, it's running against AWS API and so I need to run it on local hosts because that's where I have the libraries and all the dependencies to ask or communicate with AWS API. A bunch of other things that I'm doing here is that in the role itself, I define it in such a way that it will take on a static variable. So for example, when I would like to simultaneously create it 10 different instances with 10 different configurations, I would have to run the role 10 times in the playbook. So to make it more efficient, I just override the variables in the playbook and I use looping mechanism that allows me to define in some more complex structure what instances I want to run and then loop over them and pass the expected values in every run. So this is an example of how a playbook can look like and since I said that the roles can be shared and then we can use something called Ansible Galaxy for it. This is a prescription for using Ansible Galaxy which is a simple mechanism how you can define from where your roles should come. By default Ansible is looking to a very specific directory in the path. It's called the roles directory and if something is not defined there and you are trying to use it in your playbook, it will complain and will not run but we can define external sources like for example here that the role that I'm actually calling AWS-EC2 should come from a different Git repository. This way we can define and create our automation in such a way that we can create this small building blocks that will be living in different repositories outside of our playbooks, outside of the repository with our playbooks and then anybody can reuse them later. So we would like, since we are talking about workflow and workflow is as I said, highly tied to CI system as I showed in the first example. This is a, these are two examples how we can execute our automation. In the first picture on the top we are directly running Ansible Playbook command which is something you can think of naturally but one of the problems with executing or binding the CI with the automation in this way is that it's wholly dependent on how much power you have in the CI. So when we want to scale and when we want to run more tasks and we want to automate much more of our workflow certainly if your CI cannot scale with it you are in the position that your automation cannot keep up with your demand. On the other hand, the second picture on the bottom doesn't directly run the playbooks but rather queries an Ansible Tower which is done through Tower API and therefore the execution is not done on the CI system itself but rather on completely different transparent system that provides us execution Ansible Playbooks as a service and to take a look how Tower is providing us that to understand some basic blocks in Tower. So before we can run any job or any Ansible Playbook we have to define all the parameters that you would have to define in the Ansible Playbook command if you run it from CLI yourselves or in the Playbook YAML file itself. So we have to provide credentials that will trigger for example, they can be SSH keys, they can be AWS credentials and so on. We also have to provide some way the inventories to Tower. We have to tell it from where it has to take should take the roles and the playbooks which is done through object call projects and then we can start defining our job template. Last resource object that I would like to mention here is that it's jobs and jobs is just the execution of the job template so it's a running instance. On the left side we can see that not only we can define all these parameters of Ansible in Tower we can actually segment them so not everybody has to see all the credentials, all the inventories, all the projects. That is done through a role-based access control system that Tower comes with and it allows you to create organizations, teams and assign users to these organizations and teams and then assign the objects to particular organization with the correct permissions. So for example, when you have people that only want to, only needs to execute the playbooks but don't need to make any changes to them, you can define it in Tower in such a way so your automation allows you to only run the automation. So this is how you will bundle the job template with all the parameters so you will tell it which playbook to run from which project, which inventory and what credentials it should be taking. On top of these basic blocks, Ansible Tower provides us additional features. One of them is that a workflow that you can put all the project updates, inventory update tasks and the job templates itself into something called a workflow and that way you can build different pipelines. So when we would look at the example from the previous few slides where we were defining how to run our deployment through CI, then what actually was run when we trigger it through Tower is run this workflow and what it did is it updated our project where we define all the automation, it run a job template that provision our EC2 instance, then it updated the inventory from AWS, so we have the new instance in our inventory, it can continue in the deployment, then we deploy the application and if anything happened and it was unexpected we could always roll back. You can also through Tower enable something called prompt on launch, which is just a functionality that allows you to ask the user for the input when he's trying to run the job template. This way you can provide arbitrary extra data variables or change the defaults that you define there. You can also trigger the job templates in completely different way. So we have an ocean of callbacks there and when we look back what callbacks allows us to do is that in this step when we were provisioning EC2 instance we had to do additional two steps, we had to update the inventory and then we had to deploy the application. But with callbacks we can shorten this cycle by just provisioning the EC2 instance with something called a callback script that will have a specific URL of the job template which is deploying our application and when the instance is booted up and running this URL is called and we don't have to write the next three tasks in the workflow but rather the instance itself will trigger this process through the Tower API. So let's go back. Next good thing that Tower provides us on top of pure Ansible is that when we would run the playbooks or the whole automation through a CI system or command line we usually lose a lot of information. So Tower provides a lot of output Ansible provides a lot of output on what it's doing and when you want to enable or expand your automation and scale your pipeline then you would like to know what happened who triggered it and what was the result and there are many tools for that so you can use, you probably can use Splunk for monitoring of logs and other metrics you can use Elasticsearch, Logstash, Kibana Stack and Tower allows you to send all this extra data about the runs to these services as well so you can manage all your monitoring and logging through specific tools and you don't have to have your logs through different parts of your workflow and your toolbox. Next Tower solves our scaling requirements in two ways. It has an option clustering so you can create a cluster of Tower nodes which are not for high availability but they are for spreading the load so for giving you much more execution power that you would have on a single instance. The second approach is isolated nodes which is a special type of node which is not connected to a cluster but it allows you to execute the playbooks the same way as you would do from normal clustered instance but this instance needs only be able to answer on SSH since Tower will transparently allow you to run any playbook and any automation on this isolated node by uploading all the necessary parts of it and then collecting all the outputs so we are still having the same functionality as if we run it in a clustered instance. This is handy in cases where you have secured part of your network or DMZ where you don't want to put additional services but you still want to use the same automation. And finally, what Tower allows you to do with these clusters and isolated nodes is it allows you to put them in something called instance groups that give you the ability to dedicate resources. So you probably will need some resources to run all the testing environments all the provisioning to production and you don't necessarily need to have all the resources assigned to for everything so with instance groups you can segment which tasks run on which instances and that way you can better utilize your infrastructure and again it's all provided transparently through Tower so we are not tidying into any additional tool in your workflow but it's all done on a single location. And last thing is something that we added in the latest release is something called job slicing which is a mechanism that we introduce for scaling single job execution. So when you run a playbook either from command line or even from Tower before the latest version it was only run from single host which has the limitation that how much things you can do and how much in parallel if you can do the things it was defined by something called forks and the forks are just parallel processes. So when you for example wanted to run your automation again thousand servers and you set your forks count to 80 so you needed 80 processors to be on that single server to parallel run all the 80 processes and all the 80 parallel connections to the servers and it was going sequentially by every 80 servers to until it finishes. But with slicing you can do something cooler and that's defining how many different groups of your inventory will be created. So for example when we had a thousand servers and we set slicing to 10 then we would actually run the same playbook 10 times on 10 different servers but every time only at against 100 servers and this way we can scale and create a much better execution times. So put it all together and this is how a simple automation looks like. So on top we have our playbooks that are provided to Tower. We have Ansible Tower that provides us the role-based access control, provides us the visibility and what's going on when we execute the jobs, provides us the user interface either through the web UI or the Tower API. So it's a normal REST API and then it runs transparently everything that Ansible Engine provides you. So everything that you can do with Ansible Engine you can transparently then do in Ansible Tower as well. So now let's take a look how we can actually plug all this functionality and all these features into our DevOps pipeline. So first thing that I would like to take a look on is the stages of building and testing. So Ansible isn't a built to, isn't a tested testing to. While you can still execute arbitrary commands through it, it's not meant to replace your CI. But what you usually need to do in these stages of the DevOps workflow is you need to build infrastructure for testing. So when we go build unit testing that can be run on the CI itself and then you probably in the integration phase or the acceptance test you need to have some additional infrastructure deployed. And for this, you can utilize Ansible as a part of this stage by using the same automation for creating the test environments as you would use for creating a production environments and essentially bringing all the environments to be identical. Next, we have releases. Again, this is something that Ansible isn't particularly built for. So it doesn't have any specific modules that allows you to interact with various repositories or allows you to upload something to various repositories like Artifactory or Nexus if you know what they are. But you usually do this through the CI too. So you install some plugin to Jenkins that allows you to upload things to your location or you write a best script for it. I argue that you can do the same thing with Ansible by just utilizing the generic shell or URI modules. If you are using Docker images, then you are on the better side because there is a native way how to manage Docker images through Ansible. But with the URI or shell module we can also manage a release process to arbitrary tools. And this is the example how it could be done. So what this set of tasks is doing, it's uploading and promoting a built artifact from our CI tool. So on the first task, I've written this whole automation against a tool called Artifactory which is a universal repository management system. So you can have under one umbrella or your RPMs, jar files, Docker images and anything else. So this tool allows you to have a promotion process which is a process how you can have multiple repositories for the same artifact and you can promote the artifact between different repositories depending on which stage of your DevOps workflow you are. So for example, when you build the pull request and it's just built from the first development changes it will go to development repository and then when the first test pass it will go to additional QA or staging repository and if all the test pass then you can promote it to production. And to do that you can write Groovy script. So what this automation is doing is ensuring that our promotion plugin is present then making sure that it's loaded and we can use it. And during the pipeline when our test pass we can promote by just squaring the API our artifact to next stage or the next repository. And this is something that as you can see again parametrize so it's not static. So if you want to add different or you can run it, you want to run it against different repositories or different builds you can still do it. The only thing that you need to provide is the correct variables. So this automation always works as expected. The last two things that we can connect Ansible to in our DevOps workflow are deploy and operate. These are the two parts that Ansible was actually built for in allow you to use almost all modules for doing these two things. So initially or usually people start with dying to deploy through the CI to itself. So they write some simple scripts inside the CI tool and that will deploy the artifacts to their desired location. But by using Ansible we can again detach this process from the CI tool and putting it back in our automation language so it will be unified and not independent on which CI tool we are going to use eventually. Some interesting modules that you can look for for deployment are modules that can manipulate monitoring because usually when we are deploying things we want to make sure that we are not figuring any alarms that are making sure that our servers or our systems are running fine. When you have a clustered environment you probably need to integrate load balancers. Then we have this provisioning modules which is probably everything in the cloud section of the documentation so as I used the EC2 module or we can use Azure modules, Google Compute Engine modules or OpenStack. If we are deploying our software through some system services then we can interact with them as well and the deployment itself can be done as I showed before with the get URL or we can use simple copying of files. The last thing is something that is very nice to have when you are doing growing updates you want to make sure that the deployment actually succeeded and if you are adding a new server to the cluster that you are adding something that is working. So some examples how we can again take something built in artifactory so we can simply just call a URL and deploy the desired artifact to the desired location from the system that we are using. And as I mentioned we can also verify that after we deployed our software and it's for example web service that it is responding to our request and it's working as expected. This is a basic execution of your module which just takes a URL. I'm defining here something called a health endpoint which is usual way how to define a health check for your web service. And I'm expecting that it will be turning an HTTP 200 code and then in the content of the return from this call I'm just looking if there is a word running. You can look for any arbitrary text string here just depends on how you define your health check and what is returning. If it returning JSON you can even process JSON here. So some more complex example is to put it all together we can define the whole rolling update scenario in our Ansible automation. So there are two new things here. We have something called post tasks and pre tasks which are the concept in Ansible that execute some additional task before we execute roles or some regular tasks or something that happens after all the execution from the roles and regular tasks happen. So what I do here is in the, before we even start deploying anything I just remove the server that I'm going to deploy to from load balancer. I'm then waiting that there are no existing connections to the server. Afterwards I run my deployment automation which I defined as a role. So it's replicable. Afterwards when it's done I again go and add the server back to the load balancer and I'm doing it after I check that the server is starting to respond. So the last part is operation which operate phase and which is basically data operations after we deploy software. So if you know what the data operations are it's basically everything that you do after you successfully deployed your software to production. It's continuous monitoring, it's managing logs, it's continuously patching the systems, having a disaster recovery scenarios implemented. So one example can be for example here that after I deployed my great log service then I want to update how I'm managing all my indexes there which is the place where all the data are stored and what this does is just deploy a new cron jobs that will move my data between different tiers of storages and archive what I don't need anymore. The second example here is something from a disaster recovery point of view is that for example in production when I have a Postgres crustler one of the masternodes can fail and I need suddenly promote a slave instead of doing it in manual way or executing some script I can define it as a playbook which will do this task all the time the same way. So what it does it just makes sure that configuration file is in correct location. It turns off a configuration option and in one file promotes the slave and then restart the service so I'm running now the database from the slave and not the master. So to summarize this talk one of the things that you should take away from it I hope so is that the DevOps workflow is just a framework for implementing DevOps practices it's not what DevOps is all about that Ansible Engine and Plus Tower gives you a scalable transparent automation that you can use throughout the whole workflow. You should think of utilizing engine to provide a common language how you define your automation. Use Tower when you want to delegate usage because it allows you to transplantly execute playbooks in the same way through either API or the user interface and you should utilize Tower when you want to scale your automation because it has the concept of clustering or isolated notes and allows you to just 3D automation as a service the rest of the tools in your workflow and since DevOps workflow will to the most part it always heavily dependent on your automation because most of the steps are dependent on it therefore the overall scaling of the workflow will be always based on the automation tools that you choose and how they can scale themselves. So that's all from me. Thank you.