 Okay, I'll start. Can you hear me fine? So I'm really tired. I'm from I work for reddit. Is it better now mute? How is it now? Okay, so I'm really tired. I work for reddit currently in the container management team and the topic of this presentation is automating your data center with Ansible and manage IQ and There will be a Q&A session in the end, but you can also interact with me if you want So the agenda for today is first I will give some motivation and present the problem that we wanted to solve. I Will then go on to describe manage IQ and Ansible which are key players in this solution the core of this talk will be presenting manage IQ and Ansible modules and We will discuss manage IQ automation engine and the current state of integration Inside manage IQ is Ansible and then we will mention some items from the roadmap so as I said I work in the container management team and Our team's mission is to provide a solution for container management For running containers in clustered environments based on OpenShift and Kubernetes An example of what that means is there is a team inside reddit Called the OpenShift Ops whose mission is to manage a large number of containers of Clusters who are paid for by customers These are customers who want to enjoy all the benefits of an environment like OpenShift, but they don't want to have to deal with the overhead of managing Such a cluster and we started talking to those guys a few months back They are really an awesome team and we've learned A lot of stuff from them. So we try to understand what they need so The third thing that they need is when they need to deploy and provision a new cluster Obviously, they need some kind of consistent way of doing that Because the environment is quite complex as you will see They need a very fine-grained solution for monitoring where they can Define exactly the kind of triggers that they want or what kind of situation they're looking to Be notified on. They also need all kind of metrics to understand workloads and Utilization and both of these these items are achieved using a Hocular time series database that is running inside the cluster So time series database is the right kind of database for metrics Basically, every row is a timestamp and a value They also need aggregated logging from all of the host both for infrastructure and for the application logging For that they using they are using an elastic search cluster also deployed inside the environment They need to deal with charging customers, which is basically related To measuring because it involves connecting some kind of cost to to a metric it can be a physical cost a physical Thing Like CPU or memory and it can be felt something virtual because maybe want to charge based on the usage of a certain container image And there's a lot more and it's very obvious that we need very good automation here And that's what this talk is about. So let's take a step back and talk about The role in general of an ops team today So you can see organizations today consume a multitude of platforms from different sources for different purposes And if you don't have one place where you can look at it all and make sense It might be very hard to manage and This is what manage IQ entails. So one open source platform to rule them all Managing Q is an open source management platform for hybrid IT It is the open source project that powers reddit cloud forms And it's basically a single pane of glass where you can look at all of your VMs containers storage network The first thing that you get is a rich inventory which is up to date and it's cross linked Which means that if you have like a Kubernetes cluster running on top of Open-stack VMs, you will be able to drill down from an individual container to the VM on which it is running and to the host Manage IQ collect performance and utilization data follow from all of the all of your environment Which enables you to understand your workloads to do capacity plan planning finite news resources And that also includes reporting based on performance and charge back Which is kind of the billing that we talked about smart state analysis is the name that we give to the ability to inspect object in real time I'll give you an example from the container world is that we can mount a container image inside the cluster And we can figure out what's running inside including packages and we can also test it against Latest CV data using open scap So we can find out if there are any vulnerabilities in a container image Compliance is the ability to define in free language certain conditions That's can save an object is compliant or not And also take action take action on it. So another example for more containers is Mixing up with the previous items We can define a policy that says when a new container image is discovered in the cluster Scan it if it has any CVs of let's say high severity market is non-compliant We can also get open shift not to run that image after that Service catalog is abundant list of resources where end user can consume resources So that the user can just order a VM for himself and he doesn't have to open a ticket for IT and Tenancy and roll-baked access control means that all of the previous items are like an admin can find grain Who can do what who can see what etc. And there is the automation engine which I'll get more into In a few minutes. So let's talk about providers It's a key obstruction in manager queue Basically each provider is like a source of information and they are modular So first they are more modular in the sense that each provider abstracts all the logic of talking to one external system as well as Understanding its objects and it's also modular and the community has been working hard Lately at separating the different providers to different repositories so that each provider can have their own Release and like development life cycle and we actually have people from the different communities working on their provider So we have guys from over to working on the overall provider. We have guys from Ansible working on the Ansible provider, etc So providers are divided to categories. This is not a complete list by the way I just got tired of clicking around in my development environment, but you have your cloud providers You have your infrastructure providers You have your containers providers Storage Middleware providers Network providers and Configuration you can see Ansible tower is here in this category. So a few nice pictures This is the container provider dashboard So it's a specific provider and what you can see here are different objects that come from this cluster These are like container or Kubernetes related entities. You can see Aggregated node utilization as well as per node Take a look that here. There's a very busy node and another interesting thing like example is that you can see pod creation and deletion trends because Kubernetes is very good at figuring out if a container crashed somehow and starting it on a different node But if if that happens a lot suddenly you might want to know about it This is the topology view which is kind of the best place to get birds I view on all of the objects in your system So you can see here remember the utilities like busy node. You can see this is our provider This is a node which is running one pod which is running one container and this is another node Which is the super busy node This is a summer view of the provider We have one of these screens for each object in the system So you can see here felt and attributes as well as tattoo says well inventory refresh last happened as well as Relationships to other objects that this provider has So let's talk a little about Ansible So I want to talk a lot about Ansible because I think this is like the ninth or tenths Talk about Ansible I'll just mention a few items that we need for to understand the demo So you have your inventory in Ansible which are the objects lists of different objects that you are going to operate on and You have your playbooks which say what you are going to do There is a special kind of playbooks which are called roles They are kind of a subset and these are playbooks that also include additional resources for example if you have an HTTP D-roll the As opposed to the playbook the role might include also configuration templates that you might need so all the files and resources that you need modules are the building blocks and Like you have a user's module to create users you have service modules to control services, etc and Ansible is idea important. She's the mathematical quality of having the same result whether you operate a function once or twice Like the function absolute value or if talking about Our what's interesting to us is deploying a server Once you finish deploying it you just want it to be up You don't care if it was already there and you had to do nothing or if you had to install it Let's take a look at a real-life example. It was mentioned in the previous talk It's a project called OpenShift Ansible. It's developed under the OpenShift umbrella and it's used by all kind of people to deploy OpenShift clusters So this is a Regular inventory for deploying a two-node cluster So first we mentioned the objects that we are going to use we have master's nodes NFS and This is our NFS Actual server. So let's see what you do is such an NFS. This is a playbook So you can see this is the playbook that is going to turn the resource from the last slide to an actual NFS server It's going to install NFS utils. It's going to configure a file Register the result of that in NFS config and then if and only if the configuration change It's going to restart to actually restart the server. So the configuration will Go into effect. Does anyone see a problem with that by the way? Can anyone guess? So It's only starting the server. It's only restarting the server if the configuration change. What if we had a server that had Already had NFS with this configuration, but it wasn't running So it won't be restarted So moving to manage a Q Ansible module So the state we are currently now is let's say we have manage a Q instant is installed somewhere We can either have that inside our OpenShift cluster that we manage or outside We can have it on a container. We can have it as a VM appliance and we just We just provisioned a new OpenShift cluster Let's say we did that using OpenShift Ansible and now we would like to connect our new cluster to the manager queue environment And configure it So this is where the manager queue Ansible modules come into play They work on manager queue using its REST API Using a Python based client, which is a natural choice for Ansible which is Python There are four existing modules. We have manager queue provider and user. These provide cloud operations on these entities and Also with reporting if there were changes or there was no changes the provider module Will after adding a new provider will check the authentication and if everything is valid It will also schedule a refresh for all the objects from the provider There is the policy assignment, which is we talked Previously about policies, so it allows you to connect the policies that you want on a specific target We have the manager queue custom attribute, which are simply key value pairs But they're quite powerful and we would see why So all of the modules support SSL verification as well as configuring certain attributes of the manager queue Instance you are working against in environment variables So we will see a first demonstration of creating a new user We will then add a provider to the environment We will add some custom attributes and we will assign a policy. Let's see how I am on time Okay, so this is manager queue You can see we have no providers and the user is an administrator So the first thing we are going to do is we are going to add a user. Let's take a look at the playbook So this is the name of the module manager queue user. The names are very important in Ansible They are descriptive and we will be able to see the execution afterwards You can see the user has some properties and it belongs to a group and these are the manager queue Attributes which will return in all of the playbooks. So let's run using the Ansible playbook command okay, so successfully created the user and All the attributes Now next we will use this user to log into the system with a very strong password Okay, so This is our new user. We just created now we're going to go into compute containers and providers You can see we have no providers in the system And the next thing we would like to do is to add the new cluster that we installed to manage a queue So second playbook using the manager queue provider module So you can see different attributes of the provider What's interesting to note here is that we have two endpoints for this provider That means that manager queue has to talk to two different Entities one is the which they happen to be the same one because we have a route inside the cluster But one is used to get inventory data from the kubernetes and OpenShift master and for events And the other one is the popular that we talked about that we collect metrics from Let's run this playbook This is going to take a little longer because as we said there's actually an authentication check Against the entity that we are adding which is the OpenShift cluster It's optional, but if it succeeds We can see that there was a successful addition both authentications are valid and so an inventory refresh was scheduled So now we are going to go into manager queue and we'll be able already to see all the objects of Our environment So this is our new provider. So you can see it has all the attributes that we mentioned as well as the different endpoints and Already has a complete inventory of all the objects currently running inside the cluster Next we will take a look at the topology view for this provider which kind of springs into view So we can see all the different objects in the system. Okay next You can see that there are currently no Custom attributes or you couldn't see but trust me so this is a simple playbook it's going to add those key value pairs and This one similarly to the next one it has the entity on which we are going to operate which is provider identified by name The Ansible state name of the module. This is the inventory. So what's interesting is Because we said you can define policies in free language Here for example, we have an expected number of nodes So if we wish we can define a policy and this is something that the ops Needed we can define a policy to the checks if the expected number of node is equal to the observed number of nodes That we actually get in the inventory By the way, there are sections for the custom attributes and the default one is meta data They were added. We have two different lists for added and for updated. So all of these Attributes were added and now we will be able to see them In the system. Yeah, and this is our expected number of nodes. Okay now to the To the last module. I think it's a little bit more interesting So the last model is policy assignment. What we are going to do now is we are going to attach a compliance policy to our provider and The policy is called our open scar policy, which is a policy that we provide out of the box for container providers We'll see it In a second. So again, we have our target entity and we have the name of the Policy that we want to attach so attaching this to this provider So of course we have access to all of these things through the API. We're just showing the UI because it's nice So currently the policy is not assigned again running Ansible playbook So this is our last playbook. The policy was assigned to our provider So the content of this policy is what we mentioned before it's find out if a new image has suddenly appeared inside the cluster and then smart state and analyze it and If it has like check Cve vulnerabilities against the latest content from the relevant source And if it has any high severity vulnerabilities It is it will both be marked as non-compliant in the system and we will also Annotate OpenShift in a way that OpenShift will know not to I to run containers based on that image anymore So these are the different parts of that policy and there is also the Explorer the Control here it is the control Explorer. We can you can really see what are the definitions of policies, okay So this is the end of this demo so moving on to manage a queue automation So obviously it provides the ability to automate recurring tasks the base design pattern that autumn the automate works in our state machines Where you define different methods which are states and you define state transitions between them if something goes Well, go to this state if something goes bad go to that state And it's quite powerful. We've did kind of complex things using This for example, we have a feature of installing a complete cluster an OpenShift cluster using Ansible OpenShift Ansible with the automate So it's currently in Ruby you define your different states in Ruby but the Ansible team is working on having Ansible Playbooks here is like first-class citizens so you can just write Ansible if you want from the automation We will keep the Ruby It will add to it and you don't have access to the complete data model But only to selected objects that we want to expose And it's fully supported in the REST API and you can also run it through the UI which you'll see in the next demo You can import those from Git repositories which allows you to use community prepared automates we call them automate domains So next demo Okay, so We've actually been using Ansible Inside the automate for quite some time although we do not yet have the Ansible syntax Supported, so you'll see how to like run custom Ansible code using Ruby. So here we have a rev provider This is in compute infrastructure providers and we are going to select VMs these are all the VMs that are running inside the rev provider Let's go to a specific VM. This is this gives us access to cockpit on that VM and it's not installed currently So this is going to be like a very simple example that will show how we are going to Provision cockpit on that VM so you can see the Automate is namespace. We have our example general install cockpit and what you see here We basically it's a very simplified example The actual code that we are using to deploy cluster is a little bit more complex But what you can see we are simply writing an Ansible playbook dumping it into a file and running it so Install cockpit using they are module start the service using a service module and last open firewall if necessary Dumping it into a file. Obviously when we did features using Yeah, right. Yeah, we will see that in in a few minutes We just didn't have this provider before so this is what we used to do But we will also see the we will see the tower provider. Thanks for the comment It is kind of funny what we are doing here. So dumping into a file This is the explorer that allows us to run Specific the simulation. So he allows us to run specific methods from the automate We have parameters and we run it so you can see the namespaced name of our method installing cockpit so Cool install cockpit start an enable and open the firewall So that runs successfully and again, this is an example of running Ansible because this is kind of an Ansible tech from the automation engine, but Most of the automation isn't are it currently written in Ansible But it will Be great to have that integration soon So we are going to the same VM and we are going to open the console again and By the way, if you haven't had a chance to look at the cockpit project, I Suggest you take a look at it. It's really easy to install and They made some really interesting like tech choices So this is basically a user interface for Linux systems Here you can see metrics, but you can also see that it's not collecting metrics while no one is connected to the UI Only when you connect it start connecting collecting the metrics to an agent You can see the services You can see containers It's kind of if you need to manage servers kind of the first thing that I do before installing something more complex is usually cockpit cool, so Let's talk about the Ansible tower provider. So Ansible tower is not open source It's like the enterprise Solution that Ansible had around Ansible It has UIs. We will talk a little bit about it in the next slide, but reddit is committed and Ansible are committed to Creating an open open source project around the existing code base You can read more about it here at ansible.com slash open tower So in the next release of manage IQ, it's going to be installed included downstream and upstream we are not going to install it obviously but We will give instructions on how to install it if someone wants and in case it's installed We'll know how to work with it And it's a configuration manager as mentioned So what you can do the first thing that you can do is you can just use playbooks inside manage IQ from Your Ansible tower provider Another thing that you get from this provider is you get a two-way sync which means that you can From one side you can use hosts that come from manage IQ inside your Ansible tower to work on them as your inventory and You can also use job templates job templates are an abstraction above Roles that allow you to get very complex things done using just the minimum set of Parameters that you really need to configure. What are we going to do next? So First we are working on New modules we are adding the modules that we need but we'd love for people if someone finds it useful We'd love for people to use it and maybe send some codes and some patches So currently we are working on policy modules, which allow not only to attach policies, but also to create them We would have some more advanced example playbooks. I just saw up here this week and For packaging we are we are working towards having this in the Ansible extras which will mean it will come with every installation and This is the repository with it is hosted It's hosted in Daniel Korn's repository. This is Daniel. He originally wrote these modules. He loves snowboarding contributing to open source and this is his beloved football team so To sum it all up what we saw today is how to use manage IQ to Manage like all of your environment We saw Ansible used as a simple and powerful tool and we saw the different flavors for combining manage IQ and Ansible together Both using the modules to configure manage IQ from it manage IQ using the Running Ansible and using the provider integration. So these are the slides of this talk This is a blog post I recently wrote about the Policy bit the open escop stuff if you want to check it out and That's it questions Go ahead. Yeah. Yeah, we actually did that in for production features But in a little bit more complex Manner, I'm sorry repeat the question. So yes, won't we just dumping? Like Yamal into file. So yeah, that's what we basically did We did some features using that the reason simply was that we wanted to Leverage the automation solution that we had Inside manage IQ and that we wanted to we have the great open-shift Ansible project. Obviously, we didn't want to rewrite code It's a huge project. It supports like a vast majority of Configuration we didn't want to do something new. So we simply ran Ansible from the automation Any other questions about manage IQ in general? Might be able to answer those Yes Yeah, manage IQ is the upstream the downstream is called cloud forms and they are very much in sync Any other questions? Cool. So thank you very much for listening It's a little short Right. Oh, I'm sorry. I needed a sign for this I should have added it in my in my slides that way Because Yeah, I put it inside Thank you. Thank you. Yeah, I'm coming Actually the best resource I can recommend on The Open shift Ansible is amazing You Testing, yeah, is it testing? Yeah, okay I should have started shop to Yeah Hey Josh, I'm at even one to yeah, like to My gosh, sorry, okay, this is about turning the