 It's okay. I can kick. Thank you. Thank you. Okay. Hello, everyone. My name is Moon Lu. I'm currently a software engineer at Gaptech. So today we'll share with you about configuration management with IAC, which is infrastructure as a code. So here's the agenda for today. We'll cover what is infrastructure as code and what is configuration management, followed by why and how. For the how part, we'll compare the popular tools like Ansible, Source Type, Papi, and Chef. Then we'll dive into Ansible with an example to show you how the tools can be used. And also we are going to see some Ansible code. Lastly, it's in the cloud world. There are many managed services. So where does configuration management tool stands? We'll cover that. Okay. So before we start, can I quickly check how many of you have ever used or created a virtual machine before? Can I raise your hand? You have to do it. Nice. I saw many, many hands. Then any of you has ever, do you have experience with configuration management tools like Ansible, Source Type? Okay. I saw some of your hands. That's good. Okay. So for infrastructure, it's usually referring to the hardware, software, and networking that's required to deliver IT services. For our talk today, we are mainly focused on the environment that's using virtual machines because for configuration management, it's still mainly to manage the software and the service on the virtual machines or the bare metal servers. Then IAC is actually the process is to provision and manage infrastructure to code instead of a venue process. So what does provision and manage of infrastructure mean? What does configuration refers to? Then let's take a look at typical web application. So from here, you can see there are a number of different types of servers like the web server, app server, and DB server. In order to get your application up and running on this environment, you will require the provisioning, configuration, and deployment process. So in terms of provisioning, it's to create your server, your networking. For server provisioning, you create a virtual machine with the operating system installed on the VM. So at this point, your web server, web server, DB server, they are all still the same because they only have OS installed at the moment. The difference comes when you are doing the configuration stage, where for your web server, you are going to install NGX. Like your application server, you're probably going to require Node.js or Java or Ruby, depends on what your application is written in. Then for your database, you are required to install your database engine and do the configuration setup. There are also some common services, like NTP service, which is to help you to synchronize the time on the server so that all the servers have the same time. There will not be a different timing. Then for deployment, it's mainly to get your application code running on the app servers. So hopefully now you get an idea of what's creating and managing the information about and what configuration of the servers is referred to. So before I see things are done manually, take the configuration, for example. If we want to configure the NTP service, system will usually need to connect to the server remotely one by one. Then you have to modify the NTP configuration file to start the NTP service. So it sounds a trivial task, but imagine you have many, many servers to do and also there are some applications that actually require a lot of configuration. So doing all this in a manual way is just a lot of steps and also error prone. That's why we need to use IAC to automate this process. Then for the configuration management, there were many focusing on the configuration of software. So these are the popular infrastructure IAC tools. You can see that since Papi is so stack, answerable cloud formation, Docker, Kubernetes, blah, blah, blah. So if I need to do all these tools, you may be confused what are these tools for. So here's how we can categorize the tools. For provisioning tools like teleform and cloud formation, it's a menu to help you create cloud resource, such as networking, compute instance storage in the cloud. Then configuration management tools, we have Chef Papi and server source stack to help you manage the required software on servers like virtual machines. Then we also have server templing tools. For Docker, it's to create container images, PEC and the vagrant is more on the virtual machine side of image. Usually you have your dependence of the application bundled into this image. Last one, we have orchestration tools. They help you to deploy your application. For example, Kubernetes, you can deploy your code into the container image. Then you deploy the containers. Kubernetes also help you to load out updates, do some monitoring, auto scaling, and auto healing in case when container is done, you auto automatically start another one. So come to the next question. Why do we want to use configuration management tools? Well, first of all, you help you to automate the menu process. Imagine you have to configure 10 servers or 100 servers that require the same setup. Doing this in a manual way is just repetitive work and also boring and also helpful. So with the configuration code that's ready to use, you are able to deploy and the setup new server in a much faster, more reliable way. It's also more scalable because you can just do everything in a very fast and persistent way. So the versioning will also help to track what change has been made and when. You also enforce certain coding conventions, these tools, compared with other hot scripts. So the code will be quite easy to understand and limit them. Lastly, the logic is also preserved in code because configuration management tools usually are quite readable. So they can serve as documentation for how your server should be configured. So another benefit of these tools is that they can help reduce configuration drift. Configuration drift is over the time when you apply one or more changes on the server. So the server can become slightly different, especially when you are doing things manually. So these will lead to self-taught configuration bugs. That's where we have to diagnose and all reproduce. So these tools can help reduce the drift because everything is managed by code. Then no server should be different if they are only configured by code. So now let's take a look at the configuration management tools. Answers also set puppy and chef. These are the popular tools currently. They are all open source and they have been around for more than 10 years. So they are all quite mature and provide rich features. Also, they are still actively being maintained and developed. In terms of the difference, look at the configuration language and management for answer-by-source stack. The tools themselves are written in Python and the configuration language is actually just YAML. So it's quite easy to understand and to get started. Then for puppy and chef, they are written in Ruby and to get started with the configuration. You actually need to learn some of... You have to learn their own domain-specific language that will make it a little... It's more difficult to get started. In terms of setup, Ansible will be very, very easy because it has client-only architecture. For example, I can use my laptop as an Ansible management node. I just need to do blue install Ansible. Then I can get started and start to write the configuration code. For source stack puppy and chef, they are using the server and client architecture. That means you have to configure both the server and the client. So more stuff to configure and set up. You also need to ensure the communication between the server and the client are there to set up properly. Okay, so let's take a further look at architecture. For the server, client architecture we have source puppy and chef. Then Ansible is the client-only. So in terms of setup and management, we talk about this now. So how do they work? For the server-client architecture, normally the server has the latest version of the configuration code. Then the clients will frequently call the server and get updated version of the code on themself. Then they will apply the changes by themselves. Then for client-only Ansible, usually your configuration code will be only available on the management node. When you want to apply the changes, the changes are pushed to the server, to the clients and the commands are executed remotely on these agents. Then we have the push and the pull, which are the different models, different ways to deploy the configuration changes. As you can see, they're actually quite closely related with the architecture. For example, like Ansible, because there's no server, so the only way will be push. The configuration code is on the management node, then you push the changes over. Then for... So Stack is also a little bit special. It's because for so Stack, they allow the sort master to push the hook scripts to their clients. That's why it can also do push, but it's still mainly a pull model, so all the agents actually try to fetch the code from the master from time to time, and then they're going to apply the changes on the clients. So for the push style, it's simple to manage and set up, but you may not scale very well when there are thousands past servers, because they all rely on one single master server, and it's doing the connection concurrently for all the agents together. And also in a distributed environment, when the network is not very stable, ever handling, there's no retry mechanism, can be difficult. For the pull model, so you will require more set up and management, but you will scale well because the agents will appear to contact the master and get the change. In case of failures, you automatically retry and apply the changes. But the additional set up could be just more effort, and also you have to make sure that the agents, the demons on your client is running all the time, else your code could be updated. Okay, so which tool to use? Well, there are pros and cons in each of the tools that I've already discussed just now. So it depends on what's the scenario that you are using. So for example, if you are using IoT device, there could be many, many devices, and then the network connection could be unstable between the devices. So the push model might not be that good. So you probably want to choose pull model instead. Then choose and switch wisely. For example, you are already using tools like puppy. Then you probably don't need to change to another one just because the tool may sound more modern or cool. The last one is just to try out, get the feeling of the tools. So no matter which tools that you choose, just follow these best practices. We should treat our infrared code the same way as our verification code. So we want to do version control. We want to make small changes and do code review. And don't put everything into one single file, big file. So break them down into small and reusable modules. Then in terms of testing, infrared code is usually harder to test compared with application code because it's really difficult to do unit tests for infrared code. So the least will be test your infrared code in lower environment before they go to production. The last one is change of behavior. So this is actually quite important, especially if your team is currently used to the way of doing things manually. So you need everyone in the team to get into this idea of using this tool. Say if anyone is not decided to go to do things manually and make change of the whole thing on the server, then the next time when you deploy your configuration code, the code will fail. And slowly people won't trust your code. And then people will revert back to the old way of doing things manually. Okay, now let's come to Ansible. So we'll show you a demo in a screen recording later to configure NGX to host a static website. So this is the setup I have. I have three EC2 servers. So the setup is in AWS. I have three EC2 which are virtual machines in AWS. And they are all behind load balancer. This load balancer is probably facing and it has this DNS name. So once everything is configured properly, we can go to the browser and hit this URL. So this Ansible management node is actually my laptop. So you can reach to the EC2 or SSH. And these are the two false Ansible codes. The first one is the host inventory. We basically tell Ansible how to connect to the host. What are the hosts that I want to connect to? And then the other one is the playbook. So this playbook will tell Ansible to do a number of stuff to how to configure the servers. Before the demo, let's quickly see the manual way first. If I'm to do this manually on my laptop, I run it to SSH into each server by one and then install, for example, the NGX RPM. Then also create some required directory on the server to host my website false. Then also need to make additional configuration changes on the NGX configure file. We start the service and et cetera, et cetera. So doing one server manually is okay. Three is repetitive. Imagine I need to do more or imagine that I need to do this again after some time. If you one week or two week, I won't be able to remember all the details. So with Ansible, we are able to do all this configuration with a few commands. So just now was to show you that the EC2, the virtual machine is up and running. But when I try to hit the URL, it doesn't give anything cause the server is still just, there's nothing on the server yet. So now I come to the code. It's to run the Ansible command with this playbook called web servers. And we will do some basic stuff like install NGX then create the directory. So notice now this is changed. But if I run the same command again, it will show okay instead of change because the change is already there, there's no need to apply the change again. So this is a second playbook which is to copy the static websites over to the remote server. So now I come back to this URL in the browser. You can see the website is up and running. And come back to the, this is the target group in AWS. So you can see the status change to health, you know. Okay, you cannot see. Anyway, I need to go to the next page. Okay, so here's the directory layout for the Ansible code in the demo just now. So usually we have Ansible folder inside it will have all the Ansible code that you require. There are playbooks like web servers to YAML, sync to YAML. That's where you define the tasks that you want to do. Then some important directory will be the inventory file. That's where we put the host IP or the host DNS name inside. Then another one is the files. You can see there's a crony.com. This crony service is just another, it's just an NTP service, but that's for right health, 7 and 8. Then there is also templates. So the difference between files and templates, this one is the static configuration file while the templates one can interpret some variables inside. So make this one dynamic. We'll cover more later. So just now in a demo, I just need to go to the Ansible directory and run these commands. So Ansible playbook, this dash inventory is to tell Ansible where the inventory file is. So it's on the inventory host file. Then the last one will be the playbook. So this one will be a shorter form of the inventory. It's that share instead of the inventory. So more on the inventory file. This is how we can group the servers together. So say now we have a web servers. I put the DNS name of these web servers, but they can all just be IP address. So it can be a 32.2.1.4, et cetera. This is tell Ansible how to reach out to these servers. Then below is the variables that we define for this specific group of servers. You can see there's a domain, which is the DNS name for the load balancer that we saw just now. You'll see this value can be accessed in playbook level and template later. Then we also have this Ansible user and Ansible SSH private key file variables. So these variables are defined over there so that I can just run the command like this without passing through all these values inside here. So the dash u is to specify which SSH user I'm using, and this is where the SSH private key locates. So a little bit further is that you can also, this is for web servers. You can also define the database servers with another set of IP and their own variables. So now we come to the playbook. The playbook is Ansible to tell what to do. Basically it will start with a name. It's a description of what this playbook is about. And then the host, here we put a web server according to the inventory file here. And then become yes. This is normally required because during SSH usually you cannot use root user SSH to the remote server. But when you run the task, you actually need the root privilege. That's why you have to become yes. Then is the task. So for tasks, here have two tasks. Each of them will have a name. This name is also optional, but good to have to tell what this is about, what you want to do. Inside each task you have the modules. For example, here the module will be the yam module and then the second one will be file module. Whatever that's behind the module are the arguments that you pass in for this module. For this for the first one, it's actually tell Ansible to do the yam install, which is to install the package NGX and then the state specify is the latest version. Then for this file module, it's actually to create the required directories inside this directory. So here you can see there is a domain that's enclosed by the double curly braces. There are the value that's defined in the inventory file here. So you can access this value directly inside here. Then this viz item is like the for loop in Ansible. Ansible will loop through all this stuff and replace the item value with the one view. So there are some common tasks like copy the configuration file from your local to the remote server. This is one example of how we set up the NTP configuration, copy the configuration file over. So the source is crony.conf. And if you look at the folder directory, you'll be under the false directory. Then there's a crony.conf. What's inside this crony.conf is just these lines which is actually the Singapore public NTP servers and plus some more lines. What this task will do is you'll copy this file from my laptop to the remote server and the destination will be the etc crony.conf. And also you'll have to set up the owner and permission for this file. So just now the copy module is only able to copy false static configuration files. If you want to make the configuration files dynamic, then we will need to use this template module. And also Ansible is using Jinja True Templating. So how it looks like is that the variables are enclosed with these double clip races. And this domain variable is defined in your host file, in my true file that you saw just now. So the file directory will be like this. There is a template for the download template. Also need to know that normally for the Jinja template you have to have this .j2 false fixed. So this is the last bit of Ansible code that we're going to see, which is the handlers. If you look at the example here for handlers is to copy the... is after we copy the NGX configuration file then there is a notified restart NGX which is defined here. So what it does is we will restart the NGX service. Only when there are changes to the NGX configuration file. So if there's no change then the NGX service will not be restarted. Okay, so in the previous demo the task handlers are all put inside this web server demo display book. So now we want to organize this differently so we can create something called roles. For example, you can have a common role then you can put with this common role we can set up the NTP services. So you can put the task related to NTP into this task folder on the main demo then you can restart your NTP service that handler can go inside handler folder. So in this way your playbooks now will look like this. You have hosts then you define the roles is common and web tier specific stuff. In this way you can reuse this common role for example for DB server so other type of service. Okay, then for the multi environment directly out it's beside the inventory folder you can create more folders like production folder, staging folder that's where you can put your host file in and they probably will have different variables defined as well. Then the playbook will inside here and you also have the roles. So in terms of the master playbook you have become like this you can import the web servers playbook and DB server playbook and inside the web servers you have common role and web tier roles. With this structure you are able to run the playbook in this way. So this one will deploy everything into production and this one will deploy production but we just limit to web servers the configuration change will not go to DB servers. This is also the same. So for configuration code usually we want to then have the same set of code but it's just between environments it's just the variables that changes. So a bit more in front there is Ansible Tower this is the enterprise version of Ansible it provides features like graphic user interface dashboard and also the role-based access control job scheduling, etc. But it's enterprise version so I have to pay for this license. Then there is also Ansible Galaxy so it is a large public report of Ansible roles and normally with the remiss there will have details about how you can use the roles but I would say because each environment setup can be different so you probably just use them as a reference instead of just copy and blindly apply to your environment because you may not need everything. Come to the last part for the cloud world previously for the application deployed on the virtual machine when you do provision and configuration and deployment so now in the cloud world together with the usage of containers you do not need to configure things like NTP service, NGX, Node.js, etc. because they are already bundled into your Docker images and in terms of deployed Docker images to containers you probably don't want to create the EC2 instance and start to set up your own Kubernetes cluster or manage your own Docker nodes you are more likely to use the managed service like such as ECS or EKS then same for the database part since there are managed services like RDS you don't need to create the virtual machine and then install the DB Engine yourself anymore hence there are much less need for configuration management tools so is configuration management going to be that? No although there will be required less as the need will decrease but not all the software are suitable to run in containers or running in servers and also software as a service or infrastructure as a service is just someone else's service so someone still needs to manage the service okay, summary what is EC and the configuration management tools EC is to manage and provision in virtual code then configuration management to install set up managed software on the servers so why it is important is faster, more reliable and more scalable to use them also help to reduce the configuration drift and how? we do a comparison between the popular tools and also we look at example of Ansible with inventory, paper and work such a so in the car world configuration management won't be that that's all for my talk there is puppet mode which is separate from its side project of puppet which is intended to run with like a puppet diesel on some servers via SSH so it's actually the same as the one I sent so if you like puppet diesel or you like to work with a puppet ecosystem then the same tools for the SSH Ansible, Django, push model model as well in some way or way, but still possible I understand customers orchestration for this whole stack actually was invented exactly as orchestration was first and then after some success they decided to put configuration management there as well because first they can and Ansible is less visible and Soul Stack is visible so if you look into Soul Stack you can understand that it was invented like that's okay we need this let's do this shortcut to do this at least you can use virtual machines in two ways first one, which most of us I guess is we are using them as visual servers to help the server with some life cycle and usually they will stay there until we don't need it anymore or we need to upgrade it using virtual machines in I guess it was original intention of virtual machines you can create it you can put something there and if you need to do any change it depends on how big the change is right in the middle