 So let's get started. I'm excited to talk about things which I am already excited. So it's my second time at Force Asia and this time I'm going to talk about Terraform and the main idea is I want to bring some points to people who already use Terraform so that you maybe write it a little bit better. Okay, so first of all a couple things about myself. My name is Anton and I live in Norway. I'm originally from Ukraine. Norway is pretty cold compared to Singapore, so we have several meters of snow right now and over the last few years I've been actively involved in Terraform, HashiCorp, user-group activities, AWS user-group, DevOps days, OSLA organizing and so on. I enjoy going around different countries and speaking in different meet-ups and being involved with Terraform and AWS communities. So if you have questions about things which I'm going to present, don't hesitate to contact me using any of these channels. I can just say a couple words about projects which listed here. Terraform Best Practices is something what I started summarizing about half a year ago and I try to keep the top date with latest things which I find in the community and I hope you find it interesting. Also, you can always reach me on Twitter and GitHub to see what I'm up to. Before I move to the next slide, let me ask like, can you right-hand if you use Terraform on a daily basis? Like, anyone use Terraform on a daily basis? Nobody. Excellent, because this is a really advanced topic. Okay, so if you have never used Terraform, then I'm afraid that you will be a little bit disappointed because there are a lot of code, a lot of Terraform and a lot of a little bit complicated things. So as I understand, no one is using Terraform, which is pretty unusual, but nevertheless, thanks for coming. So Terraform AWS Modules, the project which I started or I actually joined it, but then I adopted it and became main contributor to this from about 2015, 16. This is a project where a community contributes different Terraform AWS Modules in order to build reusable blocks like VPC, security groups, auto-scaling group and so on. And by the way, who is familiar with AWS? Can you raise hands? So, okay, good, good, good. So yeah, it's good because there are a lot of words which I don't really want to explain, like what is RDS, what is security group and so on, because otherwise we will never finish. Okay, so this project itself is very useful and was downloaded more than two million times so far. If you need to build things like VPC, it's unlikely that your case is unique, so don't hesitate to look into those building blocks. So Terraform was started in 2014 as a tool which allows to just write, plan and manage infrastructure as code. As code means that this infrastructure will be possible to recreate in different configurations in, let's say, different accounts, different regions and so on. So this is actually making DevOps movement possible. Terraform configuration file usually looks like this. In this example, we have block AWS region where we specify variable. We have a couple of resources which we want to create and after several commands like Terraform init, we download some dependencies like provider of specified version and then we run Terraform apply and it will actually create resources which we want. In this case, we are creating S3 bucket and we can see the last line is that this bucket has been successfully created. You may think why Terraform, why not AWS Cloud Promotion or if you are on Google, why not Google Cloud deployment manager or if you are even using Azure, Azure resource manager. So and the answer is that Terraform has more than 100 other providers in addition to this, which means that it is possible to manage very large amount of different types. It can be your G Suite access, for example, your Dropbox files and access, New Relic metrics, Data.Permissions, different type of dashboards, whatever else you can figure out. There are enormous amount of providers out there created by the community. So this was basic of Terraform. Now let's get into actually looking what Terraform is and how we actually use it. So let's start with simple example. In this example, we are creating just a VPC, we specify some properties and that's it. That's all what this example is doing. Then we will figure out that our project will grow. We'll start adding new resources, integrate with different data sources, apply for different regions and so on. You may think, why? Because we also need to integrate with other providers and in this example, we are adding AWS Internet Gateway resource because we want to go to the Internet. And we start to add couple subnets and by the time when we actually going to add other resources like not gateways, routes, routing tables, our network stack for AWS will look much more than just 40 lines. And main TF file will eventually grow up to 10, 20 kilobytes and 300 lines of code or more. And we are talking just about network stack, which is basic of everything. We are not even talking about deploying other sophisticated resources like autoscaling group or load balancers which are necessary for our application. So as we can see the code size is increasing constantly and the dependencies between resources are getting worse and worse and more complicated. And naturally a solution which has been designed for Terraform is called Terraform Modules. So Terraform Modules is self-contained package of Terraform configuration which is managed as a group. There are a few types of Terraform Modules. The first one, which is resource module. It is a great example of something that can be open source and it has no logic building inside of it other than just creating resource in very flexible configuration. This is an example of security group module which is just fetched from specified endpoints. In this case it is Terraform registry and we specify which version of this module we want and we specify different arguments which want to pass into this module. So the reason why resource module is necessary is that it is possible to aversion it as a whole while individual resources cannot be managed, cannot be aversion. Let's look into a real example of security group module. This is piece of code of security group module which may seem very complicated but in fact all what it does is just creation of security group. But in enormous amount of configuration. For example IPv4, IPv6, self-references, computed values and so on. There are about five or six hundred lines in order to create just security groups. You may think that you will not need this kind of situation like why to make it so complicated. And the answer is that if you are working with Terraform you may know that it's not possible to specify empty values in some arguments. So that's why you need to figure out all possible permutation which will work for your case. In this case for security group there are a lot of situations when you cannot specify on one rule both IPv4 and IPv6. So you have to make two resources different. Another type of module is called infrastructure module which is essentially a composition of resource modules. It's a good place for your company to introduce different naming standards, tag in or enforce some things which you don't want to be customized by developers or by users of your modules inside of your organization. For example you want to make a module which is available across the whole organization where people are not able to turn off encryption. So it doesn't matter if they want it or not but there is no way for them to turn it off. That's why you have to make an infrastructure module where encryption is always true. It is also a good way to fulfill missing bits of Terraform like you may use different tools to generate parts of your code using JSON net or cookie cutter or other type of preprocessing. Infrastructure module's invocation looks pretty similar. You specify source of the module and you specify a bunch of different arguments which you want to pass to the module. Inside of the module itself you are invoking different resource modules. That's the main difference between the resource module and infrastructure module. So as a small summary there are two types of modules which I want to keep as separated as possible while a lot of people don't feel that this separation is necessary. Resource modules are those which are maintained on Terraform AWS modules, GitHub organization and infrastructure module is something that usually people have inside of their organizations for their specific needs. Was it clear so far? Good, because we are getting deeper and deeper. Right, so let's look into how to write modules. And then we'll look at how to call them and some tips on how to actually use Terraform. So the first tip which I want to give about writing resource module is to not write them. Really go to Terraform registry which is an official place where you can download and see source code of about 650 AWS modules for example. And there are lots of other modules for other providers like Google, Azure and so on. It's maintained or community can upload their stuff there and some of those modules are used by many, many people. Some of them are pretty bad quality so you will have to find something. But anyway it's usually a very good place to look into and start. Don't go to documentation and start writing it because you think that your case is so unique, most likely not. If you decided to write Terraform resource module, try to always hide implementation details from your users. You don't have to be expert in specific implementation of RDS for example. But they just want to specify that they want to have let's say an SSQL or MySQL. And then you inside of the module actually figure out what kind of arguments each of these resource may have. For example in this code we are creating an MSSQL if it was requested or MySQL or Postgres in other cases. And as you can see line 14 says that time zone can be specified for MSSQL only while it's not possible to specify it for other resources like MySQL and Postgres. Which is one of type of details which you don't want your users to be experts in. So hide it away from them and output just first not empty database address which you have. So in this case it can be either MSSQL address or any other. Also if you host your infrastructure modules in one big repository inside of your organization and you call this module multiple times. For example if you want to create IAM users and you have a lot of other infrastructure modules in one organization or in one repository. It will have to be downloaded multiple times. So what it means a telephone will download the whole repository and it will just change directory to the place where you want it to be. So if you are making let's say hundreds of IAM users you will have to clone repository hundreds of times and use just small subset of this repository which is wasting of time and tracking. There are few solutions one is to use a project like MBT where you can build just small artifact for each version module and use that one. Couple things to avoid inside modules. So this is something what I want you to look into and tell what you think is wrong here other than hardcoded value of region. So I can give you a hint. The problem is not on line 1, 2, 3 or 5. So which line has problem? Yes exactly and what kind of problem is that? I'm talking about what's wrong with this block in general and why it is wrong for the module. Exactly so the answer is if we have different users who want to configure access to AWS provider differently like first guy wants to use default configuration in order to access AWS and another guy wants to use shared provincial file they will not be able to use your block because you expect that they will use assume role. In fact no they will not use it. Some of them will use environment variables which are set in the machine and some will use shared provincial files. So the solution for that one is to not put provider block at all in the module. And second one is provisioner which is very often bad idea. In this example we are calling local exact provisioner after VPC is created and this block itself is bad idea. It's actually bad even for AWS instance. So let me explain why it is bad for instance. First of all let me explain why you think or you may think this is good idea. Because AWS instance will be created and then provisioner local exact will be executed on your machine so you will call Ansible playbook. You will connect to this newly created instance and you will do whatever is necessary. There will be situation quite soon when this block provisioner local exact is not possible to use on launch configuration. So this means that if you start using AWS instance and you put a lot of provisioners there, you cannot use them on launch configuration because the triggering point for launch configuration is not creation of launch configuration itself. It's creation of resources which during auto scaling activity. So in AWS user data has been introduced. In other providers there is some sort of flavor of cloud in it. This code shows that this user data will be called on the instance only where instance is started from the instance itself. So it means that operator who actually run Terraform apply it can be CI or it can be individual users. They don't have to have physical access and always be online in order to execute this. It means that code is already separated from me triggering something and going away from actually running Ansible playbook. If you need to have execution of provisioner, let's say when something is done, you can assign this provisioner to new resource. New resource is special type of resource in Terraform which is actually just new. It's not real resource. It's not physical resource. And this code will be executed AWS CLI when VPC resources create. This pattern is very common if you need to connect some... If you need to actually call some CLI to let's say add mission functionality which is not implemented in Terraform. There are few trades of good Terraform modules the way I see them. First of all it's very important that Terraform modules are actually documented and contain a lot of examples which people can execute. So if module is also feature rich which means that it supports not just your specific edge case or your specific problem but also wide range of similar situations it's much better. It means that people will most likely use it. Also don't try to put defaults which are specific for your particular setup. So don't cargo names, numbers or anything like that. Make it obviously for people to require this. And also if your resource is about creation, let's say security group, don't try to create security group and also some IAM permission. It's very seldom when you need to combine these things together. So always make them as small, as friendly as possible. And I put test on last step here just because I don't believe the testing of resource module is actually meaningful. Because a lot of situations you will see that it works on my machine but doesn't work on your account, on your machine, on your region and so on. So I don't believe so much in testing as in actually showing how it's supposed to be used. And you can read more about this on this URL. So, small summary, we figured out that it's better to not write them because there are a lot of them in the registry. And try to avoid providers and provisioners. Most likely you are not going to need them at all. And if you try to not put them in the module, you will have your life much, much easier when it comes to the usability of these pieces. So let's look into how to call modules. First of all, as we can see that over time, the amount of resources and the amount of different moving parts is just increasing. And we start to think how to organize code and how to orchestrate different invocations of the Terraform code. There are two patterns. The first one is all in one. It's where you have a few files like mainTF, where you invoke different infrastructure or resource modules to create infrastructure you want. So the bad part there is that it obviously has an effect on everything. So the scope and the blast radius is large and you will figure out that it's getting pretty hard if you are working on a large amount of resources. So in reality, it means that you will have to wait every time when you run Terraform plan for minutes to five, ten, it depends on your size of infrastructure. So the good part here is that you actually have to declare variables in your place. And another pattern which is on another side is one in one. It's where you have much smaller blast radius. And you have to describe, separate infrastructure module or resource module as independent as possible. The good part here is that you will be able to reduce blast radius to just necessary things. As in this example, it's also important to realize that you're going to work in these places quite differently. So your VPC is not going to be changing every day or every hour while your application can change actually quite often by developers. And by separating this, you will have guarantee that you're not going to destroy something shared like VPC if you're touching application and so on. So I'd like to see hands. If you're using Terraform, how do you structure or how do you group your code? So on one hand, you have all in one. Does anyone use all in one where you have main TF file and you put everything there? Nobody. Cool. And one in one, who feel that this is like better? No. Okay, I can tell you that the most frequent answer among people who use Terraform is somewhere in between. It's very seldom when you need to stick to one of these because it's just unlikely that you will be like starting your day and thinking like, oh, today I'm going to make everything perfect from day one. No, it's not going to be the day. So another type of question which I'd like to emphasize is there are different ways and different needs, different tools to actually orchestrate invocation of this beat. One way is that if you have all in one where you have invocation of multiple modules, you may use minus target and specify each module separately. Or you can use make file to invoke it to change this invocation. Let's say create VPC first and create instances and so on. So you have to call Terraform at life in different directions. And in Terraform, some people may think that this is a good way to do. This is not a good way to do. That's not really a good way to do. Don't try it at home. While you can call Terraform from Terraform, it was not designed to do that. And I saw some people struggling and thinking that this is not a joke. So when I show this slide and some people take photos, I remember their face and then one kid will die. So don't try to do this at home. So Terraform calling from Terraform is not a good idea. Terraform world there is Terragrant. Terragrant is a tool which allows to orchestrate invocation of infrastructure modules. As in this example, this is Terragrant. Terragrant is a tool to invoke Terraform. So Terragrant configuration file looks like this. Where we specify from which source we want to get, like which module we want to get, and what kind of dependencies does this module have. In this case, we are talking about EC2 instance, which obviously require a network to be present before. And we also need to pass arguments to this module. So in essence, that's all what is Terragrant doing. So you have just one configuration file which describes which module you need to use, what it depends to, and what are different arguments. Unfortunately, FWARS does not contain dynamic values, which means that you will have to somehow provide subnet ID, which you fetch from network module and put it here hard code, which is a pretty bad idea. So I fix it using small hook. It is a very logical way to do things in Terragrant, because there are ways to execute before and after hooks on different commands. So I wrote small shell script which is just doing set operation and replace these values. Or you can use modules tf, which I will talk a little bit later if I have time. So right now we are looking at how to call modules. I can tell you that one-in-one works much better over time. So if you are hesitant about where to start or start to increase your code in one file or in one directory, and your Terraform configuration is about 10 kilobytes, then you are definitely doing something wrong. So try to split it a little bit earlier. You will see benefit and productivity more efficient. So now let's look at how to work with the code. So when we have a requirement to add new features to our infrastructure, it is usually easy. But sometimes it makes sense to create new resources or use existing resources conditionally. So for example if we want to make reusable infrastructure module where we get information about VPC ID, if it was present, if it was not present then we create it. In this case we are using data source to find information about existing VPC ID or we create the one if it was not specified. And we output just the VPC ID back to the user. Working with lists in Terraform 0.11 is not so easy because they always have index. So in this case user list of SSH public keys, user 1, 2, 3 and 4 has index, like each element has index, which means that user 1 is like 0, then 1, 2, 3. And if we want to delete user 3, it means that user 4 who is going after will have to be deleted and recreated. So for information, like for resources, like files, it's probably okay because we can delete last element and we'll recreate it. It's totally fine for things like public keys where we delete it and we create it again. While for stateful list, for example, as an example of stateful resource is AWS IAM access and secret keys, which we simply cannot allow to be recreated every time because every time we'll have new access and secret key for the user. And that's not acceptable in most cases where user whose name is started with Z will have new access and secret key every time when me, because my name starts with letter A, will leave the company, for example. It's simply not acceptable. So there are solutions like involving JSON net in order to generate templates. So this is an example of JSON net template. JSON net is language by Google which generate JSON from data structures like just plain JSON list. So this will produce JSON output and Terraform natively works with both HCL, which is HashiCore configuration language, and JSON. So this piece of code will produce valid map of modules and outputs, which will be treated by Terraform natively, and for Terraform it will be absolutely fine to understand this code. So let's look how to involve this. First, if we call JSON net and we specify which template we want, the output of produced file will look like map of maps. In this case, we are calling specific IAM module and we pass different arguments to this module. And then we'll later execute Terraform init and Terraform apply to get this infrastructure, this user's create. The benefit here is that if somebody will be removed from the original JSON input file, the new code will be generated and only this user will be removed. Nobody else will be touched. Another way is to integrate. So if we have created some resources and we want other machine or other script or even other human to be able to get the entire command which can be executed, we can output the entire command and use just reference as in this case, the application file will have ACL and then we can execute it in shell like this. If we want to execute some of these commands automatically every time when something has changed, we can just use new resource and local exec provider again. We don't need to use resource AWS web, firewall and provision or that because we have to use new resource for that. There are many different edge cases when it comes to testing and it works on my machine, it works on my account, in my project, in my setup because there are lots of different configurations. Things like AWS region can be different. We are not even talking about GovCloud or China which are always behind and have very different features available at any point of time but also publicly available regions in AWS have very different features that like EC2 classic link or IPv6 availability means that it can work in one account for one user but will not work for another one. Also, software limits is pretty often problem that certain type of instances or certain type of IP addresses are not available in my account but can be available in another account. This means that testing any kind of infrastructure in development or in test environment usually has almost no sense unless you test it in production account with production settings close to production as possible. So things to avoid in Terraform. We have probably a little bit of time so I'm a little bit hurrying but it should be a fine in time. So if you have any arguments which you think it's a good idea to specify in CLI and you will always remember all of this dash var or dash var file, the sequence of them usually it makes much more sense to put all of them into TFRs files and just include these files either automatically or using single arguments but don't try to combine all of this var, var, var file and so on because you will simply forget about them very soon. Also try to not use all of this target parallelism which gives you more granular control of what Terraform should be doing. Terraform is smart enough to understand what to do itself just by looking and analyzing your code. Usually Terraform workspaces are evil if you try to manage long-term environments like production, depth, testing and so on they always will be much better if you manage separately in different directories. So don't try to use Terraform workspace inside of count simply because there will be a lot of situations when readability of your code is much more important than saving 2 kilobytes of code. I mean we are not paying for disk storage anymore so it's much much better to write more code which is very easy to understand. Try to avoid dependencies in modules so that one module should not be using several layers of modules it will be very hard to maintain and debug so at most one to two levels is enough. So as a small summary the main issue with Terraform 0.11 right now is the ways we have to work with lists especially on big structures can be a little bit tedious so be careful with that one and try to use Terraform and try to proceed Terraform easier so that you don't have to use all these countless features which Terraform provides it's not necessary. You will have much easier life if you just write things much smaller like don't use all features of Terraform at once. I see a lot of people try to use, they read documentation and they try to use all parameters and workspaces and then code looks like spaghetti and they blame Terraform for that. So instead just don't use half of features which Terraform provides and you have very easy to read code in half year. So you may think what does it mean for the future? Why there is title? So Terraform 0.12 beta is out there. It will solve some of the problems related to lists and few other syntaxes, sugar things like for example HCL2 is going to be simplified which means that you're not going to write all these double quotes, dollar, purely brackets if you want to reference elements and few other things. So the syntax will be just simplified. That's really a good point. There will be different ways to loop. You can use for or you can use for each in order to configure different properties of resources. And finally you can use correct conditional operator while left and right parts are not executed simultaneously as it was with 0.11. Actually for me it's bad side because now I have to test my code twice which means I have to make sure that I provide values so that left part is executed then I run code and see that everything was created fine and then I have to make it so that right part is executed and run the same code twice. For me personally as Terraform modules maintainer this is part of the problem. Extended type of variables you can specify what kind of output or input you expect. In fact it can be least string, maps and so on, templated values that you can link between different resources so that modules can be now linked. And much more is on this blog post. It was written approximately half a year ago and it's still pretty relevant. As I said Terraform 0.12 beta is there but it will not fix your code. It will be backwards compatible with 0.11 and you can still use it. You can even try to use it right now because it's available for download and you can run it on your dev and test environments. And I also recommend to use whatever existing you find is much better than just start write Terraform after reading the documentation because other people have made solutions for significant amount of work rounds, problems and you will just get much more productive if you read any Terraform modules which you find on registry countless amount of utilities. I usually tweet about many of them on my Twitter account and yeah I have a lot of them. Probably all of them start on my github profile because I like to see what community is up to. In general AWS has much better support than any other provider. Approximately 80% of tools just for my rough estimate is for AWS while about 20% is for the rest in total in Google, Azure. So yeah, I have bonus. I know that I have a little time but I still have bonus which is this one. So this is, you know something like this, right? You probably saw something like this before where you have, so CloudCraft is a tool which allows to draw, visualize your AWS infrastructure in the browser. So you can connect different components, you can specify different type of resources like auto scaling, EC2 instances, load balancing and so on. And then you connect all of this yourself in the browser. You can also import your existing infrastructure to be able to connect elements and visualize. You can also automatically update when something change in your real infrastructure. And approximately a year ago I was looking like we have AWS console where we have been clicking historically for quite some time and then we have tools like CloudCraft where we visualize something and then we have Terraform where we describe everything as code. These three tools are not connected in a way that we want them to be. Like if we look by job description, this is, let's call it random guy, is using this one. Okay, random guys, they're going console, click everywhere, let's call them random guys. This is tools for Cloud Architects who are usually thinking about different types of resources, how to connect them, what kind of properties, instant size, open ports, different solution architectures and so on. And this is for DevOps engineers who are describing whatever they figure out as code so that it actually works. And about half a year ago I released, or I actually released it earlier but then I made it publicly available for everyone on CloudCraft where you design whatever you want and then you get use of building blocks of AWS infrastructure and save it as Terraform configuration. So this is exactly what Modules TF is for. I have some stickers here for those who want to get it. It's open source project, it's available for free for everyone to be able to generate your infrastructure as code from visual diagrams. So you go to CloudCraft, you can sign up or sign in with free account. This feature is available for free for everyone and you draw whatever you want to have and then you click export. And the end result will be zip archive which is potentially ready to use for you right now. The thing is that this code which will be generated is good starting point for people who want to get started with like enforced best practices. So code structure, different tools like Modules, Terraform AWS Modules and Terragrant pre-commits and few other things are automatically pre-installed for you. And the structure of the project is the same as you visual as you draw it in CloudCraft UI. So if you specify that you want to have different instance type or different auto-scaling and how they are connected, so if you specify that auto-scaling group, application load balancer, security group, VPC are somehow connected, all of these relations will be taken into account and converted into Terragrant. So which means that whatever your cloud architect has been thinking about, you don't have to go and look into diagrams again and think what do you mean here, whether we have to do this or not, it will be one-to-one match. This is ongoing process and I'm going to invest more time on this one to make it much more complete because now I have some things hardcoded but most of this, not most but actually everything of this, you will have to review and customize it for your needs. Yeah, but it's potentially ready to use. And it's MIT license because you can just go there, focus or contribute. It's written in Python 3.6, it works on AWS, Lambda, using sort of less framework. Yeah. So yeah, thanks. Anyone has questions? Yep. Can you repeat? The thing is, in step 5, like you say, what do you do with that? On one hand, you need to share that with everyone who's using that phone. On the other hand, it has all your secrets and things. So how do you deal with that? Do you have a lot of cues like you said earlier and what's in your experience about what it is? Yeah, that's a really good question about Terraform State. So quite obviously, this is one of the pain points for many people. Your state file is not containing all secrets which you didn't want to have in plain text. And there are many different solutions how to do this. One of the most popular one is you don't have to give access to all developers in order to read from this file. So you may have centralized, let's say, Atlantis. Atlantis is a tool which can be connected to your GitHub, GitLab, and since yesterday to Bitbucket as well. So you can connect it with your repository and developers from full request will be able to write Atlantis plan and Atlantis will execute Terraform on centralized place. So this means that developers don't have to have access to the state file themselves. They still will be able to read some of this if they provide some malicious code so it has to be a little bit more secret than before. But the other way is if you split your one big state file which contains everything and secrets into at least two state files where secrets are used only when it's necessary, you will be better to go with having one big state file. For example, there are many different ways in AWS itself to manage to not use secret stuff which will end up in state files. For example, you can use IAM roles, you can use BGP encryption for IAM users, access keys, and you can use different types of properties provided by the AWS provider itself. Approximately, as I understand, in a month, maybe in two, there will be Terraform solution which will allow people to not think about state problem anymore. So it will be a favorite free by Hashikov. Thanks everyone.