 If no questions, then it means that you didn't hear me, okay? so Now this thing doesn't work Right, this thing doesn't work. So I will have to use something else Okay, my name is Anton and I'm big fan of open source and big fan of oh, yeah You see we fixed printers with big sliders. What else? Excellent So still my name is I'm Anton Bobenka and I'm not from warm place as previous speaker So I get used to snow I live in Norway for the last 13 years and less than that I try to stay active with open source and for the last three years I'm actively contributing and having fun with Terraform AWS organizing different events like devils days in Oslo different user groups travel around and try to Make people aware about this acknowledges and how to use them properly so Some of you probably familiar with Terraform. So I just like to hear Faces and hands who know what is Terraform and who use it daily Right. Okay. Good. So it's about half of people and I Think some of you actually use code which I wrote or which I Contribute to or which I maintain for the last three years Terraform AWS module is one of the most popular one which was downloaded two millions of times and the last year and a few other projects related to Terraform I actively Stay in the community and listening in what kind of questions people constantly asking like how to structure my projects How to solve this thing how to do that better and so on so three months ago, I started writing Terraform best practices and It has not been finished yet because things are constantly constantly changing and I have to earn money and I have to Find some customers. I'm not employed by Hashi corp as you may think but Nevertheless, I write a lot about different things So if you have questions after this talk or about Terraform feel free to find me by my nickname This nickname is everywhere and if you are thinking what is this module CF? Yes, I have logo for this as well and I have stickers so more about this later Terraform AWS module is the project to which is collection of reusable components for traditional things like VPC security groups RDS and many many more it started More than a year ago, but a year ago as well as Release of Terraform registry This became verified and now they are used by lots of lots of different people. So Why Terraform right this talk is about Terraform and something about it. So Terraform was created in about 2014 And the only purpose of it was to be able to write plan and manage infrastructure as code This talk is Maybe a little bit more advanced and I assume that you know a lot of these things already So I will not go in details and I expect you to actually read the commendation in addition to this talk Documentation is excellent. So Nevertheless It works almost So this is how main TF file looks in Terraform you describe some variables you describe some resources you have some some dependencies between different resources and So you define some Information about AWS region where infrastructure is going to be created. You define what kind of resources like a bucket Run some comments Yeah No, but that's that's how it is Unfortunately, the projector is not so great, but yeah, I can read what is on the slide so you're uncommon Terraform in it to download dependencies and After that you're uncommon to To create these resources like Terraform apply and it will show you what exactly is going to be created and You confirm it and then you have these resources created Right. So now slide on white background illustrate that different cloud providers over time come up with different Solutions to Terraform like before Terraform for example, AWS cloud formation was created by AWS With similar purpose Google Cloud has come up with its own solution to deploy resources and Azure eventually came up with something similar But you may think why Terraform existed in first place Right. The main reason why Terraform exist is that there are much more than This there are more than 100 different Providers which allows to configure different resources on different systems And this is one of the biggest sales points why people should use Terraform sources Whether I have to stick with cloud formation if I'm not going to leave AWS, no, you should not use cloud formation So Terraform is universal tool for everything where we have an API an API believe it or not Exists even for Dropbox files even Gira issue can be created with Terraform There are many different use cases when you want to be able to configure your resources like data dog metrics or Control permissions to different objects. So everything what is available through API can be controlled So now let's begin with More or less real projects where at the beginning everything was fine everything fit into single main TF file And it was just fine, right? So we have I cannot rely on this That's a good idea So many geeks in the room we fixed Excellent, but we have not fixed the clicker yet. So I will stay here. No problem So on this slide we see how Terraform configuration looks when we just need to create basic virtual private clouds within AWS So it's simple nothing special But eventually your project will grow and you add different things into it And you may think what exactly you're going to add and you are going to add resources regions providers Whatever else you can figure out you integrate everything with everything. So it should remind you That you will continue editing main TF file and now you just add the internet gateway, right? Because you want to access internet good idea then a little bit later you figure out that you need some subnet and Then you probably need something else So you will add a lot of different Things and by the time when you figure out that you actually need to access Internet from private subnet you need to configure routes and routing table and all of this together Your network configuration in Terraform will end up in about 300 lines at least it can be much more So you may think what exactly is going on with main TF of course It's growing and it's growing Approximately 10 20 kilobytes for just networking stack with more than 300 lines of code Dependences eventually will get worse and worse. That's Natural solution why we're here is a terraform come up with terraform module. So terraform modules is just a I don't know So yeah, actually speaking about modules magic modules is a reserved name by Google You can go to it so Okay Right so modules. It's really important to understand from Terraform documentation are just self-contained packages of Terraform configurations That are managed as a group. That's the only thing which you have to understand about modules from Terraform documentation There are several types of modules like the first one which is the easiest one more Very reusable one because it's possible to open source and it's hard to invent something there It is resource module where just the resources can be created in different configuration nothing else There as in this example there is a way of using security group module of Which is hosted on Terraform registry and we specify that we need to use version to zero zero and we provide bunch of Arguments to this module. So that's all what we have We We have information Yeah, you may think what exactly the purpose of resource module and the main answer is it Such resource modules can be versioned while individual resource block cannot be versioned But if this is not convincing then think about security group module and this is a real code within the security group module It is creating it is creation of easy to Security group module in different different configuration Believe me, you don't want to write this code for your project because I hope this is not your business I mean, you probably don't earn on creation code in Terraform and selling So the code itself is about 600 lines and it can be very powerful way of abstraction next slide second type of module is infrastructure module, which is just a High high level because it consists of resource modules and it often enforce something What is related for the company like tagging naming and so on and that's exactly where where it makes sense to reuse Existing tools like json that or cookie cutter or different preprocessor make file whatever else to fulfill missing beats of Terraform 011 And infrastructure module in vacation looks pretty similar. So we have module hosted on the registry and we specify a bunch of different values which we pass as arguments to the module Inside of this module. We are just in walking resource module for VPC application law balancer and many other things So just to summarize There are two types of module resource and infrastructure and resource is very reusable one So let's look into how to write modules So the first tip which I'd like to give is to not write resource module, but just check what is existing there There are more than 600 modules published in the registry Many of them are very good quality. So don't be creative unless you really need people So if you actually decide to write some modules then try to Hide all this complexity as I showed for security group somewhere inside of this module as in this example People who are using the module they don't really know difference between Implementation of time zone in a massive scale versus my seat So they're just calling this module and then inside of the module. We are using we are creating resource based on Certain criteria like in this case if name was a skill server then create MS SQL Otherwise create my sequel and use time zone only for MS SQL As an output of the module we are outputting The only result which is available there So user doesn't have to know that we are actually creating one type of database or another Different implementation details are totally hidden second thing is size of the infrastructure modules if you think that It's okay to just host all of your infrastructure modules in the repository and just use part of it It's often gonna to be cloned every time when you are invoking all of this So your module will be just several kilobytes of code while the whole The whole repository is several megabytes then you will have to clone every time the whole repository Which is just an efficient especially if you're making lots of small resources like I am MBT is your solution to split one One big gift repository into several and you can do this Few things which I'd like to point which you have to avoid so first of all is never use providers in modules and The only exception is that you may use logical providers like template random local HTTP external But not in cloud providers or any other state providers You may think why it's actually bad because I did this and it works Yes, exactly. It works for you, but it will not work for Peter or vice because the way how the terraform is invoking providers configuration is Not possible to override So if you put this code inside your module and you want and you ask why people are not using my module That's because of this Partially another thing is that provisioner itself pretty pretty bad idea to use in all resources I know that a lot of people are using them, but I just try to show why this is good Even if you are thinking that What what can go wrong here? Or I'm just going to use it in EC2 resources because it makes sense like my instance is greater than I'm calling Ansible playbook which will just connect to the safe address. Yes, it's probably fine for some time and What can possibly go wrong? So the good solution here is that if you can use Things like user data on AWS inferences then use that one if you can use Autoscaling group then use that one as well The benefit here is that if you start using Provisioners you will have very hard time migrated into other project or into other Yeah, into just other Project or if you want to distribute it somehow So the way when you actually you say that no, I absolutely must use my local dependencies and I want to use it Use my local machine when something is created then use no resource for that No resource doesn't create any physical resources. So that's exactly the place where such things can be implemented and And few trades about to do terraform I was working with terraform modules for pretty long time and then I discovered that it actually makes sense to look around and think what are Common mistakes which people are making and why they're actually mistakes and I ordered this in the order of Importance the first the most important thing in terraform model is the commutation as an example and The least important thing is actually test If you disagree with me, you're welcome to be or F which will be tomorrow slide later Fissure rich is important feature off of good terraform module because it means that I Can use your module and use it for my subset of things and someone else can use and everything should be fine so that's really one of the One of the core feature because some people think that it works for me. I don't care. I wouldn't improve it That's fine But if you want it to be actually good and reusable then it has to cover as much as possible different situations and never hard-code values like Some some things which are related to just your company or your project things like environment name or company name Or even tax everything should should should be customized or at least all the reason and Make sure that your code is actually readable even by you after a couple months It's good suggestion because believe me, you will you will be using the plane to figure out that that you wrote this code Okay, so let's I Don't know someone says that I have very few minutes left. I think I have plenty of time So, yeah Just summarize We looked into how to write modules. So first of all check out registry dot or a form dot IO and point something there It's it's likely to be excellent starting point at least if you're not going to use it entirely and Avoid things like providers and provisioners at all in modules use it somewhere else outside and let's look how to call more Okay, as we understand the amount of resources Still keeps growing we figure out that we just Made our module we put something there. We somehow have to orchestrate this invocation We somehow have to call one module and pass dependencies to another one and so on. So let's look into different patterns here So the first pattern is all in ones where you have to declare variables and outputs in fewer places, but The last radios is pretty big You may use some things like minus target as an argument and specify that you just want to take care of this specific subset of resources it will work for some time, but it's not the idea and Everything is blocked at once means that if I'd like to cooperate with some of my colleagues on this project When they run it, I cannot run it. I cannot run plan or cannot apply it. I will have to wait and if we are talking about Resources like cloud thrones or if we're talking about Azure provider things can take minutes Sometimes hours. So this can be very strange that you have to wait So Another pattern is one in one Where things are getting much more isolated and it is possible to work in an isolated set We have different Responsibilities, for example, if we're working with VPC, it's unlikely that we're gonna have to change VPC every day, but it's probably a good idea that we or it's going to be quite possible that we're going to work with application configuration code quite often because we'll use VTOPs and we'll change all of our TFRs file from CI server. So then it's a good idea to separate and make much smaller less ratings But the only downside here as I understand is that we have to copy paste a lot of variables and outputs if we want to use this pattern So now let's I don't see so many hands obviously, but the question is What kind of ways do you prefer where things are organized like this? Can you raise your hands? Okay, at least one I see no too. Okay. Yeah, but not so many. Okay. Let's Who's using this pattern one and one right so much much more I can see this even now That's really good so There are a few people's raised hands all in one but Significant amount of people raised one in one so and MFA is most frequent answer Actually is somewhere in between because it's very often when you start using something You are not thinking like oh my god, this is going to be my ideal project I will make everything correct from first day, right? It's it just not gonna happen So it makes perfect sense to start with something small and then iterate to the and then when you figure out that now I have I start to have some mess then it's time So how I can actually Invoke different modules What can I do from there and? The question is what kind of orchestration tools do you use I can give you him that you can use minus target Terraform apply minus target and then specify some module or you can use make file to chain Different invocation of Terraform in it apply in it apply outputs and many different crazy things What what exactly? Are you using? Like just randomly Who's using things like target or make file and happy with it? Right, so it's pretty funny that I have so many questions and I don't see answers, but I guess it's five ten people Okay, so I have some good good news for you. You can use Terraform for that as well Right, you can use no resource to involve Terraform from Terraform. Isn't it cool? No Just don't do this. Okay, there are better solutions for versioning, but calling Terraform from Terraform is Is doable I did it But I did it just for fun to figure out that you have to treat Terraform a little bit different It's not it's not made for that Terraform was designed to run just things linearly where you change while you run plan run apply You see output and you can continue running the same things to verify that infrastructure has not changed But if you're actually thinking about orchestration in Terraform the best tool out there right now is terra-grant Terra-grant is tool is open source. It's written by grunt work People and lots of users worldwide Are using it to exactly involve and change different invocation of infrastructure modules So how it looks the Terraform TF vars file has configuration block Where it's specifying where and also what kind of dependencies this module has in this example We are calling easy to instance module, which means that network stack has to dependencies and Also, we just specify what This So that's the only thing which is, it was very easy to, to make it, and it works quite nicely. Because everyone has configuration, not like poops, before poops and after poops. So before running plan or apply, I just wanted to make sure that I actually engaged information about some of the VCA here, or bunch of other things which don't really have to be combined and put it into the TFR file. If you want it, you can use it on your TFR. So we're just going to do about the core models, and now let's look into how to actually write or how to work with code. If we have new features that are on the configuration, it's easy. I'll separate the folder. The pattern here is that we use data source and we use resource together. Data source will pick up if ID was present, and resource will be created if it is not present. And again, the output of this value will be the first node empty value which is there. So we're working with list. The problem with telephone 011 is that lists in telephone have order, which means that if we have a list of IM users, user 1, 2, 3, 4, and then we want to remove user 2, then it means that user 3 and 4 will have to be recreated because their IDs or their placement has changed. For some resources, it's totally fine. But what if we actually create IM users and IM access keys which are stateful? I mean, when someone whose name is after my name, leaving the company, or before my leaving the company, I don't want to have new IM access secret key generated for me. It will be very strange. When person with last name starting with Z will have new IM key every day. That's not going to help you. So the solution here is that if you need to work with stateful list, you may use tools like jsonnet. jsonnet is open source and it's available as CLI where a template for it looks like this. Where you configure... No, this is actually not template. This is an output. It's a template. So the template here is written in jsonnet format. And as we probably know, json and htl are compatible. We can use json file as well as htl and Terraform will recognize that. That's why generating of json is usually easier than generating of htl. So if we run common like this, then the output will be a collection of modules for each individual user. And if we run Terraform in it and apply, it will just take care of this user. It will not remove anyone from anyone in the list who's after. Another thing is related to integration. It may think that Terraform has everything that I need, but there is no way to do, let's say, a specific feature. Because Terraform is not always up to date with recent announcements by AWS. And if AWS announced something, it's likely that it will be supported in the near future, but probably not right now. So there is still need to use some things like AWS CLI where we can just use output from one resource and execute it in shell. If we need to use outer integration, then we again may use no resource for that. There are a bunch of bunch of edge cases, which I'd like to talk tomorrow during BOF about infrastructure testing, but just a small subset of these problems and edge cases is that different AWS regions as well as different AWS accounts have different things, like version of S3 signature, whether EC2 classic link was enabled in this account or in this region before, whether IPv6 is available here or maybe it's not going to be available, or maybe IPv6 was there all the time, all of this will have different effects on how you write Terraform. So be very careful and think carefully where this specific code will be executed. So what is the top limit of resources in AWS, like you may think that it works in dev environment because I'm using something very small, but it will break in production because you have the limit amount of IP addresses, for example. So just be careful about that. And there are a bunch of things which I'd like to mention related to... If you are willing to use Terraform in a way that you remember the comment which you wrote last time when you worked on this project, then the best comment is actually no arguments at all, so that you run just Terraform apply or just Terraform init and don't specify any arguments, especially if it is not secret. I mean, if it is not secret, then you put it into TFR's file. If it is secret, then you use it as an environment variable and it will be loaded or picked up by Terraform. But it's a pretty bad idea to remember that you have to run Terraform apply and then specify two variables and remember order of work files which you have to write. It is very easy to break and you will just make problems for yourself. And also you think that using target and parli-business is a good idea because I have big infrastructure and I just want to work on this specific area and it works very fast so I will just keep running with minus target and don't care about the rest. It will be probably fine for some time or for really small edge cases. But if you keep doing the same things over and over, you will simply not be able to answer question what kind of configurations I have on this environment or in production or in depth with this target so you didn't find it for everything. And Terraform workspace is evil for majority of cases. I've been in this discussion many times and I'm glad that Terraform documentation was updated not very long time ago where they actually explained quite nicely why and what exactly workspaces were designed for because there was huge amount of confusion where people thought that I can just use workspaces for depth staging codes and everything will be fine. No, it's not going to be fine. And the reason why it's not going to be fine is that very soon you will have differences between your environments and something will not be as equal in depth while it should be equal and huge amount of complications you are just bringing to yourself as well as dependency howling modules. Terraform does not have a good pattern of using or specifying dependencies between modules so there is no magical requirements TXT or package JSON whatever else we get used to in other languages. There are some attempts by community to write a Terraform file where modules will be configured in one place and we can sync them together but it's good just for the highest level. Don't try to use module in module in module in module simply because your file system supports sendings. It's not a good idea. It's just one or two levels maximum isn't it? So, for the time, as another type of summary I'd like to highlight that I think the best thing or the best advice which I can give is to proceed Terraform easily. Even if it has CLI of 20 or 30 different arguments and you think that you will need all of them at once try to not use all of them. And you may think why it is so because somebody wrote them so I have to use it and if you're even curious why it says for the future in the title because there will be 012 eventually. There are some core contributors to Terraform in this room somewhere there so you may ask questions later and they will be able to answer and hope. But the point of 012 is that you may think, you may be very excited that HCL 2, wow, simplified syntax. Cool, so I have to start using 012. The loops are going to have to be there so I can use all countless core loops core each for dynamic loads and finally left and right part will not be executed simultaneously in conditional and I can use conditional operation properly and many, many other things which you think you need but what I've just tried to figure out from the community is that a lot of people think that 012 is what they need just because I have these problems with my code for some time and 012 has all of these features so I will just start using it and it will somehow make my code easy. I personally don't think that it matters so much 011 had lots of things which people were happy with we created lots of things using it it was fine but with the amount of features which 012 is bringing into developers I'm afraid that a lot of developers will try to use much more of these features in TerraPoint codes and it will be just very easy for them to make another sort of spaghetti code. The way how TerraPoint 012 has to be used and where it's particularly beneficial is just for writing TerraPoint modules itself so that when you're thinking about I need to make a module for F3 buckets by the way this is one of the modules which I couldn't write for the last two years because there are 225 permutations and I don't want to make 225 modules so if I'm using 012 in order to use dynamic blocks and four loops and so on inside of module then it's great but I don't want developers who fall in love with HCL with easy syntaxes and they say that omega-json is bad but HCL is excellent and now I have to tell them that oh by the way you can use loops and don't forget that you can use this and this and this it will be just another level of complexity for them if you really like 012 then put all of your knowledge into writing modules and not let your developers who are just starting especially TerraPoint learning the overwhelm with amount of features and I really encourage to use existing modules and utilities provided by the community I'm sure there are some people in this room who wrote some utilities so thanks personally for all of you because I use a lot of those tools I have some bonus I can see who was sleeping so not many who was sleeping so that's good so some of you may saw this before so this is a tool called Cloudcraft where you can visualize your diagrams of your AWS infrastructure and you can make different boxes together you can think about what kind of properties or attributes all of this has to have and about a year ago I was thinking like yeah but we have TerraPoint we have infrastructure code we have AWS console which people should never have access to because there is code already so why you should be able to manage something there so it's kind of not exactly the point and a year ago I was thinking like it would be probably good idea to make generation from this visual diagram from Cloudcraft into TerraPoint AWS modules so using TerraPoint AWS modules as building blocks for AWS infrastructure and use TerraPoint as a tool to actually call all of this so that's why approximately two months ago so I open sourced this and it is infrastructure as code generator which gets your visual diagrams with your cloud architect love and draw and give you some information as TerraPoint code you can try it like this but some of you may think like yeah but we've already saw different bootstrappers which could just not bootstrap but boilerplate code which we just create in hello world but it's not actually practical and we just believe it immediately that's why I paid lots of attention to how to how to write it to be as close ready to use as possible take a lot of amount of attention to all details, connections and actually enforce TerraPoint best practices by installing and configuring different tools like pre-commit hooks TerraPoint it is open source and if you like AWS Lambda and Python you may try to open an issue there and yeah that's it so there will be some we still have some time speak loudly please don't get up and start moving because that will disturb the question round and we have a little bit of time so thank you very much for the talk in all of the time that you spent with Terraform what is your preferred state management and state locking system I know it's kind of off topic but yeah and locking how do you make sure that it's locked yeah so the question is about state management I hear this question pretty often from not AWS users because it doesn't have any block storage which is used now I mean block storage I mean if you say that you don't want to lose this file and then you delete this file you actually lose it that's not state management in Azure so a lot of people are happy with whatever they have easily available if they already have console installed in their infrastructure then they keep using console but if it's just a new project which started from zero then F3 will satisfy your needs for a very long time and I personally have not heard almost anyone who is using something else I mean of course on Google you can use whatever Google provides but there are no I have not met anyone who is saying that oh we are using just solution so console F3 and for locking and for answering questions about locking it's also whatever you have in place most of people will never will never meet any challenges with DynamoDB it works just fine alright any more questions one more time I asked if you could please not start moving because it's really disturbing for everyone who is interested in the question well thanks for the speech I would like to ask I don't know if your experience have seen something like that but I I don't know if you have seen in your telephone life something about like changing from current modules that you are using to cloud formation stacks as far as I know there's no tool for that right now and I don't know if you have yeah so it's pretty interesting question because once people start using telephone they are almost never looking into anything else and there are some competitors but it's not cloud formation so even if people are with AWS and they say that we are not going to leave AWS we are quite happy with cloud formation it's unlikely that they will suddenly stop using telephone and continue using cloud formation there are no tools as far as I know but there are some quite easy ways how you can do this since most are pretty well documented you may have some yes there are tools like telephone to generate telephone configuration based on your exit to infrastructure so that you run telephone and it will give you HCL snippet for this specific resource which is what it is and it will be able to adopt existing resources into telephone state as well but there is no need because I have not heard as you mentioned my telephone wife I think that on this topic where we've seen it is that every once in a while you see these cloud formation stacks that are fully developed and to convert something like that to telephone is not always easy for instance AWS landing zone stuff is like a huge cloud formation stack and so I haven't seen a proper terraform landing zone answering or adding information about migration from existing resources is not an interest or AWS to let you go so they clearly have something internally they pay attention but clearly what's going on in the market but it's not something what they're interested in doing that's why you don't have even access to state you cannot do anything with state so you have to literally kill this resource and start it in another place in many situations all right thank you very much now you can move