 Yeah, we can start. OK. Please welcome Anton Babenko, CEO of BetterJob. He's going to present to us some rather complex terraforming cases in PDO, you have to say. Thank you, Anton. Thanks, everyone, for coming. I will try to explain maybe in some unusual way of thinking about terraform. So, first of all, the title of the talk, as you can see, has a point in three key words. It's like gotcha, terraform, and secure. Or actually, even more, delivery pipeline. So, let's get started. So, first of all, we'll see what are key concepts in terraform and how they are related to CI CD pipeline and how terraform can be used. And then I will go into frequent terraform problems which I experienced. So, first of all, who am I? I live in Norway and I'm originally from Ukraine. So, it's my first time in Singapore and in Asia in general. I'm kind of very excited. I'm very hot here, but at the same time, it's very cold inside. So, that's why I have this jacket. I like to call myself as terraform and AWS Fanatic because that's what I've been doing for the last many years. And in my spare time, I organize different events, different conferences in Norway and meetups. So, I organize HashiCorp user group and AWS user groups in Oslo. And I actively contribute to different open source projects, mostly related with terraform modules and some things in between. So, if you have any questions about this talk or anything what I'm going to mention, don't hesitate to send me a message on any of this channel. So, as Michael just mentioned, I work for a company called Better Job. That's my own consultancy company where I spread love about these technologies which I'm going to talk about. And I also contribute to terraform AWS modules, which we'll cover in much more detail later on. And yeah, that's another project which I'm working on. Modules.tf to not confuse that tf is not related to TensorFlow, which I'm sure you've heard multiple times during the last couple of days. And yeah, that's not about TensorFlow. So, first of all, who knows what is terraform? Like, excellent, excellent. So, much more than half, that's really good because in this talk I'm not going to tell you what is terraform. So, I assume that you can go to a website like terraform.io, read the documentation, see some code, see what is it, and kind of get the feeling what is it. Just as a summary, terraform is a tool to write, plan, and create infrastructure as code. There will be talk by Seth Vargo, five o'clock in lecture theater about infrastructure as code and actually everything as code. So, if you are like me who like to describe everything as code, please go there and learn what else you can actually describe as code. I'm sure you will get some points which are pretty new. So, this is traditional AWS infrastructure for multi-tier application which consists of some load balancer and some auto-scaling group with Frontier and some web servers and some databases and so on. So, that's traditional set of resources which we can create in AWS console and get our project up and running. So, normal way or like last generation way is going to AWS console, click, click, click, and then you have something like this and your service is up and running. But it's not so good. Much better approach is actually to describe this as a code where in this example you can see that we are creating AWS S3 bucket with specific name and this name is actually consistent some random value which is random path resource and then we output this value. So, this is how typical Terraform configuration code looks like. After you describe this infrastructure like this, you run Terraform init to download some dependencies and then you run Terraform apply to see what's going to be applied. So, here you can see plan that it's going to create a couple resources and then you confirm that yes, that's what I want and at the end you get this nice my bucket sysnail as name of the bucket which was just created. So, that's eventually the whole purpose of Terraform, nothing else. But why CICD pipeline is important? So, the whole idea is that it gives you clear identification of bottlenecks and you can fix them faster and move along much more confident. So, typical CICD pipeline consists of steps where developers write code, commit this code into Git and then execute some CI pipeline and then after some unit test or functional test passed then this code or this change is actually passed to continuous delivery pipeline and it was reviewed by human or by some other things and then it will be deployed to stage environment and eventually to production. So, this image is taken from GitLab documentation and I'm glad that previous talk was about GitLab as well. So, yeah, guys, this is how GitLab use it. So, getting started with Terraform in fact is very easy. You go to documentation, you read it, you write it. You read documentation, you think that things look easy then you copy resource code from documentation, you write it. Terraform apply, you're happy and it works. I mean, yes, it works but only 60% of the time because you have one developer on Windows, then you have some difficulties with configurations and yeah, it can be different surprises. So, you think like, okay, what's next, like how to actually improve my code and then you figure out that concept of modules is there. So, the purpose of Terraform modules is that you don't have to be the smartest person if you're solving recurring tasks. For example, if you are making virtual private or VPC on AWS, believe me, you're not the first guy on the planet who do this. So, yes, there is a module for that. So, don't try to invent things which were solved by many, many, literally thousands of people before you. So, registry.terraform.io is a place where currently more than 300 modules are hosted and Terraform AWS modules repository is a place where I manage significant amount of Terraform AWS modules used by thousands of people. So, as a fact, there were, I don't know, at least half a million of downloads so far and people are using it and commenting and contributing new features. So, I encourage you to check this one before solving recurring things. Terraform workspace is a feature which is internally provided by Terraform to allow you to split your code and to be able to execute this code in slightly different way. If you read the documentation and think this is some magic which can help you further on, don't think so and skip it and don't use it really. So, talking about CI CD tools which are available now, there are managed systems like Circle CI and Private CI and many others and also if you are not willing to use managed servers and you like to manage things yourself, there are plugins for concourse, drone, Jenkins, even for Ansible. So, you can use plenty of different ways to run Terraform. But again, Bash will be required extremely often because you still have to call Terraform with different parameters and configure it differently so Bash is your best friend still. There is no better support than that. Atlantis is another tool which was specifically made to simplify workflow collaboration using features of GitHub and GitLab. It's very young project, I would say. But nonetheless, it's used by, I guess, dozens of people worldwide and it's open source and it has some really nice features especially if you don't have budget to pay for Terraform Enterprise but still want to have some features like pull request locking so that only one pull request can be merged at the same time so there are no conflicts where people change the same resource multiple times and one override another one. So, these tools like Terraform Enterprise and Atlantis can solve for you. Terraform Enterprise is built on top of Terraform open source where you have graphical user interface for common features like plan apply and so on. So, right now we have seen how Terraform code looks like what are CI CD options. So, let's look into specific continuous integration and continuous deliveries which are related to Terraform. So, first of all, if you are going to run Terraform in CI CD pipeline the only option which you have related to state management is having a remote state only, like exclusively. Second thing which you have to know or which you have to take into account is how you are going to handle errors. For example, if Terraform is executing and there are some errors in the middle by many different reasons Terraform will not be able to roll back to previous state or will change your resources to the previous state so you will have to fix it yourself. There are multiple interesting challenges with this and how you can solve it. Most simplified is give it another try and maybe things work this time. Sometimes it works, sometimes not. If it doesn't work you can say that hey Terraform don't try to make things so smart and run it in six parallel API calls to the same providers so let's keep parallelism to one. It means that sequence will be reduced so it will be just sequential execution of API calls but it may help you with eventual consistencies, problems or API can say that hey you are changing too many things at the same time so try to reduce it. And also detailed exit code can be handy if you want to see is it actually a problem or new resources has to be changed. Once you run Terraform in CI CD you have to make sure that you always get the same dependencies as before. For example, you can stick to a specific version of providers and to a specific version of Terraform itself. There are solutions for that I will show later. And once you run Terraform in it you have to keep .terraform directory between your executions so this means that once you run Terraform in it nobody should be able to change files like to replace your provider dependencies because they were already downloaded. So there are certain challenges with this if you just allow to execute Terraform in it every time. So yeah, things with private repositories in particular on GitHub is that once you start using Terraform modules which will host it in private GitHub repositories and you fork this private repository to add your own features into it but repository is still private the funny thing is that once a maintainer or parent private repository decides to delete it your fork is also gone. So I think this is kind of strange but that's life. So if you fork someone private repository make sure that you have a proper copy of this code somewhere else. I've been to a situation where a repository was deleted and somebody asked like why it was deleted I forked it. Yeah, you forked but it was still private. So anyway, so moving next CICD system allow us to manage secrets as part of their process which is fine and a very important point in Terraform is that everything secret or everything is stored in plain state file which is essentially a JSON file so they will be stored in a decrypted way in many situations. Sometimes Terraform allow to use PGP for things like AWS IAM username and access secret keys so make sure that you use these features if you can. Also providers like relational database service has possibility to want to use IAM to authenticate users but even best approach for that is once you use Terraform to create some resource using some sensitive data make sure that you change this password outside of Terraform later on. So since password will not be tracked in Terraform internally it will not tell you that hey this password was changed so I want to restore it. Password is almost always tracked outside of the system. So change it outside or change it as post-execution hook or anything like that. So one of example is random string password. So if I want to generate random password that's typically how you can get random string with 10 characters but the funny thing is that once execute Terraform apply I can still see this password here and if you have this as an output of your CI server you think that oh I prevent people from executing my Terraform configurations and nobody can see my state file so now they don't have possibility to see what is in state file but they can simply see it in output and they can see password as plain text here which is a bit strange. So kind of obvious situation but again pretty important so just don't store passwords. Talking about access control principles is principle of least privilege as you should be taken into account very seriously. Talking about access control principles in terms of AWS I am is that grant only the permission to perform task. So but even more specifically talking about AWS and Terraform one of pattern which I encourage people to use is that an example when you run Terraform in it refresh, validate, plan and few other commands you don't actually need to have possibility to change resources which you have on AWS or any other provider so it's enough to just have read only access by assuming read only I am role. So this I am role will allow you to read things from DynamoDB from S3 bucket and can read resources and describe resources on AWS which is fine but if you are about to apply this change then yes you have to actually assume admin role which allows you to change this resources and the important factor here is that read only role can be assumed by anyone without specifying MFA token while admin automatically expect multi-factor authentication token to be provided when assuming this role. So next thing about secure delivery pipeline is to ask people and to make sure that they really want to apply what they just created so there are multiple things which people can use for example they can review pull requests of each other and say hey you probably don't want to destroy this mega cluster but maybe you wanted something else GitHub control and to assign reviewers for example there is file called code owners on GitHub where you can specify if files in this folder were changed then assign pull request or to this team but to have even more sophisticated examples where you want to specify if changed in this folder then I need to have plus one from this team and not more than one minus one from this team so to make this kind of get it like approval system guys from capitol one makes check out so I have not used this myself yet but I think it's pretty promising unless GitHub implement this natively and it tightly depends on how you structure your code so I hope that everything what I was talking so far was basics and you understand it and it's fine so let's deep dive a little bit into Terraform design patterns and why they are important so there are three types of design patterns which three types of structure and thinking about Terraform code so first is resource modules, infrastructure modules and composition so resource modules is the main purpose of resource modules is obviously create resource nothing else really they don't have any relations between other Terraform modules, they are flexible in what kind of arguments they accept and they usually support semantic versioning so Terraform AWS modules is a good example of just Terraform resource modules infrastructure modules on another side is something what is on top of resource module and usually I like to think about them as something what provides company-wide standards like tagging, naming and passing these values into Terraform resource modules and this is also a good example to fulfill missing parts of Terraform and HCL for example you can use code generations like JSONnet, cookie cutter and few other things and versioning is still important but not as important let's say to follow semantic version or anything else but it's just good to have some versioning Terraform composition is actually on top of infrastructure modules and for those who don't know who here know what is TerraGrant great so this is unexplored market I see no one is using TerraGrant but TerraGrant is a tool which allows you to write Terraform compositions and manage resources by regions, by environments and kind of don't repeat yourself principles all over the place so I highly encourage you to take a look on this and see if it is for your setup so basic pipeline as I guess most of you know looks like this that you work on your feature branch you push code into this feature branch then you open pull request then pull request is somehow approved by someone and code from this branch is merged into master and then you execute Terraform apply on the master branch so master branch contains single source of truth for the code and for the infrastructure in general so sample pipeline when using CircleCI looks like this is that code is here you cannot see but here is the code is fetched from github repository then code is validated that it doesn't contain syntax errors code is run through Terraform plan stage then human approve that this infrastructure change looks good and then this is applied so all of this is described as code again we have infrastructure as code we have pipeline as code and we will have more and more things as code that's absolutely required part so pipeline for modules I guess we can skip this because we don't have too much time but it's cool and I will put these slides somewhere and there are a bunch of tools which I can just briefly say if you are like me who likes to pay attention to the style and don't discuss hey you forgot space here hey can you move this curly brackets to a new line really that's not the thing which we have to discuss in 21st century or at least 2018 so there is another project which doesn't seem to be so much updated tf lint but it's still good tool to run and be able to make sure that your code is actually in good condition it doesn't violate any principles and yeah that's it so keep calm and automate everything kind of obvious but not so much because really all these can be automated no not everything so there are a bunch of things which pipeline cannot do for you refactoring is a good example for that so state mv is your kind of command of choice when you have to move resources between state file or if you do refactoring in terraform I was running this refactoring things for so long time that I decided to make terrible project out of it where I mean terrible is the first part is from terraform and Ible is part of Ansible so this is project where I manage terraform configuration using Ansible and yeah it works well because I can refactor and do changes in about 1000 to 3 sources just by executing sequentially similar commands in a loop so in a loop of loop of loop something like this so yeah it looks terrible but it works so when it comes to terraform upgrade so you you was using one version of terraform and then you think well new version came out so I want to upgrade so you upgrade and everything break so which is not very often situation but still it happens sometime and the way how terraform handle state file is if you already upgraded your state file to newer version you cannot roll back to previous version so easily and say hey oh yeah I know that this version doesn't work so let me roll back my version so you have to do some magic literally like this to download previous version of your state file and to push it as a current version to terraform so the terraform can actually use it Atlantis is good tool to help you with force unlock force unlock is necessary when somebody execute one command and close laptop for example and think that it will finish by himself or by itself no it will not finish unfortunately so you have to sit and wait when your RDS database is being deleted it can take some time or yeah there are some resources which can take 20 minutes to delete so Atlantis can help you with this again it has support for unlocking demo the code is on this one but I'm just very much sure that it will work yeah because no maybe I can try here do you see my screen not so much so this is how configuration can you see it so this is how configuration for for this looks like so I define which version of terraform I want well I want to have state file in which region yes I'm from Europe and then I specify which version of AWS provider I want and I use data source to discover already created resources like default VPC and my licensor right so it will probably close by itself but it's life really I'm trying to demo this stuff for three times and never succeed yeah anyway let's try it like this you don't want me to pay for this anyway so yeah so here I discover maybe I can try it evaluate for free evaluate accept yes cool I have half hour to finish this so I get information about VPC subnet and which Amazon machine image I want to use so then I'm calling a couple terraform AWS models of specific versions where I specify which security group I want to make so SG is not shortcut for Singapore it is shortcut for security group and here I'm making one security group which opens all HTTP port port 80 some other ports and then I'm calling another module called EC2 to make instance with provided values and then I'm assigning this elastic IP address to the instance which was just launched so that's it as a as a result right no not right so when I run terraform in it I probably can show I executed this earlier today so it worked like this this is in case internet doesn't work and internet doesn't work now so when I run terraform get information about already existing resources like VPC Amazon machine image and so on and then it tells me what is going to be created so in this case elastic IP address will be created and few other resources which I just showed code for so once I say yes then this information then this plan is actually executed and it creates resources in different order so the ordering of resources is the best part of terraform. You never have to specify which resources have to be created first, second and so on so you just pass ID from one resource into another one and terraform will identify the dependency graph and build infrastructure in very smart way in many situations it is faster than using for example cloud formation that's fact and yeah so once these resources were created then it outputs public IP address like this and then I specify that I want to destroy it because I don't need this anymore I run terraform destroy and bunch of resources should be deleted and I say yes again and they destroy it so that's the whole life cycle where I describe infrastructure, I created this here I do whatever I want and I destroy it so going back to slides I have one or two slides summary CI CD tool is good and it's really brilliant for making automated things but there are a lot of things where CI CD tool cannot help you or it cannot help you at all or it can help you just for small fraction so and always use minimal roles and make sure that you use MFA even if your CI system is hidden and protected and everything is fine still it's better to make sure that MFA is provided for those who are interested in terraform and modules and workflows I will be talking on Monday on AWS meetup about terraform modules and yeah it's free meetup and everyone is welcome to come to that one and there will be a lot of interesting stuff outside of today's talk so feel free to go there and really the last thing which I want to get from this presentation is when you are thinking that terraform is cool tool and it has plenty of options and I have to use all of them at once no, you don't have to really it's important to use just minimum commands so that's why in example which I just showed you I didn't pass any single arguments to command line because I just used terraform apply, I didn't specify which var which tfrs file I need to use and so on and it's sometimes hard to use like that but things like terraground simplify it a lot so yeah I hope you have questions even if you don't have questions now you can always reach me later and I have bunch of stickers for this company Tashikorp and terraform and few other things so feel free to shoot your questions talk about multi-factor authentication sorry you talked about multi-factor authentication so how do you use MFA from your CIE that is you will be triggering a build and where does the second factor of authentication come in yeah so that's tricky part to implement right now so one of solution which project where I work now we are transitioning from using it in CircleCI to concourse so there we most likely will not have this kind of MFA there but there will be there will be there will be implementation of check who is like if your github username have access to this project as admin or as write then we assume that we are good enough so we don't have this check for MFA on AWS I am role but when it comes to I'm big fan of executing terraform from my local machine in exactly same way exactly same procedures as CIE would do but the way where I provide MFA from my local machine is that I have I have possibility to assume role before executing terraform so I have shell script which use my I am credentials which are valid for one hour so really every hour or so I have to type my MFA token from out to terminal it is possible to automated but it's very hard to make it like really nice and user friendly I also consider another option of using I don't remember feature of last pass where you can get or you can send application request to user and say hey was it you who requested this and it can raise a pop up on mobile where he says that yes it was me and then he press this button and it continues I think last pass and either Okta or Duo Security these two applications I don't remember this workflow but it's very hard to implement it nicely and user friendly yeah I can probably mention few other ideas which are more crazy but they look yeah they should work yeah so everything you put that into the circle CI environment so talking about talking about variables which come into pipeline like the input of the like everything that you need to put into the input of Terraform so if it comes there are few ways how you can get data into Terraform if it is secret data then the only option which you have is environment variable so you can specify so it looks like TF yeah so it looks like this TF var underscore and then name of your variable so this type role underscore type equal read only means that Terraform will know that there is role type with this value and then you can use this in your configurations so you can specify in circle CI environment variables which are available on each build so you can you can put some stuff there but again if it's about secrets then the most recommended approach is to put TF var underscore RDS password equal dummy or equal change me or equal never use this field anymore something like this and then change it once Terraform created it so then it connects you can use new resource to connect to the database once it's created and change it later the other question okay so thank you Antel and looking forward to your talk this evening. Yeah. Hope to maybe see some of you there. Thank you. Thank you.