 We've heard from several partners today. Now let's hear from another customer. We'll see how Bloom AG unites GitLab pipelines in Terraform for a single source of configurations. You know, configuration is important for consistency, security, and speed, but it can be a pain to manage across multiple tools. So Phillip West-Vallen with Bloom had the idea to combine configuration efforts of GitLab and Terraform into one place. They automated everything and used a top-down approach to standardized projects. This innovative idea may not be for you, but it's interesting to see the out-of-the-box thinking. Maybe it will spark others to innovate, like Phillip did, or to engage with GitLab on new ideas that we can incorporate. Remember, you can chat questions at any time throughout the presentation. Welcome everyone. Today I share you our way how we managed our GitLab projects and Cloud Infrastructure with Terraform. My name is Phillip West-Vallen. I am a GitLab hero and hosting the GitLab Meetup here in Hamburg. Part of that, I am a software engineer and work for an online shop, which sells flowers. If you have any questions, feel free to ask in the chat or hit me up on Twitter. So let's start with our background story. We recently started to rebuild our online shop and the technical landscape behind it because we want more speed and start into the public cloud era. So we need for that infrastructure as code because we don't want to figure out everything manually. And yeah, manual steps are really dangerous there. And yes, so we also want to automate everything possible. So what is the idea we have? We want to configure our infrastructure in one place. This means an essential central space anywhere in GitLab or in our GitLab project and a repository. And we thought about that we don't only configure our Cloud Infrastructure. There we also wanted to create or manage our GitLab organization, for example, who is in which team or what are the dependent information for this specific project as a variable, for example, service accounts or the manager of service accounts. Yes, so a part of that, we want to have that our teams have their own space in the infrastructure so they don't have to configure in the one repository. And yeah, so this means that our teams are still have the chance to be flexible so they can fulfill their needs. Yes, and one important thing every time is that we automate everything. So what does it mean for our structure, how we use GitLab and the Cloud? So we had an organization structure like in every other company. We had a top level, which is our department or the general configuration, for example. And then we have our teams. We have multiple teams. And then each team can have multiple services. And we thought about how we can use or how we can utilize GitLab for that. GitLab got groups, subgroups, and repositories. So in a group can be a subgroup or a repository and a subgroup function same as a group. So we thought, OK, we have a group for the organization, whereas one project, for example, for the general structure. And then under the group, we have subgroups for each team. And each team can have multiple repositories, which means this are the specific configuration for the services. And then we have the Google Cloud Platform world, on the other hand. And there are folders. Folders are the equivalent of the equal thing like a group in GitLab. And a folder can be a folder and a project or a project. And a project is the same like a repository for us. Or yeah, there can be a project where is what you need there. We are connecting the dots there, which service you need and connecting everything that you can run your software there. So we built our organization. Our organization is a folder. But we have for each team a single project, private environments, so a dev environment and a product environment. But this is not interesting here. And yes, and this project is for one team. So every service we want to publish there, they are using this project. So it's there. There are no next level for a folder or something. Yes, each team has a project. There's Kubernetes cluster, for example. It's perfect. So what does it mean for affecting things? We had thought about, or I mentioned before, on the organization level. We have a little project where we have general configuration for infrastructure. And then in the team, we also have a project for the team-specific configuration. And each service still has another configuration space for the specific service. So it's like the hierarchy. So we have an organization level. We have a team level and a service level. And each level benefits from the service above. So the organization hierarchy has data which are inherited to the teams, for the team, team, team, team groups in GitLab, for example. So we used or we shared data from top to down. Yes, so we did that automatically, which means, on the other hand, that we don't have a configuration from a lower level that can be lived upwards. Yeah, because it's technically complicated and you could have side effects that you can don't deploy a team or create a new team when you rely in. Yeah, if you have changes and rely on the dependent service, which is not created, or they can do a lot of bad things when you don't use top-down approach or a straightforward approach, yes. So how we share data across the landscape? We have two different approaches. One is the GitLab CI variables approach. This is done when we have to share data between different levels. So when we are on the organization level and want to share something to the teams, like a service account in the Google, we do it over GitLab CI variables. So we can use it everywhere downside there and have a clear cut between one project and the other project. So it does not affect when something change on the other side, only if we rename the key of the variable. And on the other hand, we have tear from outputs. This is when you have multiple tear from scripts and your users the same state. So when you have multiple jobs, for example, and they rely on data from a previous pipeline job, for example. And then we use tear from outputs because it's normally sharing the same state and we just configure a bridge between these two. Yes. And what does it mean when we use the data for the GitLab CI variables? It's really simple. We simply tear from had the ability to use directly environment variables from the host system when it's using the correct naming convention there. So we don't have to do anything. And when we have tear from outputs, we have data pack there where we have to configure, OK, we are importing from another tear from state. So this is the brain or the brain of tear from where any information is stored. Yes, it's configured there. And you simply ask them in the tear from way. They ask in the tear from way for the data and you can use it more easily. And an advantage of the tear from outputs way is that we can use or we have to configure one data lake or data source. And we have a lot of different or everything which is an output from the previous job. And on the other hand, the disadvantage of GitLab CI, we had to configure every single data source or data information as a single GitLab variable. Yes. So how does it look in practice? We have a heart or we have a heart for our configuration. So it's simply a single repository which prepares our cloud project and the GCP and configure our GitLab projects or groups with permissions. So if you see here on the left side, we have a teams JSON file where store a lot of configuration for our different teams. And yes, the configuration heart or the single project configuring all the base stuff which is needed for the organization, the world organization and creating specific service accounts for each team project and creating the project in the GCP. Yes. And they print a lot of GitLab CI variables because they are all shared for each team and then for each environment, for example. Yes. So we have a single point for the world organization and then we come to our team. Our team has also a project named here the project administration. And there we manage our general stuff for the team like creating a Kafka or specific service accounts for each of our services. And yeah, they are doing their team internal stuff. And this project, for example, plays also GitLab variables for the specific service, but not in the service or in the world repository directly. Yes. They are posting the variables in the group because we don't want to place this directly into a project because we had to go in the project and have to change. Or it's a lot of big management to say, OK, we need this project and the variable is there, there, et cetera, et cetera. It's easier when you put it in the group. But how does it feel to use the service? Or how does it look? In the service, he's mostly he's only consuming the variables and building the stuff needed for the software and the cloud. Yes. And this means here, there are a lot of different variables. And this is only a small part of that because the variables are inherited from the group and then from the organization group and then from the team group. Yeah, this is a lot. And it's sometimes hard to figure out if it's a very, well, yeah, it's a very bare or not. Yes. But when we start with the service, we have a template way. And then we're typing only the service name in. And then we don't have to do anything more because the CI jobs automatically pick the correct variables from themself. So another interesting question is where you have to start the Terraform state. Of course, you could easily use the Terraform state in GitLab, which is really great. But we thought, what if, yeah, we thought it would be fun when you start where you need it. For example, if you configure in GitLab variables, we start in GitLab. And when we use Terraform to create our cloud infrastructure, we start in the cloud. But why? When anything happens in the cloud or is destroyed, it's only done in the cloud. And when we configure our GitLab stuff, it doesn't affect the configuration we did for Terraform configuration we did for GitLab. So it's not so combined. Yes. And what other specifics we face daily doing with our pipelines or how the pipelines look, this is a screenshot from our project administration. This is a team internal setup. And we found out that it's smart or it's a good way to do things in a line that then or, yeah. Some configuration in Terraform depends on other configuration or it's possible that this depends on other configuration. So that means the job has to be run or the job could only run afterwards. And this is because, for example, when you're on the Google Cloud, some APIs take some time to activate. But sometimes Terraform is not working correctly and waiting until the API is finished. So you got an error, OK, the API was not ready. And this is not so cool, but we are facing this issue by running each thing in a separate job. And for example, when it's come to the Terraform state, it's better when we can have a different job for Google Cloud jobs, Terraform expressions, and GitLab expressions. And another interesting thing is that we had split our environments into two different jobs. And for us, it makes sense because we don't want to build a logic into Terraform where we have to loop our two configuration or environments. So we just said, OK, we have a variable for the environment we are currently building. And this is set by the GitLab CI job. But in the setup we have, the destroying of the infrastructure could be take some time. It's a little bit tricky because you have to start in the end to destroy everything. This is a little bit complicated. Yes. And we used the template thing for our jobs because we have the logic that we have for each environment a different job. And we are facing it's really simple. So we have a base job where the common things are stored like rules and emerge in the common script. And then we have our jobs for the specific environment, like for the dev environment. And this is extending simply the base template and set the correct variables. For example, the dev environment. Here, for example, is a correct naming pattern for setting, using environment variables for Terraform. Yes. And then we have here an interesting thing. We say we have a state bucket or we have a variable. But in the end we have a variable specific for the dev stage. But the Terraform does not know it, or doesn't know it, that it's a specific variable only for dev. Yeah. So we have less complexity in Terraform. And we have a special step for the Google application credentials. So what do I say in the last 25 minutes? So a little bit of a wrap up. So the key things you have to take with you that is really cool and helpful to automate your infrastructure and go the excess step and use it not only for the infrastructure but additionally also for example, your building tool or something. And yeah, maybe try to map your organization structure into your cloud and GitLab setup. This was really helpful for us. And worked really well at our company and here. And a little tip, it's not so important but you can still save the Terraform state where you use it. So in GitLab, when you use it for GitLab and Google when you're using for the Google configuration. And in the end, don't forget the flexibility for the teams or in general. Yes. And in the end, if you want to connect with me, there are my socials. And if you are interested in further talks, join our GitLab Meetup Group. We are also doing virtual Meetups. So I have to thank you for your attention and it would be really cool if you share some insights how you use it Terraform and how do you think about our story. Yes. Have a good night or a good day or a good morning. Bye-bye.