 with Tio. Welcome new people. Give him some applause. So hi everyone. As a disclosure, my excuse is for the nice Chrome bar. I just started with this reveal.js thingy and I can't get it to work properly. So whatever. So we're going to talk about automation and as a short presentation. So I'm Tio. I work for Ableton. And basically I tend to describe myself as someone who whispers to computers. I'm more of a hacker than a proper scientific engineer. So working with computers is pretty cool, right? But making music is definitely way cooler. So why don't we basically achieve the dreams of our grandfathers and so on and make computers work for us so we have more time to make music? So if you don't know Ableton, we are a music company. We basically build software and hardware that enable music makers from building a track to playing it live. The company was founded 16 years ago. So we're not exactly a startup anymore. Although we're still a rather mid-sized company. We're Berlin based and the company was founded in Berlin by Berliners and our founders are electronic musicians. Well, not anymore, but there have been electronic musicians for a while. Our most known and used product so far is live. Live is what we call a dough. So basically it's a software that allows you to build a track and to perform it live. It's based on two views, the one you see here with that we call the session view where you basically create clips. So it's small loops that you can play and you can arrange as you want. And right now in the, let's call it the mainstream music industry. It's one of the most used software. We also have push. So basically push is some sort of fancy MIDI controller. So if you're not familiar with music, it's basically an interface that will send signals, notes, like a small protocol that will tell the computer what kind of sound to play at what pitch and when to play it. And finally we recently released link and link is an iOS framework that synchronizes devices and more specifically synchronizes their bit time. So for example, if you're making music on your Mac and someone comes in with an iPad, they can connect to each other and you can be sure that they will play in time. So if someone is starting drum bit, for example, and you had a baseline on top of it, they will play together on time. There won't be any phase or any weird stuff like that. And we basically provided as an iOS framework to developers and they implemented in their iPad apps. So these projects are big and complex. When I say these projects, I tend to include live push and link all into the same bucket because basically there are all part of one big ecosystem. And if you think of live more specifically, it's a huge monolithic C++ code base. So we basically have, I think, millions of lines of code and thousands of files. They involve Qt, which is a huge library for basically building GUI. We include Boost too, which is a big C++ library that does all sorts of fancy stuff. And we have Python in the build system. And it's also included in push. So for the communication layer and for scripting purpose. Now, what's interesting in this is that I was working in startups before. I was doing backend and system tasks, basically. And I could use all sorts of fancy tools like Docker and all this stuff. I could deploy stuff on AWS. It was not a problem. And for one simple reason, it was that we were using Linux. And once you get out of this ecosystem, you realize that if you have to build something against OSX, Windows or iOS, well, most of these tools become suddenly unavailable. But still, we're not doing rocket science. But what we call DSP, digital signal processing, is serious business. It's a very complex topic. And most users rely on our software's reliability. So basically, if you're a Skrillex and you're playing in front of, say, 40,000 people, you don't want life to crash in the middle of the gig, right? Same if you're playing at the Bear Game in Berlin during all night. You don't want it to just stop or start doing shit in the middle of your set. So although we don't have access to all these fancy tools to help us build a better software faster and in a more secure way, we have to find a way to reach that level of quality. The live ecosystem involves long build times. So basically, I think we have the same kind of problematics as projects like Chromium, for example. We are building a huge software that takes a very, very long time to build sometimes, that involves a very complex and long workflow. Like, from a feature ID to its implementation, it can take up to months, sorry. And we also have a culture of perfection. So as I said, when we put a release out there, we don't want users to have the software crashing in the middle of their set. Also, something very specific to this class of software is that most people that work in studios, for example, don't have the software on the internet. So they won't update their software all the time. They will update it from time to time. But we also are aware of users who are running very old version of lives and are not wishing to upgrade it. So all of this leads to the need for automation. So we have to automate as many things as we can whenever it comes to build it or whenever it comes to release it. So sure, you could tell me, we could automate everything with scripts, like with bash scripts, Python fabric, you name it. Actually, we use basic scripts, basic, not basic, but scripts for the build system already, which are in Python. But if you ask me, scripts are very hard to keep organized, to maintain sometime, and I'm thinking more of bash scripts here. And I'm not even talking about extending. And yeah, let's just be honest, like scripts are cool, but scripts are hard and very often cryptic. I wouldn't read some of my scripts from two years ago. We could also use proper deployment systems. Many of you probably already know of Puppet or Chef. I used them in the past to deploy from midsize to big infrastructures. And I really enjoyed them for many different reasons. But in general, I find that this tools always involve the concept of nodes, of leader, follower. And they're not really suitable with something I really, really like when it comes to deployment, which is masterless systems. I don't want to have to think of, oh, do I have my client installed on this server and where is my master and can they talk to each other and all this thing. When I want to deploy or automate something, I want it to just work. And also something that's a bit too much to my taste is that you have a ton of useless abstractions. And here, I think more of Puppet rather than Chef. But I don't want to have to learn a new language. I don't want to have to think of where do I put all these columns and all this stuff. So basically, the systems are great for certain class of systems. But when you think of Ableton and what we're doing, it doesn't really fit. So that's why I introduced Ansible when I arrived at Ableton. If you're not familiar with Ansible, it's a deployment system just like Puppet or Chef. Or you have verse two that I would leave. And Ansible is basically declarative. So it's based on the YAML syntax. Everything is written in YAML at all times apart from host declaration that you do in any files, but that's really a minority of the files. And you just state what you want to do. And if you look at this example, you can see that you have a name for each task. And then you have a module. So for example, the first one is Ombru. And then you have a copy. And then another copy. And then you say, oh, I want a file being created. So whenever you open these files, you just you don't really need to make to make any specific effort to understand what's happening there. And if you compare that to a bash script, I can tell you it will take you probably five minutes in the most complex cases to understand what this is about. Where it has a bash script with all this dash, E, blah, blah, blah everywhere. And you'd never really know where to put the square brackets. Just nah, let it go. Another thing that's great about it is that it's sequential. So everything will be executed in order. And it will only rely on the structure of your project. So Ansible enforces a very strict structure where you define roles and you define variables. And they all work in a scope system. And once you get familiar with it, it's very easy to understand what's happening and when there's an error where it's actually happening and why. Because you just have to follow the white rabbit. Something I really like about it too is that Ansible has few but very efficient abstractions. I can name a few of them. The most common is register. So register is simply declaring a variable. So you take a task. So all the steps that you see are called tasks. And when they're executed, they return a Python JSON object. And this object is basically describing the outcome of this task. And in this case, you can say, well, I want this JSON object to be called, in the case of the first task, foo exits. It exists, sorry. And you can later on use it. So for example, if you look at the second task, you have when foo exists is false. So when is another keyword that basically helps you defining conditions? You have with items that you see in the last one. So I can create a variable somewhere in the variable files. And I can call it my precious files, of course. And then if I use with items with my variable, I can just pass around this item thingy to my expanded strings. And that's another really great thing about it, is that all of these values could basically be strings. And they can all be interpolated with the ginger syntax. So you get a lot of flexibility. And it's still at a very low level because you only have variables, conditions, and loops, basically. That's the only thing you get. But you can mess around a bit with it to gain some flexibility with some templating, sorry. And as I said, everything there relies on the project structure. And you can modify it a bit, but canonically, you always have only one way to organize your project, to organize your roles, and to organize your playbook. So the project is the top level structure. And it's the first example. And you can see that you have, it's not all the things you can put in there, but the most important are there. You have the group bars. So in Ansible, you define groups of hosts. So for example, you have, I don't know, Windows servers or web servers or DB servers, you're going to create groups. And you can attach bars to these groups directly in these files. You have the inventory where you're going to reference all the hosts that you want to talk to. You have the roles. And roles are basically kind of modules. So you're going to define a role for Nginx, which will be in charge of deploying an Nginx server. You can create a role for installing Docker, for installing Ombrew, all this kind of stuff. And something very cool to us is the vault. So I will talk about it later. But Ansible is able to encrypt all your secrets. And you can access them later on in your tasks seamlessly. In these roles, you have files, templates, tasks, and lures, defaults, meta, and variables. Basically, files and templates are basically the same thing. So you locally create files or ginger files. And later on, you can say, well, I want it being created and replace this variable with this value. Tasks are what I was showing before and that you can see underneath. It's basically all the steps that the system has to go through to generate the final system state that you want. And variables are like, well, I want to match variables to values. And another interesting thing are the meta. So it's basically for including things. So if you say that Docker relies on some fancy Linux kernel or file system that you have created, you can do it there. And all of this is aggregated in small files that we call playbooks. And a playbook is basically binding all of this stuff together. So you give it a group of hosts or a single host that you want to run your project onto. You tell it what kind of connection it should use, if it should use facts which are kind of reserved variables. You can provide it var files. So in this case, that's a vault file. So you can say, well, I have my secrets vault, which is encrypted in AS 256. You can define your own vars. And finally, you can whatever call your roles. So in my case, it would be OSX Jenkins slave, or you can also call a task directly. Another really awesome thing about Ansible, and that's mainly I think why I personally prefer it is that it works on over SSH and win arm. Everything you do is done over SSH or win arm. And what that basically means is that Ansible is able to deploy as many servers as you want in parallel on a normal laptop, as long as you have an SSH connection. Win arm is a bit more tricky because you have to make a bit of setup on the windows, on the windows side. But still, it's pretty trivial. As I mentioned before, you have a vault that comes in with Ansible. Something that's great with that is that it's yet another YAML file. You basically put variables in there and then you say, hey, Ansible vault, this is my file. Please encrypt it. I don't want people to see my GitHub, my token, my AWS token on GitHub. And then all you have to do is just when you run Ansible, you just say, hey, well, this is my vault and this is my password and deal with it. You have a galaxy, which is a pretty fancy name if you ask me, where you can upload your roles. So say you have created a specific role for whatever, you can upload it and share it there. And when you want to make something very common, like I don't know, install an open SSH server, you can be sure that someone will have made it there. But I will talk about it a bit more detail later on. Ansible is straightforward if you compare it to other alternatives. Now, I'd like to go through some use cases that we run through because we don't really have the same use cases as most companies that are there, I think. And I selected like free use case that are pretty representative of what you can do with Ansible. So basically, you can use it to automate your computer. And what I mean by that is that you can just deploy your workstation. You won't ever need another bash script to do that anymore. And deal with, oh, am I on OS X? Am I on Linux or whatever? And use this weird syntax. The last bash script you will use is the one to automatically install Ansible on your machine. And that's what they call, that's what brew calls tapping, but basically telling, for example, on brew, which is the package manager, one of the package manager for OS X, where to go find softwares. And it's really as simple as telling Ansible in an inventory file. Well, I want to talk to local host and just use the local connection to do that. And then you have two different commands there. You have the Ansible playbook that runs the playbook file I showed you, where you can call roles, call tasks and variable files and all this stuff. But you can also use the Ansible, just Ansible commands and you can pass it host and then give it just a command to run on the command line directly. And what you can do with that is that if you're on a Mac, for example, you can automate the installation of system packages and your dev tools and so on with included on brew package. So as you can see, you define a variable that's a hash, a hash map in which you say I want mo can we get and whatever. And then later on, you can just say, well, I want to install all the all the elements in on brew packages. And I want the latest version that same for your desktop app, like you want to have Spotify virtual box, whatever you like installed, you can do it too with the other module that's called cask, because of on brew cask, which is a subset of on brew. And something that's worth mentioning, though, is that although you have the interface for that directly in Ansible, as for a lot of services like console, for example, you generally have to install on brew by yourself. But Ansible can do it for you. You can write a role to install on brew. You can use it to set up your dev environment. So you can update your system settings. You can install, pull in install your dot files. And you can also pull some favorite git repos. You could also trigger some bills, some anything you can think that's really the great thing about it. So to sum it up, it's like you do the work once you keep it up to date, and you will never ever lose hours reinstalling your workstation. And I also tend to think that it's pretty great if you're looking into a good solution to onboard some engineers. Like in our case, we could perfectly say that we install all the Xcode, Tushin, and all the Cmake stuff, all the Jeep stuff, directly with just one command that that's based on an Ansible playbook that we would keep up to date. So you don't really need shell scripts anymore to do this stuff. And actually, something that's really awesome about it is that it works on most platforms. So whether you're running Linux, OS X, or Windows, it will work the same way. You might have to use some dedicated modules for Windows, but that's basically it. And you might have to set up a bit, but that's Windows writer used to it. Now, something we do too is to deploy, well, nothing fancy, but small apps in the clouds. I can think of two very simple examples. In our offices, we have a gateway. So we have a big firewall that's keeping people from going on websites, they shouldn't go home, right? So we deploy a gateway, we deploy some small dashboards to analyze what our users are doing with the software, how reliable it is. And so the process in this case is still pretty straightforward. You create a Docker image of your app and Ansible can do that for you. You just give it the path of your app where your Docker file is. You tell them, well, I want this name for my image, the repository you want to push it to. You can also tell it if you want to log to Docker Hub or to your private registry. Then you can tell Ansible, well, okay, that's cool. I have a Docker container now, but I want to deploy it. So same as with the Docker image, you would do it in a playbook. When you look at the deployment, what often happens is that in the playbook, you will first set up your environment. And what I call by environment here is more your infrastructure actually. So you would create your VM. And here, that's exactly what it's doing. So this local action is kind of a trick. It's not very obvious, which is not, we're not really used to that in Ansible, non-obvious things. But, well, it's that way. And you can describe, that's the great thing about it. You just describe what you want. You don't really need any logic. You don't really need to think of, oh, so what should happen and so on. You just think of your steps and describe them. And that's basically it. But you could tell me, yeah, okay, but how do I tell Ansible that, well, this VM exists now, and I want to deploy stuff on it. I want to put my Docker image, my Docker container there. Well, that's another super cool thing about it. Remember, I told you about inventories where you list all the hosts you want. Well, they can be dynamic too. So you can perfectly give it a Python script. And this Python script is going to return a JSON object that's listing all the hosts that you have created on AWS. And with a few tags, you can very rapidly automate the deployment. So you create your VM, you tag it with a certain tag. And then this inventory will ensure that you have a host that's matching this tag. Once you have that, that's pretty trivial. You just bring your VM in a suitable state. So if I take the example of control, which is a small data dashboard, what we do is that we just say, well, I want all the hosts in my security group. That's another way of doing it. Instead of naming your instances, you can just refer to a security group. I want all the slides to be loaded. For example, AWS is going to be all my secrets and bear from the vault. And then, sequentially, it's going to run for all the roles. So base will create some users and install some keys, SSH server is going to create, install open SSH, Docker is going to install Docker and GenX is going to install and configure GenX. And then I'm going to deploy my app. And how I do that is just you pull your Docker image. And you don't have to do this, but it's a pretty nice way of doing it. You create a systemd service to start this container. And you just start it. Because I will talk about it a bit later. But in our case, we use some sort of stateless machines. So when we start something, when we create a new server, it's already in a state where you just want to have to start a few services. You don't want to deploy anything anymore on it. So basically, automation is like playing a roguelike game, right? You just die and retry until it works. I'm not saying Ansible is going to solve all your problems or it's going to make your life super easier that would be lying. But it will make the task of finding out how you should deploy and install things and figure out what's not working much easier as with other systems. And now the important bit is the dark corners. I'm currently exploring at Hamilton. So to build the software, we use Jenkins. In the setup where we have a Linux master and we have drones veteran OS X and drones veteran windows. Jenkins is kind of the nervous system of our build process. But at the moment, it's pretty old and some people are managing it by hand. And what we're doing with that is that we're deploying Jenkins on an internal vSphere based data center. So vSphere is basically AWS for your own data center. It's VMware solution. And we run it on Mac pros, Mac mini Windows servers, we have a bunch of data stores, and we have subnetworks to isolate everything. The reason for that is that when we build live, we need to build it on a Mac. We can't say, Well, I'm going to cross compile it for Mac or I'm going to cross compile it for Windows. That's not possible for us. We really have to build it on the proper architecture to make sure that everything is in order. And the great thing about that is that the latest version of Ansible that came out, I think in January, brought a lot of improvements for Windows, for VMware environments, and even for Docker. But beware because it's still a bit buggy and the transition was kind of tough. And as I said before, automation here happens in two stages. So what we want is to create a Linux master that's constantly replicated. So it can just fall and it's fine. And then we create all sorts of drones. We don't like to call them slaves. We call them drones. And this happens in two stage, first provisioning, which basically is trying to create a reproducible machine state, something we can always start on. Like you press start, you know, it's going to work and it's going to be in the state you want it to be. And then just the deployment, which is basically bringing these templates to live. And in general, the only thing we have to do in the case of Linux master, for example, is starting a few containers. So if we were to sum that up, we have templates that we provision and we deploy this template and give them name with a convention to remember basically what's there and what's running. For the provisioning, we create a VM, we bring it in a good enough state with Ansible. We start a VM still with Ansible, and we create a template out of it, still with Ansible. And our hierarchy is pretty straightforward. We have a root template, which is the base OS. It's for security purpose. I can't really mention, I can't really get into detail, but we have to do this step. Then we have kickstart templates. So we configure everything that is needed to build. So we're going to install Xcode developer tools, CMake, whatever is needed. And the last stage is we create deployable templates, which are following naming convention, role OS date. And that's it. You just have to deploy them all. Now, a bit of sugar on top of a cake. We use console too. So if you don't know what this is, it's a service discovery system that's installed on our master and on all our slaves. And whenever we bring this VMs to life based on this template, they declare themself. And now all the system, every pieces of the system know each other. So if you want to automate things on Jenkins side, or if you want to make some scripting under the hoods to maintain some certain states, you can perfectly do that because they know each other. Now, there's still some stones left in our shoes. Very honestly, Ansible 2.0 was a disaster. There were a lot of bugs. There were a lot of regressions. They changed a lot of things that were working and ended up not working anymore. This fear is really nice. That was my first experience with it. I'm an AWS guy. I started my career. I always worked with the cloud. So it's pretty nice, but don't ask too many questions, right? Because it's a closed source project. And Windows support is still rough around the edges. It's a bit untested. What happened to me quite often is I'm like, hey, this doesn't work. I'm going to fix it. And the guy's just like, yeah, sure, it works on my machine. And last but not least, Jenkins is kind of hard to automate. And now we get to the core of what I wanted to bring you guys. Ansible is the perfect project if you want to get into contributing Python code. If you're a beginner or experienced, it's a very good entry point. So Ansible is a widely used project. It's very, very active. And I think it's a super good entry point if you're looking to commit some code in a project. And when I'm in code, I insist on codes, like not documentation or anything, because the documentation is really great already. But you can just dig into it. Because all along the journey, you will use a lot of third-party modules and snippets. As I wrote it down, don't be a hero. Just do it. There are a lot of things that are already existing that people made and problems that people solved for you. And all these modules and Ansible itself, once in a while, will have what I call less frequented corners, things that people don't really use. Like if you use the vSphere module or everything that's related to VMware, it happens quite often that something is there, it's supposed to work, but it doesn't really work. But that's actually the great thing about it. Because now you can just jump into it. And you can try to fix it yourself. And instead of being painful, it's actually very easy and rewarding to dive into it, fix it, adapt it, or even add something to it. And I really insist on that. It's really simple. For example, the Ombru installer role that we use to make sure that when you install a Mac or when you deploy an OSX drone, it gets Ombru so it's able to install the common packages. Well, this thing was mostly working except in our network setup, because we have a firewall and we open ports on demand and so on. And I was like, well, okay, what's not working? And you just go take a look at the role and you're like, okay, so this good thing, it doesn't work for whatever reason, I don't really care. It's not my problem. I'm going to just go take a look at what the Ombru script is doing for installing and trying to replicate it into the role. So you can just do that for any role you find. If you've built your own role for your own case, but you're like, well, maybe that could be useful to someone else. Well, you can just publish it. Go ahead, does it? Because you can be sure that it will solve someone else's problem. And for example, well, WinRM is okay. But Microsoft worked hard to provide open SSH support to Windows. So we basically created a role that will install open SSH on Windows for you. And it will be as simple as including this role in your playbook. And to get to the core of it, most modules are just simple Python modules. They return for managers and objects, and they follow a very strict convention. So whatever you do, it's all about trying to document first what you want you are trying to do, and then building a Python module. But it could be anything else. If you're into, I don't know, Perl, there are still people who like Perl. You can do it. You can just create a Perl script that will return JSON. And here you can see that this is a Darwin service. So that's basically a launch team implementation that we worked on. And we use OSX a lot in the server cluster. And while starting services on OSX works with launch team, Ansible was not supporting it. So I was like, yeah, well, sure, let's just give it a shot and try to do it myself. It was crazy simple. You just try to find a module that's called, in this case, it's service. You try to find it in the module score. So there are modules score and modules extra that are out of the core of Ansible. And you just go there and go in system and like, okay, so how do they manage system D or how do they manage upstart? Oh, okay, that's there. So maybe I can just copy this class, try to find the method, the key methods, what they're doing, and just put my own code there. And in one or two days, maybe two, let's be honest, launched it was there. And yeah, there was basically no pain. The only pain was this launch CTL command that's weird. Also, although the code and the feature were accepted. This change happened during a redesign of the whole system interface in Ansible. And that's one of the black dots. So just know that it might happen if you go that way. That sometimes the community is like, yeah, sure, that's great. We want it. That's super code. But we are changing stuff right now. And it might happen that it just takes a while for them to just come back to you. So just bump them bump once in a while and say, Hey, what about my feature? The community is wide. There are tons of people super reactive. It's very inclusive. I've never seen a sexist comments or racist comments. People are always trying to help. They're positive. There's no like, Oh, no, sorry, that's shit. We don't want to see that. Just keep it for yourself. It's fine. It's really like trying to improve. And you have core contributors on top that sometimes comes in and say, Okay, maybe this is not fitting with what we want for the software or something like that. But it's always constructive. And as I mentioned, it's a little bit so far. Be aware that when you create a module for Ansible, debugging can be quite tricky. There are a few things to know for that. You can use PDB basically, but know that you will have to force Ansible to use only one fork. Because if it starts working everywhere, it will be really hard to figure out who is doing what and where's the error at. And it also has a ton of logging, but you have to give it some time, you really have to think to kind of try to figure out, Okay, so this is about that. And so it just takes some time, but you will get it eventually. And yes, also something when you contribute to modules, like if you take the example of I contributed a bit to some windows modules, for example, well, it happens that the person that created it is not as involved as before, or it's in a transition state. And you might not get as good as good feedback as you would get on the Docker module, for example. And that's basically it. Do you have any questions, ideas or rents? I love rents. Hi, first, yes or no question. Do you use Ansible Tower? We thought of it. But I think the ratio price, what it brings my follow up question is then how do you ensure the state of your infrastructure? Do you have a central place where you run playbooks or every operations guy runs them from off their laptops? Or how do you do that? So yeah, maybe it was not clear enough. But we don't really rely on Ansible for that. Ansible is really a tool for us that runs some steps through steps. And then we flash things. So we basically have templates. And we build our infrastructure states with templates on top of each other. We just enter it all the time. And Ansible is just involved in creating the states all along the way. So we don't really need to figure out like, what's the state of what? It's already there. We know what's the state of what we have root, kick start, and then we have our deployment images. So we don't really need this sequential states. No, I'm not sure if my question is answered already. In our company, we have lots of small modules. And then of course, the big environments where most of those modules are deployed. And well, one big Ansible playbook somewhere which deploys everything, I think is not a good idea. So each module has its own Ansible playbook. But how do you share, for example, the what is it called roles? Like, like where to deploy to. So if you add a new sandbox, if you add new servers, then all modules and there can be hundreds, need to be updated in their playbooks. How do you solve that? What do you mean with updating roles? Okay, if I create a new module, then then I create also a new playbook. Yeah. But if I create a new environment, then all the playbooks need to know about that new environment. What do you call environment? You mean like a group of? Yeah, well, servers to deploy to. Well, you use the groups in the invent, inventory for that, right? Yeah, but how do do the many, many, many playbooks and the many modules share the same inventory? I'm not sure I understand the question. Well, I'm going to talk about it. Yeah. Sure. So focused on built system in interaction with Python, but from what I know, you also use Python as embedded interpreter in life to, to provide ability for driver producers to implement their hardware, etc. And do you actively develop this system for embedding scripts or rather, you go away from it? So basically, I think my colleague Alan was there, can talk about it. So we embed, we embed Python, but our, the project architecture is made in such a way that it's really monolithic and everything is there. So everything is in one place. So we don't really manage that. When we manage it, we have to do it by hand. If you want to update it, you have to update it by hand. Ansible really doesn't touch anything that's in the life code base, for example. It's something that it can do, though, is that we can install a bunch of tools that are needed to test this code base, but we never go in there to try stuff. It's time for one more question. Hi, thanks for the talk. Very, very simple one. In the structure that you defined in the beginning of the talk with the folders, we are used to, to have a sub folder under roles for each of our roles. And inside that, so folder, we have the files, templates, everything. Okay. Is this the normal approach of are we doing it wrong? I'm not sure. You mean you have roles, source? Roles, then every sub, every role has a sub folder. Okay. So enjoy next, Docker and so on. Exactly. Okay. And I don't know if that's right or not. Yeah, that's basically the way you do it. Well, where do you store your, your playbooks then? Your playbooks are, that's a good question. Your playbooks stay in the top level. Okay. So basically, you keep the playbook in top level, the YAML files. So whenever you come into your directory, it's already there. That's something I didn't mention, but we have one playbook per class of servers. So for example, we have a playbook provision linux master deploy linux master, we're going to have provision or six drone deploy with six drone. And then we tend to create directory for the inventory, because we have dynamic inventories there and so on. We have a bunch of them. And yeah, like you said, the roles are just in roles. And then you put them all in there. And that's really the canonical way. Any other like super fast question? Or rents? Well, another quick one. Have you configured Docker using Ansible? Have I configured Docker? Docker. Yeah, basically, we have to do it all the time. Have you configured Docker for user access? Not root access? No, we don't need that. Okay, sorry. We do it for DNS and how it, how it talks to the system. But that's basically it. Okay. Thank you all for coming. Thanks, Theo.