 So we started right? Actually, it's hard to talk with audience when you cannot see audience, but well, I will try I guess I'm doing it first time with like When audience inside is less than audience outside But anyway, so I'm Yuri. I am from SP digital. This is part of SP group We're doing some digitalization things for SP group and I will speak today about our transition our migration from salt stack to puppet and why we did this What we learned that why? Well, what to do what do not do and things like that and then I will say a bit about what's Next step after we will finish this migration So in first words migration is not finished yet. It's in progress. So We are still like maybe 40% done but anyway So What's the scale of our organization? We have about hundred developers and We have a lot of products. So our developers split into product teams and So We as a info team actually doing Kind of platform for our developers for product teams So this is usually persistence layer databases all the stuff which is common for everyone and And Yeah, we are just six persons and we are running right now about 200 VMs in Azure cloud We have on-prem part as well, but I will not talk about this part here in this talk because it's it's complicated So Yeah, usual question when you see the topic who what's wrong with salt I guess someone using salt here No Someone using puppet here someone monitoring YouTube channel for comments Okay, so let me know if there will be any question in the flow Okay, so what's wrong with salt first thing actually when I joined the company I Started to use salt first time. I had some background with Ansible and puppet already and it was like first meeting with salt So it was a bit hard for me like I hit heart into this So main When I will explain what's wrong with salt or why why I prefer puppet I will explain it from point of view of team So this is team white Meaning so why it's good for team or why it's bad for team because if you are just alone Doing things you will just use what you like, but if we are working in a team you need to take care about the team so yeah First thing is I feel it inconsistent. It doesn't mean it's inconsistent in terms of managing virtual machines so if it's applied state or applied some Something on your show machine it will be applied there. So it's consistent, but I'm talking about different inconsistency when you know when you Doing similar things in different modules at different states and they Behaved in a different way and they have different names so in some state you you can see like The names like install it and or present in another model you can see similar things name it differently and This is like you so you need to keep a lot of Names a lot of context in your head when you reach in something and when we debugging it so then It's actually Error prone so For example, you can do mistype like do a typo in and write require instead of requires and You will never know about that The only way to know that is apply and see that the actual Steps sold it. They are different one from what you expected and There is no way to see that and actually yeah, I tap it a bit earlier, but anyway, so it's hard to debug because Diagnostic messages are about nothing like hey, there is some error. I cannot compile Dataset for example, I So it's really hard to understand what happens and where is the error It's even worse when you write a lot just before and then you're trying to apply it first time and you see that Hey, there is error in somewhere Then it's YAML. Well YAML is good and the bet at the same time It's good because it's easy to write and More or less easy to read Yeah, it actually is more or less easy to write as well because you can hit anything like no Which is false and you never know about that until you hit into that But when you're reading YAML every day like hundred Few hundred lines of YAML every day and debugging it. So even you're using it as a Language like programming language. I feel it's wrong. So it's not programming language. It's it's declarative Configuration language is different. So I guess YAML is a bit overhyped and people already started to mention that so next This is actually most critical part I'd say Because I am okay with ginger in Ansible because they did it in more or less okay way But sold it in a really wrong way. They apply it YAML They apply a ginger on top of YAML. So they're generating YAML by ginger Actually and so if you need some iterations like you need for loop or some to do something or you need some conditions you need to write ginger and This ginger will like between those ginger block you will write YAML It's okay if you're writing just I don't know a single state for hundred lines But if you have thousand clients and complex logic there I'd say good luck with debugging that Because it's it's hella hard So and next which is actually Because of previous one so you cannot lint it because to lint it You need to do parsing of ginger. So you need to write core of salt to do the lint Which is again, it's insane Firstly Initially, I wrote this because this this was my impression So I spent few days trying to find some unit testing for salt I didn't found But recently I just mentioned there is salt check It just happens incidentally. I saw it somewhere and then I realized it. It was three years ago Already, but nobody know about that. It seems or I didn't found it in Google at least I Checked get history yesterday, and yeah, it's from 2017 there. I don't know why why it's was so Hard to find it Okay, so this is my impression from salt why I don't like it from team view perspective So why puppet again from team view perspective? Yeah, this is actually what what I will talk about today, but seriously First of This is actually same with salt, right salt heif agent puppet heif agent. So it's more or less same I will not keep here more time. So then It use the main specific language. So this language actually specially designed for task area So it's not YAML. So you can write meaningful constructions meaningful things in meaningful language, and it's declarative And actually I did mention it here. It's uh, I Can say it's functional language. So it's immutable. You cannot change things incidentally Which is really good. So yeah, you need to take care about that You need to write a maybe a bit more, but you cannot just accidentally change some parameter somewhere and it will affect everything so then There is built-in support for syntax check. There is support for LinkedIn. There is support for Style checking so you can keep your repository your puppet code in well In good form in good condition Which is really good when you're working in a team because team usually have different like every teammate has different vision of style and Whatever else. So you need to somehow take care about that with puppet You can just put LinkedIn with correction right in your OECD pipeline and you will have no issues with that Then you can use a spec the usual tool for Ruby world and people usually know a bit at least about that and it's really powerful and Then you can use a respect which is really similar to respect to do a certain tests So this world is really really good tied together Then there is Hyera, which is so if you're not familiar with puppet I can say this is kind of Mine in salt or this is kind of variables. Actually, I almost forgot. Yes, but yes, it was variables so single database of say things which you will apply to your actual code actual statements and this is hierarchical database I will spend some time later on this and Good to think there is parameters out to look up, but I will explain it a bit later as well and then It's written in Ruby So you can use anything which is done in Ruby world The whole Ruby ecosystem, which is actually really great. I entered that world really late Just maybe a year or two ago and I still think it's really nice because it's it's very very well designed So next question is why we decided to go with open source puppet Why we didn't get puppet enterprise that good question and The answer is actually simple because we can I Seen no real reason. Well, I'm okay to pay money for good thing. It's it's okay But I don't like to pay money for things which actually limiting you so and I don't like having a graphical user interface Because this is actually making you well usually not you but someone might decide to okay I have graphical interface so I can just do anything there and Nobody know what actually happens after like they changed some parameters and there is no code behind that So I'd prefer to put everything into a git repo and then I have I have audits because I Can see transactions there. I have everything there. I have visibility So for me as a for developer for old school sees admin even it's still better. I don't like changing things directly So that's why We did that But maybe we'll change our me Our decision later, but because of puppet remediate or things like that or puppet CIS compliance So it still can be changed Yeah, so but Usually people complaints when you're talking about puppet People will say hey, it's hard to use. It's hard to learn. It's complicated. I don't like it I will just run Ansible. I write 100 lines of YAML and it's okay Yeah, if you manage it, I don't know 10 virtual machines and you're just doing it once like after deployment Right. Yeah, it's okay. Just few sensible just our puppet bolt. They doing similar things But if you manage in 100 VMs or more or thousand servers like with it in Lazada in pre Ali Baba Lazada You will see that Appliance states every time from set of servers like that pushing model Actually do not work well in that in that situation because every time you will see that this ever cannot be connected via SSH Because of network drop or something happens or something else or you will forget to apply the changes if you're doing it manually so you changed your states and apply it to subset of servers which you actually For which those changes was intended and Then you forgot about the rest and then you change something else Then you go to new server applied there and see it doesn't work anymore So I prefer if you just push it to get and then magic happens and your changes are applied across the fleet So you don't need to care about every single server Yeah, and yes about second case I'm actually pretty agree with that development is getting slower because you start into right unit tests This is the main reason of that I'm spending maybe 50 50 percent of times like reading manifest reading I'm not doing the test drive and development I'm usually reading code and before then reading test for it then during the reading test you realize that well I Did something wrong and you change in code you update in test So it does duration actually take time, but after that Yeah, it's saying 90% your code will work more or less okay So you will not hit into big issues at least you check in the logic before it hit production So you're not debugging it in production like it doing like you're doing it with salt or encebel maybe But yeah, you can write unit test for encebel as well And now I can say that you can write it for salt. So please do if you are using it So I don't know maybe audience have different have heard something else about puppet Why it's hard to use or why don't they don't like it? So I'd like to know about that so I can maybe Mention something about that during my talk so Please gather some Something if people will mention So we have an invitation from the speaker for Their own experiences with puppets specifically what's good or you're asking about puppet Yeah, why why people do not choose pop or why they don't like puppet maybe because I'm a bit biased I really like it because I saw really nice installation of puppet in Lazada So I understand how it works and why it's a good for big team for big distributed team even So well anyway, so if something happens, let me know Okay, I will continue them So now I will start talking about puppet and The main slide of this talk I'd say puppet is about the state this This is all you should know before you starting to use puppet actually so when you write your code and Some manifest so called it manifest it will push it to so then you need to deploy it to puppet master and puppet master will compile it So yeah, it's compilation. It's not like Interpreting this step-by-step so it will compile it into catalog Which is entity holding all the States so this is definition of desired state to which you want to bring your system and Then when agent come from not agent asking for catalog for this node and Then apply the changes according to the catalog for this node so agent actually executing some Ruby providers Passing them catalog and those Ruby providers change your your system to be in this state. So you don't need to Yeah So it's not sequence of steps how to bring system to this step so to this state It's actually it is the state So you don't need to care about steps how to bring it like Inensible or so to usually you can see that people are writing steps like download this file Then execute some command then do this do that install this package. So this You can consider this as state as well as long as you are caring about Immutability so when you so if you apply to gain It will be in the same state here. You don't need to care about that more or less Well, you can hack it but usually you don't need So you define in the state from the beginning Okay So but this actually give you some there are some drawbacks this So you cannot easily Rename or move file if you are not managing it So if you want to change to rename some file on some virtual machine, you cannot do that from your puppet code Because there is no state for this. It's not state is change its transition And then you cannot read file from Agent from node and then make decision on your in your court on puppet server because your puppet server have no idea Well, it have idea, but I will speak about that a bit later But you cannot just read file from that then You cannot just execute some binary and do something depending on on the result of this that then binary call When it happens or your agent because again your master your puppet master have no idea about what happens on agent It is different way But you can see stars here, which actually means that Well, there are some ways to do thing and the easiest way is to use facts So if you want to read file you can create Fact which containing contents of this file, but it may hit you hard after so be careful about that then Again, this means that you need to care about things like when you're changing state when you're doing transition of state Like you cannot just remove rename file So if you had some file before and then you decided to rename it You need to remove old one and then you need to create new one So it can be just next line in your manifest But anyway, you need to take care about previous file Nobody will remove it for you Yeah, and usually after you remove something and you think okay, it's already applied, right? I can remove it and Doesn't mean that it's already applied because some virtual machine may be down in that during that period or they may maybe some work drop in so some Catalog maybe not appears yet on some subset of your servers. So in Lazada, we decided to keep it for one month in speed digital I'd say well we have small fleet so for us I guess one day or seven days should be okay because we are running puppet every 13 minutes. I guess nothing should stay down for more than one one day or Seven days at least then Yeah, this is another thing which Which good to know before you're starting to do things So if you need to do changes on different nodes in specific order, you cannot do it easily with puppet Because usually puppets is puppet agent is running once per Period like once per 30 minutes by default or once per hour how you set it and There is no synchronization between different agents between different nodes So you need to take care about that if you need this or maybe you prefer to use orchestration for this I Will say about that a bit later So we can return to this Okay, so then I will explain how we are going to use it and how we are using it Yeah, so this actually again, there's another main slide of this talk So if you are going to use it use it in this way, so you please use git please use your any CICD Software you like we are using Jenkins and github enterprise But before in was other way using git lab and get lab actually quite I'd say I miss it now because it was really easy to use and you have git repository and CICD in the same box, so just Just out of the box. Okay, so Then I Short and I saw actually people just reading manifested right on puppet server right in environment directory Without any version control without anything It works, but it works for some short time after some short time We will hit into some issues because it's hard to manage and it's hard to check I prefer I'd say please start with control repo from from the beginning make it This is actually git repository when everything you write through sites all your manifest and There is your higher data and there is your test and maybe something else some useful scrapes or whatever else so This is definition of that When you're starting it's really nice to have some example and there are three Links to useful examples first one is puppet labs own example But there are no tests at all. So it just pure repo with puppet Skeleton and that's all so there is no irrespect configuration nothing So friend of mine actually my ex colleague from Lazada did another Control repo skillet on which is based on what we had in Lazada It's more or less same as puppet labs one, but there are aspect Helpers already pre-configured so you can just start reading test there and you will have something and there is PSI-CK I know I know sure how to pronounce it right because I don't know those people So they provide some Kind of building blocks for your repo Which you can use but it's very specific So they have their own vision how it should be and that vision actually reflected by this repo But you may find it's suitable for you needs Please read it Then I'd like to speak a bit about roles and profiles so this is how your code organized it in control repo and So role is actually If we are defining some configuration for Layers fleet, we can we can have some roles there Like your server can be positive server or Jenkins master or Jenkins slave, or I don't know Something like Grafana or Prometheus error. This is actually role role of this particular server and Not usually have only one role. So this role doing this thing like this role is Prometheus error but This role is actually Combined from different blocks like if this Prometheus error, for example, and it's running in Docker You need actually Something to set up Prometheus container there something to set up Docker there something to set up virtual machine actually So this those things are called profiles and yet different Again one thing about roles. I miss it is that role have no configuration. It's just definition So when you split in your code shuffling your code between roles and profiles Usually you don't need to configure role. You need to configure profiles It's hard maybe to understand when you did zero Okay It's already Yeah, have a lot to say. Okay. So this is for example examples of Profiles we have like you can see it after Then next question how to attach your role to profile. I mean how to attach actual role and set of profiles to your server you can use Hostname like match by hostname, but please do not do that. Do not encode your metadata into hostname. This is wrong way Then you can base on trusted facts So when you're generating certificate in puppet, you can put some data inside and you cannot change it as it's only way to change it to regenerate Certificate so we are using this way now and then most the flexible way is to write something Which will assign this for you, which is called external not classifier. I Actually didn't so many examples around but I'm pretty sure there is something on GitHub and Then higher can be used as the NC as well then I guess I will skip most part of higher because I'm short of time already and Yeah, so this is just database. So this is configuration you can see after And here is how it's reflected So you do the fan layers of your hierarchy and it's going from top to down So and last one wins. So if you say something specific for your note, you can write it in note specific Then another nice thing about higher is doing parameter lookups for you So if you define class you can specify the values of these parameters of class in higher You're out of time Okay, I heard 40 minutes, right? It's a 30 minute slot. So you're 25 minute talk. Okay. So let me dance already two minutes over. Oh Why I was sure I actually I did it I will do it really faster William is Okay, I need to go a little bit of time take take a few minutes, but please don't take all the math I will skip part but So how about ICD? This is steps you usually need in your CICD next slides will be about What software to use to do that steps first of all there is puppet development kit This is really new and now it's support control repos as well. So this bundle of software Everything you actually need in your puppet development, please check it out and use it Then there is lint syntax checks or speak once I were doing integration test Then puppet litmus doing acceptance tests. So those wrappers actually doing a lot of things So you don't need to care about just please use them and that's all Then this is what you need to deploy. So you don't need to deploy tests. For example on your puppet server Usually I have just this set of things and how to deploy is actually You will see your task is just to bring this code to this particular place on your server and then There is workflow for example like you're doing check out you're changing things pushing it to servers CICD doing branch and deploying it to puppet master as a branch as an as an environment Then you're going to server you can apply it on that server this particular environment And then if it's actually okay, you're just doing usual things like man doing PR and merging it into master Actually import in production. So master branch usually call it production in puppet Yeah, and have fun after So I will skip out about secrets because I have no time, but you can read about that then orchestration there is no orchestration and opens our spuppet, but you can use those tools and We actually decided to go with Korea latest one, which is framework. Actually, it's not solution So you need to write a load there Here are some examples of what you can achieve. It's actually a successor of m-collective That's why the binary call it MCO, but it's changing So you can see and this is really nice thing. You can write playbooks orchestration playbooks in puppet DSL Which means you can use same language for configuring things and for Orchestrating these those things for example here is my playbook doing open CSDB cluster restart And you can refer to it later because I cannot say it's well-documented, but you can at least have some examples Then yeah, this is what I did during migration already and what I'm doing right now You can see that well, it's not yet published what I'm working on up and sourcing it So one day I hope I will be able to share it for free in in GitHub This is where we are going after this is Immutable infrastructure in short words that mean that you are not managing every virtual machine You are managing image of virtual machine like we're doing it with images for Docker and Then you're just recreating your original machine based on that image It sounds really great because you have immutable things, but I have a lot of questions I cannot find answers for So this is set of questions which are open for me, so maybe you can suggest me something here and Thank you. Sorry for like Compressing so much information in small time and if you have any questions you can reach me in Twitter or In LinkedIn or in Telegram chat here Thank you all so much William doing a setup and while he's doing that invited questions from the YouTube channel questions in the room or comments Feel free to ask So you are looking for you I for provisioning or for you I for monitoring Okay Okay, so we are using terraform for provisioning because we are running in the cloud But we have on-prem and when we thinking about going on-prem well It seems we will use terraform there as well because it's way and where but there is Razor Software made by Puppet as well and in open source world. There is Foreman which doing more or less the same and But it's mostly UI and set of things around Actually, everything a set of things around to DHCP and But that does what protocols but you can refer to Foreman and Razor and then for monitoring well, it depends on what you will use we are using Prometheus and Grafana and It's actually Configured by Puppet so just configure it by Puppet and use it. That's it. I think we should move on given the time Thank you very much. Thank you