 Cool, the arrow is in the right direction. Hello everyone, welcome. My name is Walter Heck. I'm talking about using Puppet in a traditional enterprise. So the subtitle says, don't say DevOps. The idea was that in that environment, we slowly implemented all the things that DevOps kind of does. But they didn't want to use the term DevOps because it was too modern and new for them. So then we don't call it DevOps, also fine. Who am I? My name is Walter Heck. I'm a CTO of Olin Data. And we are doing a lot of puppet stuff. And this talk is to explain to you one of the projects that we did last year. So the company, the specific company name I won't reveal because they're not because I don't want to, but there isn't really any need. I gave the same talk in FOSDEM this year. And after the talk, four people came to me with individual company names and said, is it that company? No. So the things that we've noticed are pretty common across the enterprise environment. So all you need to know is that it's a large managed hosting provider in Holland. And they were of the traditional kind. And when I say traditional, I mean silo is everywhere. There was a monitoring department that I have never spoken to a single person who actually works in the monitoring team because every communication went through ServiceNow tickets, which is sub-ideal if you want to figure out how to get a server automatically monitored. So in total, I spent about two weeks hunting down how a server would, in a normal situation with them, get backed up and monitored, which is obviously not really what you want to do. Another example was at some point we decided that because we're privatizing everything, it means that you're putting all the configuration in a central location. So we wanted to talk to the security team and get recommendations from them and see what was the security policy. It took a week and a half to hunt down the official security policy. And then in there, it said, you are not allowed to store passwords or any sensitive data in a reversible way. So you have to encrypt it in a way that it's not decryptable. But that's really great. With Puppet, that's exactly not what you can do because you need it to be decryptable. So those kind of things happened left and right. The environments that we were looking at was pretty much everything. In the first few days, I asked a few times, hey, do you have any Oracle or do you have any Windows 2000 or do you have any this or that? They said, Walter, you can stop asking. We have 15,000 servers. We have every combination of software that can possibly run, and it's all still running. So while I was there, they phased out a network with the name of a company that was acquired in the year 1999. And that network was turned off last year. So bits and pieces of legacy. 15,000 servers, around 200 clients managed by about 20 teams. So about 70% of these were Windows servers, and a lot of it was Snowflakes. All very uniquely crafted servers that had been set up at some point in the last five years, usually, where nobody really knew why it was set up the way it was. When you want to start puppetizing such an infrastructure, you want to generalize as much as you can so that only the things that need to be unique in servers will be unique. And yeah, that was a bit of a challenge, because very often there would be a technical application manager who would do a whole bunch of things. And one of the things that he would do in a day was sometimes adjust a configuration setting. So it's tricky to get these people to adopt something like Puppet and try and de-Snowflake their Snowflakes. So that was a good challenge. So lots of really sensitive engineers. It's also not something very unique. I've seen one of the things that I do is I give puppet training. I had a group of trainees from that company. And one of the guys was a number of them were Windows admins, and they were not happy at all. So one of them, before I even took my code off, basically he was already complaining and saying that he didn't want to do all this Linuxy stuff. And he didn't feel like it at all. And on the second day, in the middle of the training, he stood up. Apparently he had been making himself really angry. So he stood up and he said, I will not have any of this. And he walked out and went home. So I've given maybe, I don't know, 50 puppet trainings. And I've never, ever seen anything like that. But yeah, that was an interesting experience. On the customer side, what happened was that there was a lag in receiving the things that they would ask for. So they would ask for a new application server or an update of software or whatever. And it would take weeks or months to get the things that they wanted to achieve. And on top of that, for a managed hosting provider, these days, people are hearing stories from others where in Amazon, you can do this in two seconds. Obviously, if you have a snowflake that has a super specific configuration of software in it, that's a completely different story than spinning up a new host in Amazon. But most customers don't necessarily have the technical ability to understand the difference there. So lots of these. So why they chose Puppet for this project? They first, they did a proof of concept because of these 15,000 servers, about 70% was Windows servers, which is the largest Windows project I have personally heard of. But they had some Linux departments that started using Puppet and were very happy. And then they did a proof of concept on Windows. And that worked out quite well as well. As for why Puppet and not Tool XYZ, I think that for these kind of really large environments at the moment, Puppet provides the enterprise capabilities that not many of the other tools actually provide. So they evaluated other tools, but none of them really came any close at that time. So what and how much? They had about, as I said, 15,000 servers. We were a infrastructure team. When I came in, I was one of one and a half people assigned to building the Puppet infrastructure and about over time between four and 12 people building Puppet modules. So normally, you wouldn't want to have a separate team building Puppet modules. If you're going to adopt Puppet in a large enterprise, you would want to have Puppet modules built by the people that actually need them so they have the capabilities that they are actually requesting. It's really cool if you build a Windows desktop module that can manage people's desktops, but if nobody needs it or it manages the wrong things, then you get an endless back and forth between the builders of the module and the actual users of what the capabilities of that module should be. But at the same time, they did an inventory of which modules they would need before the project started, and it came up with a list of about 140, 160 modules, which because of the Windows environment, most of them either didn't exist or existed in pretty poor condition. So they got a team to start building these modules so that they could be put into initial production. And besides that, there was about three or four people helping the customer teams roll out Puppet on their environments. So there was a Puppet team that was building the Puppet infrastructure to be used by each team, and each team then needed to be on to Puppet. Am I losing myself? Yeah. So the tools I'd chosen when I came in, there was a Garrett setup that had been set up by someone who left two weeks after. So before I was fully aware of how things were done, that person left. And since there was not really much documentation on what he set up and how, that became a bit of a difficult situation. Eventually we migrated to GitHub Enterprise, which I am not a fan of proprietary enterprise software, but GitHub Enterprise really for such a large environment where you're basically building an open source community inside a single enterprise is great. GitLab, of course, is the other tool that really helps. Besides that, there was an ongoing effort to set up Jenkins and get it to run tests. But I got really turned off with Jenkins. I spent personally about a month trying to get it all to work. And the problem was that with Jenkins, it has 700 plugins that you need before it starts working. And then you change one setting, and the whole thing stops working, and you have no idea why. And it's all point and click. So not a success. But eventually it was working. The other thing that they wanted was to puppetize Jenkins itself as well. And I don't know if anybody of you ever tried doing that, but it's a complete nightmare. Jenkins uses these XML configuration files. And in order to puppetize them, you first have to click around in the GUI to make the changes. Then you look into all the plugins, config files to see what actually changed. And then you puppetize those changes. Not a very pretty process. So the original plan and what got me really interested in this environment was that they wanted to have what they called a puppet king. And the idea was that all of these different departments needed puppet installations with a puppet master and a puppet DB, et cetera. So we thought, let's build a central puppet master that manages all the other puppet masters. We had it working in a proof of concept, but then unfortunately that didn't work out so well. I'll tell you in a second why. Obviously, when these kind of large environments are sold to management, there are very pretty stories of when this is done, you will have a single button and you push it. And magically, all your servers are managed and you can spin up new machines. No problem, it's all good. While that is theoretically possible, I've seen very few environments where that is actually possible, and especially not these enterprise environments where there's snowflakes left and right. So I think that that was a bit of a oversell, which made it then quite difficult for the engineers to live up to the expectations that were set by management. So that was a challenge. Originally, the idea was to control everything from the central puppet team. This was something that was there before I came in. And one of the first battles I had to have was to not want to control everything because it's a horrible idea. You have 20 customer teams that want to make changes and want to make adjustments to their infrastructure. There shouldn't be a central team overseeing all of these changes. So the original plan was that in order for any puppet code to change to go live on a puppet master by a team, it would have to be approved by the central puppet team. And they did this thinking that the customer teams didn't have any puppet knowledge, and the central team did. So that was a good idea. But luckily, we managed to talk them out of it. It's a scary step to give people autonomy, but it's really the only way this is going to work in a larger environment. So I told you the story about the puppet king. Unfortunately, it turned out that once we had a pretty serious proof of concept working, they were using puppet enterprise. And I had a question to the support desk. And they said, you're doing what? No. No. That's absolutely not supported. And the funny thing is that this was looked over by pre-sales guys and approved. So that was four months of engineering down the drain, which didn't make me ecstatically happy. But in hindsight, it was a good idea. The reason or the problem, the big problem with having a puppet master that manages all other puppet masters is that the SSL certificates get really tricky. Because normally, on a puppet master, there is also a puppet agent. And that puppet agent has certificates signed by itself, by the master itself. But if you do a setup like this, you will have certificates that are signed by the master of that master. And a puppet is not really well equipped to handle such a scenario. So it becomes a bit of a SSL certificate nightmare. So in hindsight, it was a good decision to step away from that. When we started, I think, the first two weeks, I spent probably about seven days discussing things with people. Because I was the first person with serious puppet skills. Before that, people had played around, but not really in a large production environment. So I was, how to say that, covered with people asking me left and right, how do we do this, and how should that be done. And there was large questions that were, as of yet, unsolved. And after two weeks, we sat down and we said, OK, the project manager is taking in all the questions. And other than that, we switched to a scrum-based workflow. And that worked a lot better. We could do scrum, because we didn't have anything in operation in production yet. So until we had stuff in production, we worked with scrum sprints, two weekly sprints. And after that, we switched to Kanban. And slowly, things became easier. And people knew where to go with the questions. And the engineers were actually left to do engineering instead of answer scared managers questions. So that made things a lot nicer. The whole thing was quite a challenge, because you're looking at a large, traditional enterprise where one of the things they did about a decade ago was outsource a lot of stuff to a company here in India. And after they did that, they kicked out most of the engineering talent in the Netherlands to save cost. And the result of that was that by the time they wanted to implement Puppet, they didn't have any local engineers anymore that could really do this. So lots of training and lots of hand-holding to get people to understand what they needed to do and what was expected of them. One of the challenges that was for me personally quite annoying was that lots and lots and lots of changes in management structure. When we were setting up GitHub Enterprise, I remember having a discussion about names of things. And I said, OK, let's name organizations after teams, and et cetera, et cetera. Three people got up and said, no, we're not absolutely not naming anything after anything, because things change faster here than you can ever imagine. So if we name something after team XYZ today in three weeks, that team will disappear because somebody had a change of mind. And we will live forever with an organization, with a name that doesn't apply anymore. I grudgingly agreed and said, no, it's probably not that bad. Literally three weeks later, they changed a bunch of team names. So we decided to just use counters. So we had puppet team or customer team 001 as identifiers to ensure that this would actually last until future times. The other thing that management changes is that every time somewhere higher up in the management tree, something changed. We had to deal with someone learning to understand what it is that we're actually doing and why and how. So multiple discussions for no real reason. In general, I think that in these kind of environments, if you put in a group of skilled engineers, you should trust these engineers to make the right decisions as long as you have set the goals for them clearly. So yeah, one of the problems that we had was that there were simply not enough puppet experts in the country to help out with this project. Because simultaneously, a large bank in the Netherlands decided to move everything to puppet. And somehow, the amount of puppet challenge is very limited. So they ended up hiring basically every freelancer and engineer they could find that had ever ran a yum install puppet agent. So that was a bit of a challenge. On top, there was an outsourcing company. And they saw this as a great opportunity to have a bunch of people that had never touched puppet, to learn puppet. And while I'm very happy that people are learning a new technology, doing that in a large production environment where there are already so many unknowns proved to be a bit of a challenge. So we looked at some of the modules that the module team was producing after a while. And after an hour of crying in a corner, we decided to do some training and some guidance on how things should be done. So that was definitely an interesting challenge. Yeah, I told you the story of the guy that ran out of the training. And as I did more workshops and things to get people up to speed, a lot of them had just irrational reasons to not want things. Some reasons were actually valid. You have to imagine that some of these guys have been working there for, I don't know, 15 years. And puppet is literally the seventh tool that promises to be the end all of all configuration management problems. And so they would just sit there and go, yeah. What you say, it sounds all really nice. But this company is not a normal company. So this year there is budget. And next year there won't be budget. So don't get too excited. And in the end, they proved to be not entirely wrong. So frustrating. The future of that project, when I left in December, they were at a stage where the first teams were really using Puppet, and things slowly started to make sense. But still, you have to imagine that a lot of application managers now needed to switch from the mindset of, if my customer requests that, I'll open tools, settings. I go to the third tab, and I click the fourth check mark. And I'm done to a full Puppet workflow where they need to go and figure out which setting in code that actually is, especially in Windows. That is definitely a challenge. Once they find that setting, puppetize it, then run it on a test environment, and then finally run it to a production environment. So a lot of their daily tasks just simply got a lot more complicated. On top of that, they needed to learn Git and Puppet and GitHub and Jenkins and all of these things. So it's a steep learning curve. But yeah, my contacts that are still there, they say it's going in the right direction. So that was it. Oh, I thought they said questions. So that was my session. If you have any questions, feel free to speak up. Before we start to the questions, I'm doing a Puppet workshop tomorrow. So this is a bit more advanced Puppet, so you can see how to do things in a production environment. There are still seats available, so if you're interested, feel free to sign up. And I'll see what I can do with you. Hello. Hello. Hi. How can we differentiate Puppet from Chef? How can you differentiate Puppet from Chef? So how can you differentiate a Mercedes from a BMW? Mercedes is for business class. It's a lame answer, but really, in both cars, you can drive, and they have four wheels and a steering wheel, just that the engine is built by other people in a different way. And Puppet and Chef is exactly the same. You can do the same things. They have different bells and whistles. The point is that you should be using one of them or one of the other tools. But in order to, these debates are really difficult. Some people will like a BMW better, and some people will like a Mercedes better. And that's totally fine. Over here. Yeah. Yeah, so how did you solve the storages of passwords which need to be one-way encrypted? Yeah, so that wasn't solved, at least not when I left. The problem is that Puppet and highly sensitive data is currently an unsolved problem. There is a Hira back end that does encryption, but Hira EML is what it's called. The problem is that the Puppet master that builds the catalog, the configuration to send over to the agent, retrieves the password from there and decrypts it and then sends it over the wire. So the configuration that is sent to the agent is actually plain text passwords. And this is particularly painful with Windows machines that need to join an Active Directory domain, because you need a domain administrator's password in order to get that done. So yeah, that was an unsolved problem. The storing of two-way encrypted passwords, we got the security team to adjust the security policy because there is simply no way you can store a password one-way encrypted and then hope to use it somewhere else. Thank you. Hey, yeah, excuse me. So actually, we have spread across multiple data centers, and we are thinking of a distributed Puppet master setup. This is for non-enterprise thingy. So what do you suggest about the CA authority in this case? So how should we go about that? So your question is you're thinking about a Puppet setup in a Windows environment and how to go about it? Linux Windows both is a mixture. Yeah, that's really not a simple question. There's a lot of best practices. I think that maybe we can talk about it after this talk, because I can literally talk for three days about it. So I think that if you've already done some Puppet, you might want to come to the workshop tomorrow because there I show a whole workflow that is good for production level environments, and that could maybe give you some enlightenment as to how to handle these things. Sure. Thank you. Hi. So I have a question on the similar lines that he asked Puppet versus Chef. So I'm asking Puppet versus Ansible. Sorry? Ansible. Yeah, same thing. Do you like, well, actually not the same thing. Do you like a BMW or a Mercedes or a bicycle? To go a little bit more in-depth, there's nothing wrong with Ansible. It has a different use case than Puppet. Ansible is a orchestration tool with which you say push this configuration to that machine right now. With Puppet, you say, I want this machine to stay in this state for continuous time. So the idea behind it is different. Actually, a lot of people are using Ansible together with Puppet. That's an option. But really, it depends. I see that in large enterprise environments, Puppet is unbeatable for all the things that it provides. The reporting and logging and all of these things are at the moment unbeatable. Ansible for, let's say, a SaaS environment is a very valid alternative because there you deal with single applications. And it's more of I want to push configurations to these places now. So it's a more simple environment. And there, I would even say that Puppet might be overkill depending on your environment. OK, thanks. Anyone else going once, going twice? Thank you very much.