 And I submitted the proposal for this off, not because I consider myself a Puppet, I consider myself a Puppet user, but I know there's a lot of people using Puppet, and so I wanted there to be a place where we could all come together and talk about how everybody's using it and potentially have a work-on collaboration. So this is a off. I am able to facilitate it, but really I want people in the audience to share their experiences and wisdom on Puppet. So I thought, first thing, well first I want to ask a couple questions. First is how many people in the room are already using Puppet? That's pretty good. I guess other people in the room, there are other people in the room that are here to learn about what Puppet is. So I think many of us, at first we can talk a little bit about what Puppet is. Maybe actually the best way to do that is to have people talk about what they're using to help with poor amount of people. But Puppet is basically a framework for a lot of people to compare it to CFMG, although slightly different than that, but it's a framework for maintaining a state on a whole bunch of machines in an automated way as the kind of thing that you would deploy to have enough machines that have made more sense to do it centrally in less work. Although in some regards, it would be just as much work, but one thing that's really nice is that it gives you all our ability and ability to deploy systems and that sort of thing. So I think we have minds out there, why don't we go ahead and have some people who are using Puppet in your name and in your organization and as much as you're able to share about what you're using Puppet for, what you're using to do in your environment. Bill, do you guys have mics? Yeah. If I can say the screen so I can read what I've written. My name is Trent Thompson. I'm using Puppet at Trinity College in Melbourne. We're only about six years old. When we started to gain up and on, I decided that because we had some old machines that we would just use it on the machine and use it for everything on that machine. I found that with Puppet, it's best to ease into it because it's quite complicated and that's the approach we've taken on people. Just ease into it by managing one thing at a time. Are these mostly servers? They're all servers. Keith O'Brien, I work at an RGA's community company and we're using it to manage four mail servers at this time and it's to build a strong bare metal up with just core deviant and Puppet installed. We've taken the same approach and it's quite a ramp up to get started. It just started with a honeypot mail server and eventually replaced all of that service with Puppet. That's how we're going to feed. Hi, my name is Brian. I work for a system administration consulting company. At this point, we pretty much use Puppet to manage 95% of everything. We're probably managing 150 plus VMs using Puppet. We're currently the only native thing that we're not using Puppet to manage is MySQL itself within the database itself. We've got users, packages, and big phones. Pretty much everything other than that and application code that's quite decent. For us all very experienced at the university, we have been using Puppet from before Puppet had overrides. We used Lute to add them because he didn't make each of them originally. We had all of our core computing infrastructure that didn't run well in the mix with Puppet and all of the systems that our sister organization runs and contracts this administration around campus with Puppet. We also integrated files on all systems, on all package installations, on all systems running on Puppet. Basically everything that happens on any of the systems after Puppet is done for Puppet. We have about 600 machines in that fashion which have a duly wide variety of different configurations. Generally, it's rare for it to be more than about five systems that are the same. No, we don't use Puppet at all for desktops. University means to manage desktops are mostly a theoretical concept. Hi, my name is Helen Nebozis. I'm working at the Recreation Depots Network. We are using all of our servers. This is my work as a PSA. They use Puppet as well. Hi, my name is Robert. I work at the University of Connecticut. I'm managing about 50 servers and 50 workstations and we're not going to be using Puppet, but I'm very interested in Puppet. I work at the University of Connecticut. I'm managing about 50 servers and 50 workstations and we're not going to be using Puppet, but I'm very interested in learning about it. I think that's everybody for now. One of the things that I've come to realize when using Puppet is that before I'm going to deploy a new service and I want to start writing cover recipes for it, the first thing I do is I go out on my Google to see if anybody else has done this. That's one way of finding things. If it doesn't already exist, I try to write it. I wanted to ask people what things do you run into that you needed to write either just cover recipes for or be written more generic modules that you could use to deploy services and say if anybody wants to talk about some of the stuff they've done with Puppet. I can say something. We've been using it for a long time, but unfortunately when we started at Stanford there really wasn't a lot in the hardware, so we kind of wrote most of it ourselves, which means that a lot of it isn't very specific to how we do things. There's the old statement from the Mythical Land Month that turning code into a product takes three times as long and turning a product into something to ship is about three times as much time. One of the things that we're missing right now is how do we take a lot of that stuff and share it in a better fashion with other people and kind of detangle it from all the stuff that we've written internally. We're managing just about everything, so at one point or another, we figured out how to kind of server the Puppet code. It's mostly not particularly well in terms of making sure the whole process, but you know, everything from my SQL server to LDF servers, Kerberos, KFCs, they're all managed in public. Yeah, sorry. I'm not one. I work with Condit and we've been using it for about three years and basically one of the things I've always struggled with is to avoid word duplication and so we're working on creating public modules that can be reduced, a patchy module or whatever module to manage that everyone. Unfortunately, we lack the time to properly do that, as we said, because it's pretty hard to share those things, but public is one of the cool things about public that makes it much easier in other systems to share those things because the syntax is readable, so it's kind of easier to learn. So we can pretty quickly figure it out and start using it. So yeah, whatever reason we're trying to create a kind of set of package that has these documents where all the modules are on the public. So if you guys are looking for something to manage any piece of structure, whatever is still that question, there's already a bunch of people that have published various modules that have a directory of modules. So it's not yet one single authoritative source of modules, at least there is a directory of modules. My name is Jeff Crompton again. Like Russ, at my organisation we've written a lot of our own methods and to some extent I think that that's not a bad idea, maybe in your first time grant because you do have to learn the syntax to some extent. So just this last couple of weeks I started using two cases of other people's modules. So to some extent you've got to write grant benefits and in one case I'm using someone else's managed module and they're writing my own module to extend it, to customise it to our needs. In Verizon, if our workflow is generally been when we're first starting a service you write a very generic recipe that does nothing more than just copy a couple of files which are things like that and grab it over time and grows but it's always kind of organisation specific for a while until it gets to the point where it's like you know this would be really nice if this part was generic and then we usually ended up splitting it into a generic module and a site specific module or just a site specific class when we make the generic module such that you can set variables to do different things. It would be site specific, I think first of all to answer the question that you asked what are some interesting things that you're doing with puppet and I think two of the most useful things that we've started to do with puppet that people don't always realise is very, as possible is that we're auto-configuring audios and they're actually quite easy now with an audios module and unit audios not to say there aren't there shouldn't be improvements that we make but this would change the fact that I no longer have RSI because I take long configuration models which is great so those two things I think have been a big win for me but to go back to the way the thread was going about the modules and sharing the modules and that sort of thing we've written quite a number of different modules and we're trying to move most of our internal infrastructure to using as much modules as possible the hard part of writing modules that people probably discovered is that it's hard not to hard-close your infrastructure into a module and if a module is going to be shared amongst other people outside of your internal infrastructure where you have your own policies and all kinds of weird IDs and routing or whatever it is that you have written in some sort of abstract really hard so you head around and think of and when you're trying to just get something working for your infrastructure it's sometimes easier to just do your your particular configuration so the way we've been starting to move is having abstracted modules but then having site local modules that do some of the changes that we wouldn't otherwise pushing into a separate module so that way we have some of the abstracted configuration separated in the module format so that you can share the module with other people the public ones are then internally. Your comment about part of abstracting is that the best way to find when it's not abstracting out is to get other people to start using it or you start using other people's stuff and you realize like hey this is like clear coded for their organization and then you fix it and the more people you have working on it so it's going to release early release help and kind of thing because then these things get discovered it's made more generic I think a lot of the stuff that once you've cleared out there we get a passage all the time from people that they're like oh hey I'm using this and I want to add this support for doing something that my infrastructure needs I didn't really use public but taking a couple of ideas that came up in the previous talk I was wondering if it was possible to with these modules push them into the package that they and put them in a public domain directory in that package you can pick which bits you want to select like water check or something like that where the individual package delivers resources I've been maintaining that package for making modules and making videos and files I've been trying to do that or to have them move themselves on their own and stuff else the way I'm surprised that he's using it is how other people are doing it but we maintain all our public stuff and give repository and the modules that we are more maintaining or using from other people or get some modules and so that's how we're working for making it anyway so it wouldn't be that useful to have things in the day in packages I think it's also maybe the wrong place because you don't need them on the deploy system I can try to answer that because it's something that people will ask there's somebody that takes care of it so there are such modules out there I want to package them because I want public to be more useful out of the box because out of the box public doesn't do much you can follow the little sample things they do so you can configure it you know the whole style of whatever and to follow the samples and to work your way up the very useful things to work with other cases and stuff the problem is that you don't have the standard set of modules yet and once we have that you can create a package that's going to be useful for everyone but right now it's still up for debate where the standard package is from the standard days of modules is or what it is or who's managing it or how we share it and all of this so we have started this discussion on how we are going to manage that and I think one of the goals should be to have a decent package for that that I don't think we're planning or are they yet answering your other questions is to have a consistent environment on every server we're learning so we have like 60 servers and we don't need to install every one of those and create the system in users everywhere everything's already done for us and it's consistent everywhere find this sync configure so we don't we know where we are and it's just like as if we were working to create the custom distribution but it's easier it's just basically I have a comment very clearly and one thing I've been telling you recently is that as soon as I get older my energy starts to get worse and one thing I can really look up to very important stories and that's how I kind of feel about Huppet I used to be able to pull all this stuff in my head but so many services were maintaining that I can't remember what config files what and that sort of thing so it's really nice to be able to take all this stuff write recipes that do all the magic put that on revision control to keep it in the working sense I can stay there and I need to work on it I can go there and read it again in front of my memory under ideas I'll put a note saying you should start packaging stuff that should be work on policy that what modules we should package and I think maybe that's my main approach in that problem which modules are standard modules if we don't wait for the rest of the public community to do that if we just say two things in modules run a little bit of policy start writing what we've seen get it wrong a few times then that's fine I'm just going to speak further in the great way about that the Huppet configuration is covered and it's different that you're treating it like from the unit so you put in a git repository things from other git repositories the nice thing about this is when you're working in a group you can set up on commit just send an email to the group so you can see what everybody's doing send a disk and you can just track all the changes that are going on in the environment and another thing about this you're talking about your own memory but it also helps with institutional memory if somebody leaves you can see exactly what they were thinking when they made certain changes why didn't you change the variables in that specific file and you have a commit message about that I mean it allows you to create but I think that itself is going to be Huppet that's really valuable and to some extent you're still putting in a lot of effort in some ways it might be easier to just go ahead of the file on machine or something it's more effort to check something out and make changes in a recipe about the packaging of modules I agree about the idea but I'll leave it to myself I don't know if we're there yet but I think that on a daily basis I'm making changes to the modules that we're using committing to our gift repository a package that I wasn't changing in some way we have to get to the point where we're understanding how we're going to change packages for our infrastructure on a regular basis or come up abstract enough so that just installing it and then having local something that's changing or not quite there yet but I'm too tied up trying to publicize my infrastructure and be able to change those things and I'd be very happy to be able to do that but I think if someone does do that not necessarily me I wouldn't do that it's kind of going to force the issue because it's going to be harder it's going to be more packed but then you're going to follow about the course I know I can't explain this what do you think that happens with modules that you can have specific modules that overwrite these parts as a module so if it's just on that one there you go I think that having that common shared public modules package is a very good idea I'm not going to use it because I ought to be I can barely keep up with my own infrastructure but I think a lot of people would use it and again people that are interested in working on that can base the packages on the work we're doing I think that's a great idea I don't think we should do this way it's just that the other thing I like about public is the way it allows you to share that practices is the fundamental thing to be able to version control system administration is a totally new concept that I've never found before you always have how to lose an infrastructure and how to do things like that and those things are always dependent on the system that you interpret it's going to be or she's going to fail regular pace and it's going to be a very persistent so what it allows you to do is to share those best practices those instructions in code so that they're formally described and executed with that machine and then when there's an inconsistency on a certain system you get reported and say on the operating system or in that environment what's different if you don't use that yet you're young or whatever and so this is a very powerful thing because it allows systemists to have more approach to their work and start sharing those common best practices which is something that's very hard to do traditionally the black magic assistant men have all this stuff in their hand and once they leave you're caught because there's nobody else that can figure out how things work and they have to maybe they've got some different issue with it but nobody else understands it so it allows you to decentralize that and share it and allow other people to basically do your practices and say well that's bad, that's insecure but that's good, that's a good idea or I'm better at using that and you can improve and share a lot more it's really quickly bringing the open source culture into the system Another approach to that style of knowledge is to set your head is not to avoid the best just avoid every one I know when I second the both bad and the idea of putting your work without a lot of clarification and control and then sending out notifications to people who want to follow those and we moved over a few years ago to get and I just love that aspect of things the ability to have history, you know exactly when everyone changed everything and when we have a really strict policy one of the reasons why we play puppets in the first place was we had a lot of systems that would go hacked up on blocks and when they say they put it back into the build system and the nice thing about puppets is that it goes along with files they do that and then puppets reverse the change and then it stops working and then they are forced to be a little puppet and that is very valuable right at the beginning and then having it all get the other thing the side bonus is the fact that every single one of us as an administrator has a complete backup of our entire machine configuration sitting there and they're all together so if anything ever happens we have some sort of cash drop infrastructure failure you know 20 different backup bodies living around there once are divided into states are we going to get to the painful night very much of the failures but they're part of it I don't want to separate that I've got a slide page this is a question that I would like to go on with if they have an important senior problem I was using both of them to manage 400 nodes lost or probably not so I consider it quite expensive operation to manage packages reported because if I turn on the maintenance and all that stuff which take a lot of time if we want to do it live by 400 or even 5,000, 10,000 computer it's really a lot of studio hours so I'm going to do some many thirdies that are seen from master from time to time in recursive ways but it was kind of high-pitched and I think improveable so I would like to know some other problems but the problem is that if I were going to maintain packages reported over 400 I think with all nodes it will take a lot of time for the package manager to update them on because it will be like let's say 20 minimum and it will fly by 400 it's a lot of studio hours but the nodes were not being used for real or as opposed to 30 seconds VR scene Yeah, I can imagine that I understand your question probably the problem is that when you this specific package is installed on the machine it takes time to install the package Yes, I'm told that they can process them quite small So I think that's a problem that you can't avoid if you want to have a dynamic configuration system you want to be able to install new packages on the fly so you can't avoid that there's just no way I think the only way you can avoid doing that is when you set up a machine you have a specific set of packages set up and then there's no way to run but after that you need to do some upgrades some processing and if the package is taking too much time to upgrade or to process well, the promise of the package that package should be fixed it's always in the KGD or it's somewhere else Puppet just does the install for you and well, Puppet has done some very good performance and we're going to go back to that later but I think there is no proper solution to your problem apart from receiving like starting from that image instead of starting from that base you can install then installing Puppet and then installing all the packages you can install all the packages and then run Puppet to stabilize the integration as you go along and change the history Perhaps a lot of approach would be this, my stable configuration when you have a change coming up you remove servers from the pool or stable servers tell them that they're now allowed to change and once they're back to a stable known state and they're qualified, you make them back into the pool of available customers so that the only things that are in the cluster at the time are things that aren't changing that are in known states other than that I think maybe there are a couple different layers of problems maybe that you're running into maybe is that Puppet by default is going to just wait every 30 minutes or so and then run configuration manifest and if that's too long for you to wait to get this to work then you might need to be doing some sort of Puppet run pushing or forcing those clients to install those packages sooner or using something like AmpliActive which is using published and screened middleware for Q pushing and controlling which servers are going to do the installation the problem is the packages themselves take too long to install and that's I think a problem outside of the public that needs to be figured out through some of these suggestions that might be stated or stable so one of the things we've noticed as we've been growing our infrastructure and adding more and more machines it's almost a thundering birth problem of the Puppetmaster that now we have a bunch of machines all getting the Puppetmaster once every 30 minutes and half of those happen at random intervals but there's still just enough machines hitting it that it's not load is starting to be an issue so those of you that like recipe sell like 600 machines or something like that probably the things that people have done that problem or there's a very easy solution because you can go out violence via DNS Puppetmaster and then just replicate your recipe or directly to your ASCII to Puppetmaster and that or to the sphere of any kind at first I was going to say I run services in a department where somebody belts you on to Puppetmaster on one behalf and as a system administrator it drives me nuts they have the thundering birth problem the solution they implemented is a large number of Puppetstarters and then prom jobs that run at some random time during the day and what do you and is there not bad enough that I wouldn't recommend it I mean I've talked a lot probably with people who are running your Puppetstarters because we share a lot of the same trouble the Google internal folks and Stanford with performance problems we're both balancing across four Puppetstarters and we still have to restart them every four hours because they're really generally the same so a few of the things that we found out in the talk that I've worked with Google one used passenger if you're using Longroll you're going to be very sad if you're using WebRick you are not scaling to more intense systems anyway so and then the other is is that and with that performance stuff in general I'm still kind of conclusion to that it's great that people who wrote Puppet love Ruby I wish that it were something that was in Ruby before the underlying the fact that Ruby is underlying architecture seems to not handle that kind of scale very well but passenger apparently is huge improvement and apparently also the latest version of Puppet are significant improvements on that so about the memory thing we've noticed that it happens with political things so we've mentioned my scale and we've actually managed to fix that for many hours of black porcelain rails from squeeze and some of our rubbery stuff like part of rails and nowadays we don't we used to have a project that started Mongrel's and nowadays we have another kind of months on our Ruby setup on our Ruby Puppet rails we use Mongrel's and it actually works we send him X for a project and we also found the transparency so it runs into the console but it's randomized for its time so the machine can roll at the same time we're doing that too, is anybody running Puppet as a dehuman on the client? I think one another well it's been mentioned that Puppet is improving in terms of running it constantly as a dehuman on the machine instead it's trying to just be occupying memory for 30 minutes in a way that it does something to a lot of people especially on a longer version we used to run things maybe a lot of people go through this maturation process on the Puppet starting out with your web rig and then moving to keep going and keep going and knowing we're at a passenger we were at this engine X Mongrel setup for a long time it started calling over so then we moved to the passenger and now we're okay for a little bit longer but now the public will have to go back are you watching something different? are you watching something different? yeah you can do it with an X Mongrel or much more familiar it's actually in the knobs so sharp but we don't have this problem with the passenger the Puppet master failed and needs to be restarted every four hours we are doing a lot of resource collection into MyStrel for being able to do things like community module or an August module and that's a very heavy resource contentious issue so we're using the which came with 05 I think which allows all the MyStrel operations that Puppet master needs to do to the MyStrel server to get queued up and released from the Puppet master that can be quick into the database whenever it gets wrapped it's a little I've refactored the Puppet master so that's a lot easier to set up now and it moves all the dependencies so you can get but we are still running our Puppet master for the old hardware now people.com it's serving clients so you can spend a lot of time tweaking all the stuff and actually geeking out a lot of performance there are issues all up and down the stack like was mentioned there were some rails issues that were coming from MyStrel timeout stuff and it turned out all of the only fixes was to install rails from it but you get things from all up and down and tweaking it and realizing that things are starting to settle out I think Puppet is getting a lot better in terms of sending files across the network there are a lot of changes and improvements we've got a comment on that from MyStrel there's Sobel and Ganef and Andreas and they said that Puppet on small machines like in Beethoven takes ages train two of them with you on machines and how would you make Puppet run faster on these ARM, RGL, MIPS where it runs an hour and more when it goes a minute on AMD64 and Andreas also wants to use Debian-Enterprise in this and also Ganef comments that at work he has to re-start Puppet and master once a day like Ron it is post-stress database connections when using installing the stack I don't really know how to speed up Puppet on an ARM based machine except to replace it with something more powerful but machine itself is a Ruby problem yes, probably a Ruby problem AMD64 machine with a capable amount of resources available to it it depends completely on what you're trying to do in your manifest how long it's trying to take to compile and apply a manifest on AMD64 machine not that bad but I've seen compile manifest and applications on less powerful AMD64 machine for 20 seconds or less but I don't really think a minute is a problem if you ask me we have a show of hands of who was doing the Debian we have a show of hands who's doing basic programized what we're doing is we originally were running modern build and from even clients and we had to do all this to be switched to passenger we're using back ported rails and rakes and all this other rails and Ruby stuff is back ported packages on the master since the upgrade to 025 you don't need to restart from the master every hour or so so that's also very good and on the client side we switched away from using the Debian apparently with 025 it's much better you don't need to do that anymore but we I just can't afford switching back to the Debian because it's been very visually affected like when the from the master fails and keeps on memory and crashing it's just one server doesn't matter if clients keep on working with existing configuration and when the clients fail it's all around your server well it's all your production server that's just something we can't accept at all so the only problem we had the only problem we had with current job based clients is that sometimes the client would hang on a graze or like various constituencies and whatever we had we had clients that would stop running for some reason would be lying to the machine and find that there's a puppet that hadn't that's been running for like a month or something like that but that's the only problem we had otherwise it was getting above 60 machines and now it needs working very well well it would be nice if the clients knew what the load-on server was instead of waiting for them to take a look at Google or use your payment pretend it was for him collected interesting one of the people working on Puppet is working on this and it has a lot of interesting ways to plug it into Puppet to do things like only run just to stay around does a lot of other amazing things the problem we're using on Puppet right now is we supported that in for the activity pieces that are needed are not quite there yet Tim, there are small moments to all of you who are having these memories it's an active record back we just dumped the memory of the processes that were leaking and it was memory hundered after a month and so on so a great active record and the second thing is that if you engage them now 0.25 is really, really much faster and the switch from to the rest there is something and 0.26 is faster 0.25 is faster from where you see it how do you do this time with Puppet I'm going to talk a little bit about staging we give a lot of thought because we go on clients who don't like us to make changes so well, except for when they ask us to make but not stuff where we change the global model when we want to deploy that everywhere so again, I put it under version control and you can use all sorts of things like framework software release tools that you would normally use for software so the approach we're going to be thinking about is using git branches and basically whenever we add a git branch to our public repository and push it to the server it creates a new environment so that you can put clients into different environments and they get their own git branch essentially to that environment and then what we can do is we can create a stable git branch and we can create a development git branch all of our test systems and our clients test systems stay on the development branch we push all our changes to development first and then it gets so if we want to pull something in stable we can do it get a charity in the stable or more likely what happens is that in six months we require a new stable essentially do what Deming does to release, do rolling upgrades of all of our clients by just switching their environment to the new stable branch and then continue on on development and that kind of model works great because when you have a version control repository sitting underneath the whole other configuration Do you have a documentation and scripts for your setup available? Not yet There is a best practices guide on the Wiki I lost track of where it is at So we've got about 10 minutes left and I was one of the last things here to make sure that how we move going forward I want to be able to capture what everybody is doing and try and start working together a little more Rise up, realize that it made more sense for us to start doing things as generic as we could in terms of hardware with other people so I have a URL here for where Rise up tracks on our own modules Rise up and a bunch of other organizations also have that together and this lower URL here is a site that we have where I guess site libraries and different organizations now are trying to collaborate on common couple modules and reason to get for everything several people mentioned the fact that if we could just get the common place to go to start with these things that becomes the facto place and then people can start contributing to it there's a lot of other examples of this in the community that I think would be good for a couple modules to follow if you looked at your unit exchange or at one common place that everybody started I think right now maybe it is but why is it people are doing things and are making their peer repository public let's try and put it in this document let's think on the way that we can start from there and I made a list of things we're doing and if you see things in there we're like oh yeah I would be useful we're doing that too let's get together and try and merge from ISEE humor with respect to speed if anyone has tried for convenience which is ravini.us not like I have that's probably regarding the project environment and stuff one of the good best practices I discovered the hard way is to enjoy things one way or the other that is when you make a change there's an effect on your mission don't do it on all your machine once do it on one machine and then machine then off and it's once on many or whatever and there's a way to do that there's a marketing convention here they're just doing a kind of creation on one note it's very easy to do that but since you're going to the best practices way it imposes you to keep our care full of something that is one very key point I wanted to mention any other comments a couple things how could you have a contract on the case so you can say put it in this group of machines on this convey then you can roll it with multiple switch like you was describing but the other thing is in regards to ravini.us we haven't run a ravini.us yet but we are doing testing with the enterprise edition and that does gain some performance benefits and memory theoretically it gets about a 30% memory production we're seeing some reduction it's hard to quantify but with public memory use it varies the other thing is I mean this isn't going to help anybody run away but if something goes forward to a group can depend on replacing the ruby memory garbage collector I mean generational garbage collector replacing it with a generational garbage collector so garbage collection operations will have an increase in performance of over 10 volts so ruby itself is largely constrained by the performance of this garbage collection so you should see an order of magnitude improvement I just wanted to quickly finish the puppet in deviant is a layoff group managed package now and it's not just me anymore working on it which is great in fact there's a number of people and I didn't do the last release I hope it's even better and we're also rolling in collaboration with the boom 2 and their work so if you're interested in getting involved please check out the aviat group and use that help and we'll put this document that we built on the dev company and probably I'll just put this on and I'll let you know