 Welcome to another edition of RCE again, this is Brock Palin and We have a website at RCE dash cast calm and there you can find a nomination form where you can tell us about things You want to show you can also find a link to Jeff's blog and all of our Twitter accounts on there and Jeff is with me again Jeff from Cisco Systems and open MPI has been a wonderful support and Alternative viewpoint in these shows so Jeff. Thanks again for helping out kind Brock. Thank you very much You're far too kind. Yeah another since we kind of have to obligatory mention our our indirect sponsors here Have a look at my blog out there. It's it's about MPI and HPC kinds of things You've got any kind of burning question about why does MPI do this or what? What's a particular viewpoint or a butake on on you know high performance networking things like that feel free to Give me a shout out. I do blog solicitation so to speak so you know feel free to ask your questions I love to talk about this stuff because this is my world But by that token we have something a little different today, right Brock. Yes. Yes, but quickly again Jeff's blog is linked off the RCE dash cast calm web page and they'll take you straight to his blog at Cisco But yes today our guest is Michael Dhan from puppet labs and the puppet project And it appears to be some sort of provisioning and system management system But I'll let him explain that Michael why don't you introduce yourself and tell us how you got involved with puppet? Sure Right, so I might be product management for puppet labs and kind of help pop it along with various miscellaneous things and strategy and Conferences and you know kind of figuring out where the next generation of our systems management platform is going to be but Been following puppet for several years now I was at red hat in my previous position and where I created the cobbler project and also did a lot of work on funk if anybody has used That platform and puppet really was impressive to me because to me it's been one of the most absolutely successful projects in terms of building up open-source community around getting system and together to share content and work together towards common goals of automating very very large computer systems and Seeing that level of cross-platform application and and then complete breadth of the solution is in all an open-source is really really neat to see so I've joined them in the last I think three or four months and It's been a very great experience talking to a lot of the same users and that kind of thing so Because a lot of interesting places we can we could take it But yeah puppets an automation platform So build the model of the way your data center config needs to be from a centralized perspective and rolls it out and makes your systems look like you want them to look and it's not just an initial provisioning but also Full life cycle management, so I don't have to reinstall a machine to Apply a new configuration and things like that So what's one of the or some of the main features that a sysadmin looking at using puppet like what would they even originally be looking for? for puppet Sure, so a lot of times when people I mean I get to the very basic level right people are still doing things manually They'll go ahead and stick a CDN and then install an operating system And then manually install and configure their packages and that doesn't scale beyond, you know five systems or so, right? And then a lot of times people will build hand-rolled scripts to do these types of things So a bunch of shell scripts and automation That kind of infrastructure takes a really long time to build up But a lot of times people will build in-house tools But the idea is can we all get together to share a common tool that has Really really good features and there's been really really well thought out versus having to kind of build these systems ourselves and all these different Corporate environment, so if you go from one job to another I can go ahead and pick the same tool off the shelf and reuse it So that's a really powerful concept So yeah puppets really intended to replace those kind of shell scripts and parole scripts and kind of in-house systems SSH loops and replace it with a more better model of your data center Think one of the interesting things about puppet is it's got something called the resource abstraction layer, right? So we have a we have a way to talk about a user But if you're managing a user on a Solaris box and you're managing user on a Linux box The commands that you use may be just ever so slightly different If you're managing user on OS X it's ever so slightly different, but in puppet language You can actually describe a user in a relatively cross-platform way and then depending on what system you run that recipe that manifest you'll be able to Infantiate that user in a cross-platform sort of way and it's very very lightweight easy to use and it's a very high-level conceptual thing So if I'm talking about a user, I might be modifying their password file I might be modifying the groups file all at once, but I can just represent that as one Concrete concept rather than blasting out this config file and running this command one of the other things that's really interesting about it is it has a it's a declarative system, so when we say that a particular Resource needs to look like something puppet will make it look like that rather than saying Go here and run these commands because if you think about it kind of like a travel analogy Say I want to get to Washington DC for instance Well for me that's driving north, but if you if you script that you don't want to say drive north because you might actually be in Chicago So puppet allows you really say I want to ensure this user is here And he has these attributes and only run these commands if you have to And that goes from services, so you don't restart services unnecessarily that you don't copy files unnecessarily And that you have a really really good model of what your system needs to look like and then puppet makes your data center match that description So let's take a step back here for a second here. Let's draw a clear distinction So you mentioned that you're at puppet labs, but you're working on puppet and there is a community Could you delineate, you know, what what are the differences and what what are each of these things? Oh sure So the puppet project was started by our CEO Luke Kniece about five and a half and first so years ago and We've recently incorporated puppet labs with you know funding from true ventures for the people who do word press and lots of other great projects and several other investors and We're essentially doing services and support while also helping build out and continue puppet so It's really kind of one of the same thing But we have a lot of contributors in the open source project the mailing list has over like 1600 people all helping each other out and sharing content and Helping review the code and contribute features and things like that. So very much classical open-source services and support model kind of thing Okay, so I think you just answered my question there was you know, why did you guys incorporate? You know, if it's a good open-source community you mentioned you were a red hat before so I'm kind of guessing that red hat is one of the Participants and contributors. Is that well? Well red hat uses puppet quite a lot They're not actually a part of our entity so to speak right, but yeah red hat uses it for managing fedora infrastructure And a lot of their own internal systems and they're pretty big fans and they've made some good contributions in the past We also do we're also doing some work with canonical right now for their cloud, which is Really nice to see and we'll be going into up to Ubuntu developer summit in about a month. It's the volcano dies down and Talk to those guys a little bit. So oh, that's cool Well, you're looking you know by incorporating you're looking to do both support as as puppet is becoming commoditized and more popular You know kind of bringing puppet to the common enterprise shop. Is that kind of where you're thinking of going? Yeah, absolutely, and we're there a lot of cases already So a lot of people that have very large puppet systems, you know approaching Even if it they're just have a hundred machines or maybe they have 40,000 machines these people do need you know some help with their environments and we're there to provide that we also do training for puppet So there's a class in Washington DC. That's coming up next month for instance That's a public training and we also do like on-site trainings if people want to learn about puppet and And there's a lot of Lot of other things we can do with support to like if you need help writing custom plugins for puppet There's one thing that we can help you out with So if you have a resource that maybe puppet doesn't manage very well today We can work with you develop that resource and get that resource kind of open source and then let everybody have access to it Which is a really neat thing to be doing So you mentioned that a red hat uses at heavily we also mentioned Conocled the guys who make a bun to so does puppet use the underlying native package manager and there's like a plug-in for apt-get and a plug-in for yam for doing rpms and there's a plug-in for yes How does puppet actually handle that? Yeah, that's exactly right So puppet has a type and provider system So what it is is we have a basic concept of a package which says install this package at this version and depending on what operating system you're on you you get a certain default provider so you use yam on you know fedora or red hat or syntho systems and you'll use apps on Devin or a bunch of systems for example and Then if you want to use like a specific other provider on that system Like if you want to use a different package manager you could still do that But you get the right one by default So there's not a lot of configuration you have to do it mostly just kind of works out of the box So is this mostly Focused around Linux systems or do you have support for other? Unix like or even non-unix like systems Mac OS 10 Windows Solaris Right. Yeah, we do have support for Solaris and OS X a good example about the OS X support is Google actually uses Puppet to manage their Mac infrastructure for desktops and things like that You also Apple's you know a pretty big user of puppet for some of their you know services as well, so Then Solaris is another good example, right? So anywhere where it's very unixy is easy to do and over the next year So we're also going to be looking at developing Windows support So I think that's going to be really neat because a lot of the cross-platform tools to manage systems are not very good Right, so we've seen efforts like this for things like sound and really really big expensive management software and software that Probably isn't what you want and kind of does like the least common denominator and not in a very good way So puppets kind of uniquely positioned to be a tool that everyone can really like But yet can really really deliver on the cross-platform level. Okay, so let me ask a clarification question So what I hear you talking about is configuration quite a bit there. Do you does puppet do? Provisioning as well. So can you take a bare metal machine and say hey puppet go make that a Centos machine puppet go make that a red hat machine or whatever Actually, it doesn't necessarily do the bare metal provisioning I previously worked on a project called cobbler, which did pretty well that for a lot of the Linux systems But what it will do is take that system from a very very base OS install state and Then deploy and provision applications on top of it So if I say configuration, that's probably not entirely accurate was we're also deploying, you know, complete application stacks We have a type that allows you to check out hostage applications from virgin control For instance and make sure their services are started or you can install packages That you can manipulate users and groups and pretty much do anything you want The but yeah, so good best practice is either start from a very very base kickstart configuration, right? So it's just enough to bootstrap puppet and tell the node when it comes up Register the puppet master and enslave yourself or you could set if you want to do image based appointment You set up a really really basic image with just enough so that when that machine comes up It registers to puppet and then you manage it from there on So that's what that's what most people do with OS provisioning at that level Okay, so actually that's a very interesting concept So, you know our usual spin on on this podcast is is HPC flavored types of deployments where you have usually a heterogeneous cluster of you know anywhere from four to hundreds or thousands of nodes But sometimes heterogeneous to because you know, I bought some machines this year and next year I got another round of funding and those same machines were available So I bought you know this year's machines and so on so it's usually a similar but maybe sometimes a little you know different mix of cluster of machines throughout the cluster and It is a big challenge to be able to manage all of that because you know managing four machines is very different than managing 100 or a thousand or 10,000 machines and so what I think I hear you saying is that puppet could be a valuable tool in an HPC environment because it allows for you know system admin, you know control of large numbers of nodes with very Well very easily I guess is the right way to say them am I hearing what you're saying properly Yeah, absolutely, and if you have a really homogenous network You could actually use puppet to even build the image right you could just go ahead and say I wouldn't do image builds and I want my configuration in puppet so I can keep track of it in version control Like I think that's really important So a lot of times if somebody builds an image and you don't know how you got it Then you have to wonder well all my packages up to date. What's exactly in this thing, right? So puppet makes it really easy to keep all that configuration in source control And then the other thing you said about you know heterogeneous versus a modest environment says Puppets really good at pulling variables up from a system We use this there's a tool called factors that comes with puppet that returns things like the IP address and the MAC addresses and things like that and we could pull that out of the system and then use it in templates within puppet and Depending on how you configure puppet it's even possible to access configurations of other systems So I can say find out where What's you know all the other resources that are like for instance configuring monitoring so I can use Basically say this node is monitored and on another node, which is say my Nagio server I can pull down all that monitoring configuration and configure it So that's that's really a kind of a neat aspect to kind of the centralized management, but yeah absolutely So what's the back end of puppet look like are you storing all these configs for all these different servers and text files one per Host or is there kind of like an overriding class and you say this host belongs to all these classes and it combines them together and tells you any conflicts Yeah, it's it's kind of very class-based the we have the actual puppet language, which you can go to Docs.puppetlabs.com and read all the information about how that works It's sort of the text file and then typically you'll either Have your mappings of what nodes get assigned to what classes and you can also assign certain variables to certain nodes You could either do that through a text file or you can also use something called an external nodes tool Which why that sounds rather complicated is really a pretty powerful concept So it says I want to plug into some external systems Say I've got my configuration stored in LDAP or a database or maybe something proprietary to my company and Puppet can call out to that and it's just basically any script that returns JSON or YAML. I think it actually is YAML right now And you'll pull down all that information and decide what nodes get what orders and Then that's a that's I think that's a pretty neat concept How easy public can plug into those kind of external systems and you can also write server-side functions that can also access the system So it's a very very adaptable way of how I want to make my data to drive my infrastructure Okay, so you can push things on to a host based on things That the host reports back saying I have this graphics adapter So you say oh you need to be part of this class without explicitly adding that host to that class Yeah, you could definitely do that sort of thing like you could write a fact that says look at my hardware configuration and return the information about my graphics card and then in the the actual thing I can say if this is true do that if not do this If you if you do have that kind of hardware variants you want to handle a lot of people that you know we're talking about mostly doing you know operating system agnostic stuff, but that's The custom facts are another case to where it's really really pluggable to where if public's not doing something I want it to do. I have integration points at the server where I can extend it and Public this part of puppets actually written in Ruby all of puppets is I Can extend it on the server side or the client side to kind of adapt and lock into those resources and do extra things Right, I could also like mentioned like we talked about package types So if I had to say arbitrarily I had my own really special package manager I could teach puppet how to use the existing file resource to manage that new package type Or if I wanted to say manage like a foosit and that was something completely alien to puppet It's possible to extend it without you know modifying puppets core code and write new modules and Using those types and it's it's really kind of It could be made to do all sorts of different things So how low a level does puppet actually understand does it understand anything like IPMI or IBM blade centers or something such that I Could very easily tell puppet make this host now a windows box or make this host now a Linux box Right. Um, yeah, so what you could possibly do something like that Which you'd probably want to do is designate like your one server to be a control server And then you'd be basically sending that information out. You don't typically use it for operating system provisioning now however operating system provisioning is one of the Things that you know, that's kind of my background right with with cowboy and a few other things So I think we're going to look into doing more things like that and there's already been some effort to See how we manage virtualization, right? So virtualization is really enabling force for that kind of thing So if I want to enable a windows box Most likely I'm going to be probably having a Linux host or VMware server or something like that And then I'd be modeling the resource and puppet to be able to say bring up these these nodes, right? But yeah power management is something we could possibly do It's a little bit outside perhaps like the model of the way we would normally want to do things just because We kind of draw the line that here's the operating system now. Let's manage it But you can there's definitely ways that that could work out and that's maybe something depending on the particular sites needs They probably want to talk to us a little bit and we'll figure out exactly what we want to want to automate So classic problem that that I've had so I'm an MPI developer, right? I have I have a cluster of 60 some nodes or something which is not a terribly large cluster But it's a big four for me to manage all by myself But a problem that I sometimes have is that oh jeez, you know a hard drive crashed So that node's been out for three weeks while I'm getting a vendor replacement or whatever and then I want to bring that node kind of you know back into the fold and Slurp in all the changes that have occurred, you know in those intervening three weeks or something How does puppet handle the you know node was down for a while kind of issue? Hmm. Yeah, so if it's the case of the node being down for a while I guess you're saying you might have made changes to your configuration at that time Like if the we want to install some new packages or make some system changes and that kind of thing Or is it more of an issue of it's got it. Okay. Yes um, so public keeps track of all the The information in it and your your manifest files, right? And then it uses that to build what we call a catalog So when the next node comes to check in and it'll get the latest catalog And that basically is a representation of not what commands to run But exactly what the system should look like for the attributes you specified, right? So if I say I must have this user. I must have this package Or this service must be running It looks at your current system and then looks at the catalog and then we'll make those changes that it needs to make So I think that's where my my travel analogy comes in right so a lot of times an automated script will say do this Do that and then if the latest instructions or just do that and you haven't done the this yet That might not work so puppet doesn't really have that problem because it's it's modeling the entire system So when the new node boots up and it registers itself with with puppet There's kind of an initialization stage where it makes the system conform and then basically The system can run on and do whatever the heck it's supposed to do is that at least right? Yes So there's a there's a couple ways you could you can run puppet one is where it would check in every so often Where a lot of people will set that to by the default is like every 30 minutes So to get the new configuration and if it doesn't have to make any changes It doesn't have to do anything and won't do anything So if you don't deploy a new configuration file for Apache, right? There's no reason to start restart Apache and then you're nothing happens, right? There's also like a no-op though where you can run it and say just inform me of changes and when I want to make a Change I can make that change manually There's another way of running it which is just to say that maybe I don't want them to check in periodically And I would be in charge of when these updates happen Where I can choose to not run the demon and I can either trick it or cron or I can use the FSH or I can use a tool like funk To go out to the nodes talk to them and say go check in and run now, right? So there's a lot of different ways that can work Also, if you were in a very disconnected environment where you didn't have the demon It's actually also possible to check out your puppet Manifests on the node itself and then just run puppet not not the puppet demon Against that code and then make it conform to that configuration So does puppet only understand configuration files and stuff that are in an RPM package or whatever the systems local packet manager or can I say I Have these files with these checksums put them here Yeah, yeah, definitely those files by itself, right? So there's several ways that can work one is that we can just use the puppets built-in file server to distribute the file So in that case, it would just be a straight-out copy and then the other thing is it's very good at templating files Which is a pretty powerful concept so all those variables that mentioned up earlier that come up from the system like IP address and Fully qualified domain name and MAC address and things like that are accessible within those templates as well as any variables that you might assign to the node or to Inside of the any basically anywhere inside the manifest So and you could do kind of operations on those variables and things like that So the the templating is a really really powerful resource to apply this kind of configuration So if you think of like Say Java app servers for instance, they're traditionally really really hard to RPM package But if I just want to deploy Their XML config files, and maybe I just have to template out a few things. That's pretty easy to do so what about Removing files from a machine some reason something kind of You know I've changed a class for this machine, and so a given file should no longer be there will puppet actually remove it Yes, it can so there's two ways that that works one is I could just say that you know file left bracket file name Colon insure absent and that will make sure that a file doesn't exist And you can also do that on To clean up like directories and stuff or the other concept is puppet out of this thing called purge and Purge is not just specific files It can also work to users, but it can say anything that's not explicitly managed by puppet in this Directory or maybe it's any users that aren't configured in puppet get rid of the other ones So you can kind of make sure that if I'm managing a system, and then maybe it's got shell accounts on it And I know that there's maybe some admins who don't completely understand puppet yet that are doing things And we want to make sure that they're all in my central kind of audible Manifest repository and version control and things like that. I can get rid of those resources that are not listed in puppet So let me ask you a slightly go off in a different direction here. So Why puppet why when you go to a you know a new customer or Say, you know for whatever reason you got introduced to a new sys admin shop or something like that They've got ten years invested in their own script based environment and things like that and it works You know probably well enough for them and so on what what's the argument for saying? Well, you should switch to puppet You know can you can you give them a scenario where they can deploy it in stages for example where they can? Complement their own environment for a while while you know a long-term switching fully to puppet or you know How do you how do you convince people to leave what they they know and have grown to? You know be comfortable with over the years to use puppet sure well First off I think for a lot of people it's not working right? So you have to spend a lot of time developing your own systems management tools And you don't really want to run a full project And there's various features of your tool that's going to get behind as you have to work on other things And maybe there's one guy that really really understands the tool and then he leaves right so I don't really believe developing in-house Infrastructure compared to using open-source applications is Probably a good strategic use of IT I'd rather plan out the really interesting projects the really hard things to do that are specific to my organization But yeah, if you're if you're migrating to puppet you can definitely start small right? So you can just model fly like say you're rolling out a new service You could start by just modeling that one particular service And then as you go on moves to incorporate more and more types of things So like maybe you want to model like just say your application servers right for starting and then maybe you want to go ahead and work on converting some other type of Server, maybe your mail servers or something later just pick one type of resource in your infrastructure Figure out how to model that and then you can move on later. Don't try to say do it all at once Okay, and Again going off in a different direction. So how does the puppet community work? Who who does the development who does contributions? You know how many contributors do you have and you know how do you add as a community because this is one of the Things you highlighted in the beginning there. You know, how do you how do you guys function? Wow. Um, yeah puppets communities actually really huge and it's quite surprising to see an open source project of that size so we have About 1650 song last time I checked and that was several months ago people on our mailing list right for just not just the users list And then there's also a development list and all development on puppet happens on the open so patches go to the mailing list and get approved and commented on and Everybody's free to you know, can contribute and so forth. I think last time I checked to the number of people in the code is Somewhere over 80 right in terms of the actual puppet source, but that's really not an accurate measure right so puppet has this really awesome concept of People being able to share modules which are basically just puppet in excess that are in a kind of a reusable format and Modules are the way we encourage everybody right there code, right? for puppet But the there's lots of people that will create all these ways of saying here's how I manage this particular message bus or this application server Or this application and we'll share them with everybody. So some of that may be actually low-level Ruby code for types but most of it's just puppet language code itself and That community itself is I think what we're absolutely most proud of because it's everybody helping each other and everybody sharing right? So even if you're not say An application developer say you're just doing you know very very large-scale system administration Or you're just a maybe you're just a really really small the university department you can get in there and share Share code with everybody and we kind of sort of united the systems administration community around this one project and I think that's really really neat because we're saving everybody's time that way Well, that's cool. This is actually tremendously fascinating to me because I'm part of the open MPI community and and we have our own ways of doing things and getting people in the community and stuff Like that and is it's sometimes a challenge to get that you know every all the cats running in the right direction How do you how do you manage that? How do you arbitrate? What happens if you know two people have a conflicting idea of How they want to you know accomplish task x in puppet? Well, um, a lot of times I think there's a nice thing about open sources. You don't always have to decide So if it's there's probably lots of different modules for weight different ways people have chosen to manage their IP tables That may be a little better for their organizational configurations And you and you're free to do that, right? So I think that's kind of a benefit of the way them the module system works is that we don't have to choose one way But as far as puppet core itself, I mean, there's there's sort of a mailing list And we all just kind of talk about that information and find out what works and so far. I think that's that's gone pretty well So when handling a large number of different machines under a single infrastructure that maybe you have multiple admins And you want to delegate so I work at a university and there's like a global image It's kind of managed, but then local units can add their own sauce on top Does puppet have a way of saying okay you for these machines can add? You know software on top But you cannot Modify this like master image Well, we're kind of working on that sort of thing So one of our new projects that we've started recently is called puppet dashboard Which is sort of a graphical user interface around puppet So one of the features it has is I mentioned the external nodes tool, which is kind of an internalism It has the ability to assign classes to nodes and variables to nodes So one of the things we're going to be doing in the near future is adding role-based access control to it, right? So so you'd be able to hand to a junior admin Junior admin, it's okay that you modify these kind of variables and add these classes, but maybe you can't remove these other ones So that that gets you a little bit there And then the other thing that can be useful if you're managing a lot of it may not necessarily handle the access control level cases But puppet allows multiple module paths, which means I could keep all my see my core security Information in one piece of version control right and check that out And then my department could have another check out and public could be configured to use both of them, right? So in my site configuration that says these are my machines in my department I would own and manage but I could still use the content from that Save the central IT created, right? So I think that's a pretty powerful concept of being able to say I'm going to extend these these other modules and And use them within my code or not necessarily extend but You know kind of encapsulate and use some of those classes and that kind of thing So you can definitely have different people with that, you know access to different repos and in using that content together Okay, so how do you handle on version tracking like say we update and find out something breaks We want to roll back to what we had before is that built into puppet or do we need to use like SVN or? Yeah, puppet definitely encourages that you use version control so one of the great things about it especially for you know audit capabilities and This goes back to what I was saying earlier about images, right? If you don't really know exactly what's in your images, that's kind of scary Because you don't know if that's a vulnerability and if you have to change it going into change it is very very time-consuming Especially if you have multiple images But with puppet there's a text record of everything and it's really easy to version control that So that's definitely something we encourage doing The other thing puppet has it as a concept of environments, right? So a node can be assigned to an environment like stage or production And then you can keep those in as separate version control branch checkouts So I can try those changes out before And then before I decide to apply the production But yes, you could you could definitely check out go back to earlier version and then rerun that manifest information and in the future I think we're probably going to look at how we can do more of this types of information I don't really think there's necessarily a true rollback necessarily just because you know like what an RPM postscript does for rollback is kind of undefined in a lot of cases But there's lots of ways you can handle that kind of configuration and version control is kind of the best practice for that So we talked a lot about puppet and what it can do and why not? How do you how do you use puppet? You always good to ask the author. Yeah Yeah, so I guess one of the weird ironies about systems management software is a lot of times We don't have the number of systems that a lot of our users have and we learn a lot from them, right? So I mostly use it when I'm basically working on it, right? So That happens and we use puppet ourselves at puppet labs to manage all of our core infrastructure So our website itself is puppet managed everything we have is rolled out with puppet or demo machines And that kind of thing and then you know we use in the course of every day when we're talking with all these customers Helping other people manage their infrastructure. So in effect, we are kind of in everybody's code, right? I guess one of the I mean when I was doing in cobbler We were deploying virtual machines and there's you have some people there I'm redoing a reinstall of 8,000 systems and as a developer the project. I might have really only installed You know 20 or 30 systems that month, but we're still very very involved with all of our users. So So what's the largest number of machines you've seen a single puppet install manage? Yeah, so A lot of times if you have lots and lots of puppet machines You may you want to be able to manage that with multiple puppet masters So I don't necessarily think you didn't consider as one machine is rather than something that you'd want to scale out But in terms of puppet infrastructure 80,000 nodes is you know common, but it's just as common as somebody has you know 20 or 100 So I think that's really nice because you've got a tool that's got a really low barrier to entry to Being able to manage and then as you grow it doesn't really care too much. It doesn't really care about the differences between Pivot or public clouds or versus bare metal deployment and that kind of thing. So it's usable kind of everywhere We have really just throwing around the numbers 20 and 80,000 very casually there. I was Well, it's not a silly as it's how you scale that you have to make some you know careful decisions, right? How are you gonna a lot of those people may be doing you know checkouts and running running things locally? But we're doing a lot of work to see how we can scale the demon to make it go faster But I had to Yeah, yeah, but the I think the key point though is that I guess puppets sort of Getting into all the cool places that you always wanted to talk about and work in here about so you know, we mentioned things like Google and Twitter and dig and Los Alamos National Labs, which we are the case study in our website about So it's really becoming pervasive right everybody kind of says all these other configuration tools that I've used in house. I want to replace and move on to something that's a little bit better well-conceived and There's a huge momentum to everybody kind of consolidating on puppet and it's really really exciting place to be right now So what's the strangest use of puppy've ever seen? I mean it's for handling configuration management of Servers, but have you seen it used on something like cell phones or something strange like that? Not so much yet, right? So you you you have to do a lot of work to get it to manage something like a cell phone But so I don't know if I necessarily have an answer to that question I mean in the past I've seen systems management software using some rather interesting applications from like air traffic control to Off-track core scambling and pretty much anywhere. There's a computer your your software can go and you kind of end up getting surprised with it but puppets kind of Going all over the place. So it's it's getting in in a lot of interesting places and Maybe someday we will be on your your iPad or something like that So as a developer myself I have to ask this question you mentioned earlier that Papa was written either entirely or mostly in Ruby How do you guys pick Ruby? Yeah, so Luke picks Ruby when he was first starting out and he just basically said it was like the language that sort of most made made sense to do it in right so I was originally a Sort of a big fan of both Python and Ruby and I'm a big fan of those dynamic languages because It sort of allows you to get something developed really really fast but and it with But it also has this level of the maintainability that allows a lot of people to dive into the project So if you were to write the application and see the number of audience and number of people that would be able to get into it Would be pretty low, right? is your your cross-section of People that are interested in systems management have a lot systems to manage but also know C is pretty bad But Ruby works well because it's a very very powerful adaptable language It's easy to learn and it's sort of like to me It's sort of like the way Pearl should have been right So it's got objects well thought out and good object-oriented principles and really good functional principles and Still really really easy to learn and in many cases pretty easy to read So I think that's a good choice and open-source projects that tend to choose those sort of languages seem to do really really well when they're You know, maybe not something that's so developer-focused. So if you're building a Java library Obviously, you're gonna write in Java. If you're gonna build a systems management application And do you want your users of that application to grow into developers of that application? it definitely helps if you choose something that's Very has a very good cost-benefit ratio in terms of development, right? So another another question that I ask all all the projects that we interview here in this podcast is what? What version control do you use and why? We use get I've been using get for many many years and I'm really really happy with it for a distributed workflow that works well on with multiple people with multiple copies of things Which is great. So you can have a staging area to work on Code and then maybe some of it gets in and maybe some of it doesn't Really has a really really good patch workflow works great on airplanes and there's no network connection Great tooling around it all the kernel guys developing it big community What's not to love right? So what's the website and contact information for puppet? Sure, puppet's website is puppet labs calm And from there you can get to our doc site Which is doc stop puppet labs calm and there's an email address at the bottom if you want to talk to us Which is just info at puppet labs calm so it's all pretty simple, and I'm Michael at puppet labs calm And we're puppet labs on Twitter, so very easy to remember And there's a user mailing list and a developer mailing list for people can get involved in such on the site There is so once you're on the website just click over to community and there's information about joining both of those lists and definitely look forward to everybody joining and we're also Down puppet on IRC dot freedom net if you're using IRC, so we definitely like to see there, too Excellent Okay, thank you very much. All right. Thank you for having me