 Take it away. OK. Good afternoon, everyone. This one, he seems to have kind of sort of worked. So he tweeted out the actual link because he got it to work. I don't know. I really actually put these up on every presentation I've ever done. It's the first time it hasn't worked, so I'm really sorry. My name is Lyle Kozloff. I'm the MIS director at a nonprofit called Asian Hope in Cambodia. I get L. Kozloff, the most interesting Twitter handle, but it's functional. A little bit about Asian Hope. We're a Christian NGO operating in Cambodia since 2001. In a single word, we do education. That means a lot of different things. Essentially, we have programs for the very core in Cambodia to be significantly richer than myself. And from the very young, our youngest kids are three. And to be not very old, but our oldest people that get to be parents are students, so 50 to 50. So we've got a lot of different programs. But that's not really very interesting in a conference, so I didn't know. Today I want to talk about the problem that was solving, the ways that I failed, and the ways that I succeeded. So why does my presentation title even mean configuration management on the desktop? Why would you do that? Basically, Cambodia, the context is Cambodia's developing nation. We have unreliable power sometimes. We have slow internet. The NGO that I work for, it's developing very rapidly. When I arrived in Cambodia six years ago, we had our internet connection speed share between 40 computers, was 512 megabit. No, not 512 megabit, 512k. 512k. 512 megabit is awesome. So you couldn't even load Gmail. But since then, our fastest site now, we have 30 megabit, which is not bad. It's still shared now between 200 computers, so it's maybe not as good as it could be, but a lot better. Previously, we had only a single site. Now we have five sites scattered across Phnom Penh. And at each of our sites, we have a lot of variation in budget. So our big international school has a big budget. We have servers. We have all the stuff that you would want. Our smaller village sites have three computers and the printer. So we have a lot of variation. And that means that I can't standardize, like, OK, we have this file server, we have this, and this, we have the print server. So it means that there's a lot of differences. So basically, as we've grown, I've needed to get nimble. Things change. Printers break. We have new file shares. We change our strategy slightly. Stuff breaks all the time. Crashes, we have minor explosions. And me as the IT manager and my assistant, the two of us, we have to respond quickly. And we were not able to do that previously. So there's kind of two options. There is the DIY. And it'll take you really far. So when I first started managing our classroom desktops and our labs, about 100 computers, I had some shell scripts that I wrote myself. I had a good base image of the wooddew. And some light customization. And that'll take you a long way. You can do stuff. If you need to add a printer and cups, you can get that set. You can push it out, and it works very well. The formal tools, things like puppet, ansible, solid, you know, all the configuration. You can scale a lot better. So my homegrown tools worked really well when I was physically present. But when I wasn't physically present or if the site didn't have a DNS server or didn't have static IPs for the host size and figure it, it didn't work so well. With something like puppet, we can scale out. We can operate on multiple sites. We have version control. We can check in the things that we do. So yeah, we chose puppet. So like I was saying, Asian-based, it's a plus and a minus, depending on what you're trying to do. For us, it's a big plus. Because we have laptops that roam around, and because we have small sites where we don't necessarily have a way to jump into the site, like we can't necessate to a neutral host and then jump into the other hosts. Asian-based means that everybody comes in, checks in, downloads their configuration, and we don't have to worry about configuring each site. It's really easy to any files that I'm modifying in our base Ubuntu. We can check in to version control. Anything in the puppet manifest that I change, I can check in to version control. And it just makes it a lot easier to share with my assistant and in with me what is changing why. I'll give you a really quick example. I got some complaints from our Cambodian staff that the rendering of the text in the Cambodian language wasn't working quite the same way that it did in Windows. So it should, but Cambodian has vowels that can appear on both sides of a central character and then subscripts and superscripts. So it's a bit complicated. So to type some of the vowels, you have to type the left half, then the right half, and somehow they measure them together. But ideally, you just press 1 and you can hold out. So anyway, the fix is super easy. And it's in some bizarre file in some bizarre directory. It has nothing to do with anything that I can understand. But you change it, it works. If I didn't have version control, I would never, ever, ever remember what the file was or why I changed it. So huge, huge, huge data. And then puppets are really extendable. We can run on a single node. So me personally, I have my own puppet manifest for my own personal configuration. So if my laptop falls in the water, then I can just check out that which my puppet manifests. Now I will new laptop, run it, and then I'll give you my laptop to my exacting needs. But there's also stuff like puppet dashboard and inventory module. So we can see what's on each of the computers. We can get more detailed information about the last puppet runs. We can see which files were changed and how they're changed, things like that. Just like an idea, who, does anybody use puppet now? Or another version of the system you needed? You got a bit of it, plus a little bit of it. Does it? So if I say things like manifest, you guys understand. OK, so a manifest could, it defines the state of your system. So the point of using a version, sorry, a configuration tool like puppet is that you can define the state of your computer and you just describe the state that you want your computer to be in. You declare it to be so the computer handles all of the details and getting into that state. Who does it? Well, thanks, Bob. I understand. One example. So say I really need the command line tool Cowsay. Do you know what it's even called, Cowsay? Yeah, so type Cowsay in a string and then it shows a picture of Cowsay. So if it's really important for Cowsay to be installed on all of your machines or even on your own machine, or on certain machines, you can define a certain class of computers that will have Cowsay. So you just try, you just say, Cowsay should be installed. The agents come and check in. They download that configuration. And they say, oh, shoot, I don't have a Cowsay. I'll go get it. Very, very awesome. There are a few choke points. Normally, puppet is used for servers, which ideally stay out all the time or at least for very long periods of time. You also have an assumption that your server you want to know, make sure your servers are the servers that they say they are and you don't want hackers to come in and mess with your stuff. You don't want them to check in with another secret puppet master that gives them bad configurations. And it also kind of assumes that you know the state that you want your machine to be in. Some of these assumptions don't really work in the desktop environment. So like, nodes don't necessarily stay up on your desktop environment. People shut their computers down. The power goes out because it's a developing nation or sometimes the class is just over. Our biggest problem with using puppet in this way is actually laptops. So we have a whole media center where we have about 26 laptops. Students come, they check my laptop out, they open it up, they get on Google Docs, they print the document, they close it, it's done. They're maybe the node is up for five minutes. And during that time, it's trying to download the configuration file and make some changes or maybe even do an auto-update hunt. Yeah. So that means that things get broken. Our keys aren't necessarily well managed. We clone everything from a single image and means that the keys that are out, the SSHPs on the machine are the same on all the machines. So that's not necessarily desirable. And the state of the machine isn't always well defined. Sometimes in the desktop, like you really want the most current updates, you want security updates because your users aren't, well, because you have users. Like your users are out browsing the web, they're doing stuff and they're like, they're out to get your computer. So things like making sure you have an updated browser and making sure that your security updates are important. But sometimes in the desktop, there's a lot and there's a lot of dependencies and you're running a lot of software, significantly more software than you'd probably be running on the server. So it's not always easy to tell of it, I want this package, this package, this package, this package, this package, this version as such. Yeah, but there are solutions. Some of them could, some of them better. So like I said, the keys aren't well managed. So what I do is I actually just define the hostname of the machine by the MAC address. So our base image comes up, it checks in for the first time, it sends its MAC address and then sets its hostname by itself. So what it means is we clone machines, it takes about 10 minutes and then it checks into the puppet, gets its hostname, we restart it and then generates a new key which it then passes off the puppet which accepts it and then things are synced up and then things happen. So I promised code in my, it was originally going to be an hour, if you want more, like I said, state of the machine isn't always well defined, sometimes it's really hard to know, like laptop shuts down in the middle of an update, that means you have to recon, like de-package, that's as you know. So this is a technique I just found where you have this file, we just call it update initiator, it doesn't actually matter what's in the file. But what's important is this piece of code where it does an update disk upgrade whenever this file changes. So that way, if I know that I want all of my notes to do and have that set of my notes to do updates, say I go in and I actually turn on all the library laptops and let them get on the wifi, I can change this file and then they'll start doing their updates automatically. Skeptical, okay. Yes, more specifically we have, we actually have a local app cache, so all of your notes are, they're not actually going out to the internet, they're going to a local mirror of your repositories. So maybe it's not necessarily the best, but I have 26 laptops, it's at 15,000 machines. How do you update this file in your routine? That goes with our Dr. Warner movie. So this file, you see the source, is it's actually inside of it. So I just go in, I go into the public master and I change the contents of this updated machine file, which triggers this refresh. Which triggers the upgrade. Yeah, this still happens even in the best of cases, like notes will shut down and build updates, so there is some babysitting that happens. I want to take questions because I think that'll be more useful than you're talking about. But essentially, using configuration management to be now, I'm really doing this infrastructure that can respond to changes very quickly. So I have a request for software, or if I get a bug report, I can push the changes out to all 300 computers at five sites within a couple hours. Whereas before, I would have to actually travel and do something or it just took a lot longer. The bad is there is still some babysitting. I don't think that they're ever gonna be desktop support or there's not some hand-holding babysitting. But yeah, I really do want to take questions. That's why I left a lot of time. Do you have any questions? And between you said that, since you are copying everything from the same image, the keys are the same, right? Yes, initial. How do you manage that? As soon as you try to register the public master, it's going to deny that this public see a little, because of the compression, you know? Yeah, so there is a very, very not recommended way of doing things where you can auto-sign every key that you're posting. So that's what we do. But on the desktop, it's not such a high-security environment. Like if somebody gets my desktop configuration, there's not any secrets in there. Like actually, it's a learning environment, so I want it to be as open as possible. So I really actually, I hide stuff on the file system for my students to find. Like Cowsave literally is installed on all of our computers, just so that I can pull it up and be like, hey students, watch this, control D. Cowsave, you guys suck. Ha, ha. I wish I could save and I did a really good job of comparing them all. It was, public had an open source version. At the time that I chose it, Ansible was still fairly immature. Salt was fairly immature. And public just had a lot of resources. And then I liked the idea of having paid support. At that point, Ansible didn't have any paid support. It was just like a community project. So I didn't really delve in and really seek out the best. Honestly, at this point, I'm considering switching over to Ansible because I think that the way I have things configured now, it may be better for us. It's definitely developing. I'm not from HUBBIT. If you're looking towards the way to Ansible, how do you propose to do agent-based Ansible? Yeah. Or is it not that agent-based? Actually, some, to somebody at a session this morning, saying that they use Ansible kind of interestingly, where they would have each of their nodes go and get a configuration file. So they go to the configuration file and then when I float away themselves. So, HUBBIT-style, like Ansible. But then it means that even if you don't know that you don't need the entire configuration, you are still going to download the entire of the files of the configuration, which you might actually not. Yeah, that's true. But if you, maybe you had to check into your game repository, then you don't look out for the chains. I don't know. Somebody said something about it this morning Yeah, it's all kind of the game repository. So it would get pulled. You just be down in the div, and the div is probably zipped up. So it's pretty small. Unless you stole five, three files in a game, any of this group, any of these. Well, actually, I can tell a small story about that. Binary files are a big problem for us. We have, we do have need of Windows XP every so often. So I have a Windows XP image that I needed to distribute to 100 plus clients, and Puckett doesn't do that well. So actually I wrote a Puckett module to install transmission CLI, grab a Torrent file, and then all of the nodes torrented it from each other. And I say how many distribute the entire forked by the image in the time that it took one. No, just download it, it was pretty cool. It was really cool actually. It's the same strategy that Facebook used to deploy the apps. Really? Yeah. Do you use a Puckett module in the environment that's nice to download? No, no. Windows works very well. You can write your own code, show it to them. Do you call it? Is there a Puckett module? No, I haven't really, I don't think there's a Puckett module. We have zero manuals. The Windows machines we have are very limited. We have our front office staff. So there's three in front of one of our sites. We have one Windows machine, sort of like a token Windows machine, and each of our sites, and then four more for our app. So a total number of Windows machines out of our 300 versions. Those ones I just have to submit to the suffering. Okay, all right. Really? This is my end. Any other questions? Oh, no. Any other questions? No. Okay. Thank you. Thank you for coming early.