 Okay, hi, I'm Sam Hartman and it's my extreme pleasure to introduce Russ Albrey who's going to be talking with Russ at Stanford University. And he has been involved in Debian for a number of years, has been involved in Lentian and policy work. But today he's going to be talking to us in the context of the enterprise track about what Stanford does. And at the, well, if we started off this morning, we were talking about how there are a number of organizations that have managed to build really impressive enterprise infrastructure out of Debian. We have a couple of case studies for you today, understanding how that works and some of the lessons to learn. And Russ is the first case study from that, and I'm very much looking forward to hearing it. Okay, thank you very much. So I work for Stanford University, and for those who are not familiar, Stanford University is a large private research higher educational institution in the United States in the San Francisco Bay Area. We're generally considered to be roughly the same class as the ID League in the United States where usually, that's usually referred to as being ID plus. And it's, we have a medical school, a law school, a graduate school business, that standard sort of thing as well as a full university across the wide array of disciplines. And about 25, 30,000 students, so fairly small in terms of size of the student body compared to a lot of institutions, but large enough to have some interesting scaling problems. So I work for IT Services, which is the central IT department at the university. And in this particular, I work in the infrastructure delivery group, which is the organization in IT Services which runs campus computing infrastructure for the university. So we maintain all of the university authentication systems, Kerberos, Weboff, Shibboleth, we maintain email routing and mail delivery. You know, we're considering outsourcing it like everyone else is, but for the time being, we have all that in-house. We maintain all of the central web services including the home page, the machines that run the home page, not the home page itself that's done by university communications, but the infrastructure behind it. The distributed file system, we use AFS as a distributed file system like a lot of other large universities. We serve the Stanford home page out of AFS. We also are taking over the account provisioning side. Basically, when someone new comes to the university, we set up their accounts, grant them services, everything along those lines. That was previously done by our administrative systems department, the same people who ran PeopleSoft, and we're taking that over in the infrastructure delivery group. We also run Active Directory. We also maintain the campus windows infrastructure inside the same group. We also have a sister group which does system administration for hire. They do system administration for hire both in the university in general, other departments that want somebody to run a Unix box for them, or a Windows box, but also for other groups inside IT services. For example, they run the operating system underlying our DNS and DHCP servers, which are actually maintained by our networking group. All of the things that I just mentioned, with the exception of Active Directory, of course, and the other Windows infrastructure, and with the exception of the actual user mailbox servers are all running on Debian across the board. It's not just an alternative platform for us. It is our platform. If we actually do outsource email, currently we're using Zimbra. Zimbra likes to run on Red Hat. If we actually outsource email or change how we're doing email, and those mailbox servers go away, the infrastructure delivery group will be dropping support for Red Hat, and Debian will be our sole supported platform. We used to be a Solaris shop. We moved wholesale to Debian from Solaris about nine years ago or so. So, enterprise. This is the enterprise track. What does enterprise mean? It's one of those sort of fuzzy words that people throw out, and it's not entirely clear what people actually mean by it. What I think of as enterprise is you have a large user population. It's a lot of problems that only happen at scale. You have a diverse array of services. It's fairly easy to run. Well, relatively speaking, it's easier to run 10,000 systems that are all part of one giant batch computing infrastructure than it is to run a whole bunch of machines where there's five of one thing and two of another thing and three of another thing, running a bunch of different things, even if the total number of machines is smaller. So, to me, enterprise means you're scaling across a wide variety of different things that people want to do, different services that you're running, and you have to figure out how to integrate all of those systems together so that they work together seamlessly. It's also about ubiquitous, invisible, and foundational services. If you're running enterprise computing, then you are the people who no one knows you exist if everything works properly and everyone yells at whenever anything breaks. At this point, it's very similar to being the networking department. You basically don't want anyone to know that you're there for most of what you do. You have diverse client platforms and requirements. Stanford University being a university has basically absolutely no ability to mandate anyone do anything ever. There's no such thing as a common desktop environment. There's no such thing as a common client platform. There's no such thing as a common server platform outside of individual groups making their own decisions, and you basically need to support everything that anyone comes up with, and also anything that any company decides that they want to give to a university for free because universities love getting free things from companies and then trying to make them work. So you have a very diverse set of server and application platforms and requirements, and that does include Windows. I mentioned that we also run Active Directory inside the same group. I wanted to take a few moments to talk about this because Windows is like this thing that in the Linux community you never want to talk about Windows. Enterprise computing means that you work with Windows. It's out there. It exists. The entire university is not going to not use Windows. So some of what I want to talk about in this talk is strategic. If you are working in this kind of environment, how do you get Debian in the door? How do you get convinced people that Debian is actually a platform on which you can run this stuff? And in that environment, Windows is not your enemy because if Windows is your enemy, you're going to lose because the way that it's structured and the way that the things that people are familiar with, it's very difficult to make that argument when you go into it adversarily. Mutual disrespect doesn't help anyone. And it's really worth earning a reputation for being passionate, but fair. It became so much easier to integrate with our Windows environment and even to take services and run them on Unix when they were appropriate to run on Unix, when I also would go to meetings and advocate for commercial software that the university wanted to run, if that commercial software was mostly targeted at Windows, I would advocate that it be run on Windows. If you do that sort of mutual respect and trade off with the people who are running your Windows infrastructure, they will be on your side. It doesn't have to be that sort of an adversarial situation that I've seen in so many different places. And from a perspective of using Debian with Windows, Windows has a really bad rap for using proprietary protocols, proprietary structures, and a lot of that is deserved. And I won't say that they don't do that. But they've also made a lot of progress, particularly with the Active Directory environment, towards using standard protocols. And a lot of the things that you hear about how Windows does not work with the rest of the world are no longer true, or are at least only marginally true, and you can make them not be true. So Active Directory is an LDAP server. It supports regular LDAP connections. It supports Kerberos-Cassapi Binds to the LDAP server, or it supports Password Binds to the LDAP server and uses TLS. Active Directory is a regular Kerberos KDC. You can point any regular Kerberos client at an Active Directory server, and it works great. I'll talk a little bit about how we run Kerberos. But if you have an Active Directory already, and your organization doesn't want to run a separate environment on Unix, you can use Active Directory as an authentication server for Debian clients and for Debian servers, and it works just fine. And increasingly, web services is kind of the direction that everyone is going towards integration, either SOAP or REST. And Windows has reasonably okay support for that, certainly as good of support as you're going to find in most, in any other sort of large application that you have to interoperate with. So for example, Stanford University has two main authentication environments at the university. We have a Kerberos realm that runs on Debian systems running Heimdall, which everyone gets an account in, and we have an Active Directory domain in which everyone also gets an account. We automatically synchronize, we treat the Heimdall realm on Unix as the primary realm. The Active Directory is essentially a slave, and we replicate everyone's accounts and passwords from the main Heimdall realm into Active Directory. And we do that through the Kerberos password change protocol and through LDAP. All the software that does all of that lives on the Heimdall servers as plugins and as various things surrounding that. I wrote some of it, we contracted out some others to be able to write some of it. All of that is available as free software, what we use. It's not horribly easy to use somewhere else, but it's all available. And we just use standard protocols that talk to Windows, and Windows is perfectly happy. So it's part of that thing of how do you get people to accept Debian as an operating system that works in this environment. It's really important to say, you know, we're not an alternative to Windows. We can live alongside it. We can interoperate with it. And our experience at Stanford has been that Linux and Debian in particular are really good at middleware, which essentially in this context means they're very good at taking a lot of small tools and gluing them together, building ad hoc databases for something and sharing that data with something else and using network protocols to push things around. And so what we found is that when Windows becomes one of the things we can push to, then that infrastructure starts looking really attractive. And we end up doing things on Debian to control the Windows environment because it just becomes so easy to push things around and that was a good place to start pushing from. So another thing about definition of enterprise. An enterprise environment is a conservative environment. And because you're that underlying magic, right? You're that underlying stuff that everyone uses and that needs to never go away. So infrastructure problems can break your entire organization. But infrastructure problems are rarely the key to making your organization succeed. Stanford is about teaching, learning and research. Stanford is not about running mail servers, not authentication environments. So the stuff has to work, but people don't pay attention to it when it's working. Organizations are very leery of solutions that only one person understands. And so that's the biggest challenge that I think you face when you try to bring Debian into an enterprise environment. Everyone's afraid that Debian is going to be that thing that the person advocating Debian knows, but that no one else is going to be able to know. Known quantities are a lower risk. So Debian, who's heard of Debian, right? I mean, they haven't really thought about it. They haven't heard of it. They've heard of Red Hat because it's on all support matrices. They might have heard of Ubuntu. It's not clear that they've heard of Debian. Did they read about it in CIO Magazine? Probably haven't read about Debian in CIO Magazine. So those are the, that's the primary challenge that you face is how do you get them to accept Debian as part of the conversation. So Debian has a lot of features, but you have to be able to have that conversation. And in that type of environment, I think you're largely competing against other versions of Linux. There's a different conversation you can have if you're trying to get it in the door of a largely, of a largely Windows shop, but I think most large scale enterprises, they already have some Unix and they have some Windows. And that's just a fact of life. And most large universities are that way. I think most large corporations are that way. So what you're, what you're largely, you're trying to get into a conversation where the alternatives are probably Red Hat or possibly some old style of Unix is. And why is Debian interesting? What statements can you put forward about this is something that we should be looking at. This is something we should be evaluating. What we found is that the large software repository is a huge selling point. And this may be somewhat unique to a university environment, but we, we have a lot of people who want to do computing and research and need supporting tools that run on Unix environments around campus. And Debian has an absolutely fantastically large repository. And a lot of the people who are doing science at the university and who are doing research at the university find that things that they want like Biopurl or the statistical computation software or R, which our statistics department absolutely adores, are already there packaged ready to go on Debian compared to a lot of other distributions there. That's a much harder proposition. The other thing for central servers, I have to say that Debian's really cycle is perfect. And I know a lot of people complain about the release cycle and a lot of other environments, but when we're talking about core infrastructure compute servers, we are extremely happy with the current Debian release cycle. And it was a major selling point when we had the original discussion of whether we were going to run stuff on Red Hat or on Debian. Fedora releases way too fast and forces you to upgrade. Red Hat Enterprise Linux releases way too slow. And the, the point at which Debian is hitting is for us the absolute sweet spot. That's particularly the case with the robust backports.org repository where you can selectively pull things from a later version of the distribution. That's a huge feature to point out because that really is just not available on other platforms. You can, with Red Hat there's of course millions of third party repositories and you can pull in things from all sorts of different places of varying quality. But when it comes to something that really does work inside that same environment, backports.org has been a huge selling point for us because you can pull just the things you need and not everything else. And the other thing is it's already integrated. A lot of the integration work has been done. I've been able to sell this to technical people by pointing out Debian's Apache configuration system for example with its modules, with its module and site handling. And the other point you can always make too is that Debian is just not that different. At this point Linux is basically Linux. That's a big concern that a lot of people have, is can we hire people who know Debian. And I mean the fact of the matter we all know that. It's yes. But there was a question. Would LSB standardization be a selling point? I can repeat that. I'll repeat the question. So the question is would LSB standardization be a selling point? For us I'm not sure. So LSB standardization is not something that's going to hit the level of attention of the people who make these decisions. So it's mostly a question of will LSB give me things that I need in order to be able to sell it to them at a higher level. Not clear. I'll talk about that a little bit later with commercial software which is the main place where that helps. So what I would sell as the argument for Debian is practicality is what matters the most. But you have to demonstrate that it's practical. So does it work? Is it efficient? And efficient is really important in a lot of different dimensions and free helps. So Red Hat Enterprise has a license and that license actually ends up being expensive if you run a lot of it. That helps. It helps to say Debian is an alternative that you should also be looking at. And the other thing that helps is that large packaging repository. That large repository of existing packages is a huge huge efficiency gain. So what you can, Debian you can sell as look at all the work that other people are doing that we don't have to do. That otherwise we would have to do locally. And then the main thing that Enterprise cares about even if they're not aware that this is what they care about. And I think this is particularly true of university environments is can it build flying cars. So the idea behind what a flying car, technology at the university is very heavily driven by shiny. If somebody, so I you know I have had this conversation with people. I want an Apple iPad. What are you going to use it for? I have no idea. I'll figure that out when I get it. And that's very true in a university environment. And it's Stanford wants to be a world leader, a world leading university. And that means that Stanford has a lot of reputation put into the idea that we are ahead of people, that we are, that we're on a cutting edge or that we let our students be on a cutting edge at the very least. And so you get a lot of technology challenges at a university. And I think that this is not uncommon in a large, in many large scale enterprises. And they come from different directions. They come from faculty wanting to do something that's cutting edge research. They also come from administration wanting the latest toy, the latest iPad, the latest Blackberry, that sort of integration point. Flexible and open standards are the key to this kind of thing because most problems are integration problems. So what, so rather than looking for things that say they on the, on the tin that they support Debian, the goal is to get everyone to change mindsets to, does it say on the tin that it supports these open standards? You know, throw them some keywords and evaluate things from that perspective and start educating people about the fact that computers are increasingly interchangeable, servers are increasingly interchangeable, because everyone speaks the same protocols, Windows and Linux speak the same protocols. So Debian speaks those protocols, it's a perfectly decent implementation of those protocols, which means you can have a different kind of discussion than about what actually shows up on a support matrix. What's shiny, what comes in shiny is outside of your control. So it's really important when you are talking about how to deploy something like Debian and Enterprise that the thing that Debian gives you is flexibility. There's all that software out there already and you're trying to, and you have all those tools available to be able to embrace the next thing that comes along. So then Enterprise has run in many environments and this is another key thing that we found about talking about how, how you make the case for Debian inside, inside this kind of an environment. You don't, if you haven't, if you if the argument comes about what one environment do we run, that's not a good place to have your discussion about Debian. Because that's not Debian's strength. Because Debian doesn't have the vendor relationships that let you run Oracle on it, that let you deal with all of these things that are out there in the broader environment of your Enterprise. You can run the core infrastructure that way, but you need to be able to interface with all this other stuff which is out there. And not all that stuff is going to be Debian. So if you get into the discussion of what is all of our stuff going to run on, the only answer that anyone is going to be able to come up with for that is Windows. And just in practice, that's where that discussion goes. You don't want to have that discussion. You have a different discussion, which is that different applications are targeted to different environments. If you can find a way to run those environments inexpensively, then you can deploy applications on the environment they were written for and your life becomes much better as an Enterprise system administrator. So you have to make that argument. That's a difficult argument because running multiple environments are expensive. And the first thing that the higher level management always says is why can you not run just one thing? And the way we made that case at Stanford is, yes, that would be less expensive, but Debian does all the stuff that we need. And we would have to reproduce on the other platform. So the expense of running the additional environment is less than the expense of reinventing that environment on the stuff we could standardize on. So you have to know where Debian is a fit and where it isn't to make that kind of the other argument. And inside an enterprise proprietary software support contracts is the hardest fight. And that's where LSB in theory could help with that, where if the proprietary software said we will run on any LSB blessed platform, then that becomes easier. But for the most part that's not where they're at and that's not where their marketing is at and that's not where their support matrices are at. And if you want to run Oracle at this point, you end up getting stuck with Red Hat because if you run on something else, Oracle won't talk to you unless you want to run on Oracle's own version of Linux. Or on Solaris. But well understood commodity services are much easier. So the place where we introduced Debian was mail service, authentication service, LDAP. And those are real well understood commodity services where all you care about is a protocol implementation and you don't care so much about a particular product. We have a much harder time with something like the university core financial system or the university HR system, where people in that space are not thinking about the kinds of advantages that Debian has. And integration of obscure free software is the place where you just have a slam dunk argument for Debian because that's what Debian does really, really well. And that's the argument that in any kind of research institution, whether it be commercial or educational, you can really talk to your researchers and say, you know that stuff that's kind of cutting edge or the stuff that somebody wrote somewhere else. It's probably already in Debian or at least there are people who are looking at putting it there. So you have to keep the overhead low. You have to figure out how to run Debian cheaply so that you're not showing up on the radar as, oh, we're investing all of this money in this alternative infrastructure. And thankfully Debian has a lot of tools available to do that. And the other key part about that in general is when you're selling a platform inside an enterprise, don't say no to question the things that people ask you to do. There's going to be things that people want you to integrate with that you think are absolutely horrible. And protocols that you really don't want to use. You can shape how integrations are done, but it's very important that the first thing you say is not dead. That isn't going to work with Debian or that isn't going to work with what we've already got because it takes a conversation in a place where you don't want to go. So now I'm going to get a little bit more into how we run Debian and what we do with it. Enterprise means customized. When you have a large site, you are not going to be able to run the stock anything. That's true for us for Windows. It's true for us for Debian. It's true for us for Red Hat. There are going to be local changes that you need to apply. And this is a great space for Debian because Debian makes it very easy to apply local changes. The fact that you can build every package in the in the in the in the entire distribution and you know that it's going to build. And if you make changes to it and it doesn't build it's your fault for your changes not because the package was broken to start with. That's really big. But so the enterprise has a different has a different problem than stock Debian is solving and it has a different problem even than Debian edu is solving. Although Debian edu is doing something very similar to what an enterprise does. We can share a lot of packages with the main part of Debian and we can share a lot of infrastructure with the main part of Debian. But we have to be able to customize it because running a local and running an enterprise environment is very much like maintaining your own distribution. You are essentially a derived distribution of Debian. You're going to end up being a derived distribution of Debian. You're not going to be just Debian. And to what extent you want to label that put a name on it. That's a different question. We don't. We actually don't. We just say it's Debian. And here's an extra repository to point at and some stuff we install. But it you are essentially doing the same sort of work that Ubuntu even as the really big example is doing is just on a much smaller scale. So to take some examples of that. We a long time ago, Stanford University made this decision that when you send email to an address at Stanford.edu we should be able to support putting any kind of punctuation in between the actual alphanumeric characters of your email address and it shouldn't matter. So I'm RA at Stanford U and I'm R dash R dash A at Stanford U and I'm R dot R dot A at Stanford U and I'm R under bar dash under bar, you know, et cetera. Kind of a questionable decision at the time but it was made a long time ago and we don't know how many people are relying on it. Postfix doesn't do this out of the box. So it's very difficult to make Postfix do that in combination with our environment the way that we do mail routing. We have a custom patch that we maintain to Postfix, which is, you know, all about 10 lines that just strips out the strips out punctuation before it does an LDAP map lookup. And but that's the kind of thing you run into. So we have a custom Postfix package that we have in a local repository with a version number that's slightly higher than the version number of Postfix and Demian Stable and we update it when we update to the next version of Stable. There's a one line patch to Cyrus Sassel that lets you instead so normally Cyrus Sassel is a the Kerberos part of Cyrus Sassel assumes that it knows what your host identity is. It loads that identity when it starts up and then you can only authenticate to it with that identity. LDAP servers are the one that run into this trouble. When we have load balanced LDAP servers we want to be able to contact that LDAP server through several different identities. There's a one line patch to Cyrus Sassel that lets you do that. It's not in Cyrus Sassel upstream although we've asked a few times and therefore not in the Debian package for reasonable reasons. This is the kind of fork that we want to make go away again eventually but what Debian lets you do is we have a version of Cyrus Sassel that is available only on the LDAP servers. We put it into a special distribution that only the LDAP servers add to their sources.list file and deploy it on the LDAP servers that has this patch. Another example of open LDAP we do a lot of work with open LDAP and we have a support contract for open LDAP and therefore our support vendor says you shall run this version of open LDAP because it's what we can support. It's not necessarily the version in Debian it's built with open SSL instead of the new TLS. We have to maintain this thing but we don't have to start from scratch. We start from the Debian open LDAP packages and then apply some additional changes from our support vendor. And again, we put this into a custom repository which is only in the sources.list file on the LDAP servers so that we don't pick up these new LDAP shared libraries on other systems and potentially break other things. There's a lot of stuff like that that happens in an enterprise and having the infrastructure to support that is very important. So when you're running Debian Enterprise one of the very first things that you would want to do is set up your own local Debian repository because you're going to have both local packages that you don't want to put into Debian there are local things for local customization, local scripts and you're going to have these little tweaks to packages that you need to put somewhere. Don't skimp on setting this up. It's worth investing some time and advance to figuring out what the right tools are how you're going to do uploads to this package to this repository. We have two local repositories. One of them is internal only to the infrastructure delivery group and that's where we put packages that are particular to our servers and it's where we put things like our hacked version of PostFix that could break someone else's world. Somebody didn't install that but mistake. So we have one repository that we only put on our servers and we have another repository that's available to everyone at Stanford University where we put additional tools additional packages that might be useful to any system running at Stanford. We're using DebArchiver right now to do that repository because that was the best of read when I set it up which was like five years ago. We're moving to ReprePros as soon as I can find some time. There are other tools out there. I've been pretty happy with ReprePros in terms of repository management. It's one I'd recommend that you start with. You need good package build mechanisms and this is another place where I'm really happy with the infrastructure that Debian has. Go find Cowdancer and go find P-Builder. Learn about how they work set them up because your systems are all going to be running Debian stable and but you're going to eventually want to move to the next version of Debian stable. You're going to want to build all your packages in advance on the new version. You also want to make sure that all the standard things we care about in Debian are all the dependencies satisfied or the dependencies written properly all that kind of stuff. Building inside of Charoot we all know this for uploading Debian packages is a nice way of making sure of that and it's nice to not have to run multiple systems to do just for builds. You have one system with a bunch of Charoots on it, Cowdancer, P-Builder. It works great. Supporting multiple releases right now is really challenging and supporting Ubuntu as an instance of that is challenging because... Not impossible. Not impossible, no, but challenging because you ideally want to rebuild your local software on each of those different distributions. This is a place where Debian could really be helping the enterprise more. We have an infrastructure for doing this with the BuildD network. It's very hard to figure out how you install that inside an enterprise and so that you can take a package, upload it for stable, rebuild it for old stable for several different versions of Ubuntu automatically and that's the kind of thing that you want to do in an enterprise because if you make people go individually build the package on each different architecture they don't do that. So for example we have a ton of stuff in our repository that's built only for I-386. Just because that's what we usually run on servers we're gonna move to AMD64 across the board but inertia is a powerful thing and because people are usually building the software on an I-386 box they're gonna install it on then they don't think to go build the AMD64 binaries which is a separate step. On package everything. So what we do inside the infrastructure delivery group is we have this rule. If it is not a configuration file it goes into a Debian package. We do not install anything in user local. We do not install any scripts out of puppet. Even little scripts unless they go into Etsy and they're a configuration file they're like a cron.daily snippet or something like that. They go into a package. This is overkill. You don't have to do this. There's nothing that prevents you from installing scripts into user bin or user local bin or somewhere else using a configuration management system. However, everything packaged means everyone learns how to package. So if we have a standard that says you have to put it in a package then everyone in my group who needs to put something on a server which is everybody has to deal with the Debian package even if they don't wanna learn how to do it. And that was extremely helpful in cross training and in getting everybody up to speed at least on the basics of Debian packaging. Because Debian packaging is actually kind of hard. It's not as hard as it looks when you start but it has a bunch of different files, a bunch of different ways to do it. The documentation is very comprehensive but also very long. And so there's this big resistance point and you have to help people pass that resistance point. And that's where it's really important to have someone inside your organization who loves Debian who really cares about it. Somebody like all of you who came to this conference exactly the kind of people who is needed inside the organization to say it's really not that hard, I've been through it before, let me sit down with you, let me walk you through it, let me show you exactly how it works. People do package on stable and people test their packages on stable. Inside an enterprise environment a lot of people use stable. I just wanted to make that point in this talk because there's sometimes discussions in Debian Devel around, well it's not really important to worry about people who are packaging stuff on stable because they shouldn't do that anyway, they shouldn't do builds on stable anyway. All of that's really true if your target is uploading to Debian. But those of us out there who are using Debian in enterprises we are also packaging and we are also using those tools. Stable is really important to us because that's what we run for our servers. We can't run unstable, we can't run testing on servers. So that people use the tools that are in stable. And so there's a lot of things where why doesn't Debian change faster? Why don't the tools move forward faster? A lot of us in enterprises are going to be stuck with whatever shipped with stable because the next time there's a new stable release we update all of our stuff but until then that's our target platform. Installation, so FAI is great. FAI is absolutely top notch. We love it. Fully automated install. So FAI is Debian's version of Kickstart or Jumpstart or something along those lines. You get, you pixie boot a system, you load an NFS route and you install software onto the box. We use FAI only to put an initial operating system onto the box. My understanding is that Debian install work has picked up a lot of capabilities in this area and you probably could just use it. We've been using FAI for many, many years and are very happy with it. You probably aren't looking at changing. There's an argument that's coming and in particularly in enterprises right now which is why do you ever build machines anymore? You build a machine once, now you have a VM template, you have a virtualization environment, just clone it. I don't like that approach and we have intentionally avoided that approach inside Stanford for right now. My concern about that approach is that I really like the cleanliness of a clean system build and I particularly like the fact that all of our servers, everything about that server is in either the build process or the configuration language that we use. We use Puppet as a configuration management system. So if it's not in FAI or in Puppet, it's user data that we know that we have to back up. For the most part we use AFS to store data so it gets out of the box and into a distributed file system. The advantage of, so I actually don't even do, we normally don't even do disk upgrades on servers. That's a great feature of Debian, Debian upgrades. That's something that Debian can say that almost no one else can and I don't want to belittle that but we don't actually use it. Our rule is that when we go to a new version of stable, we wipe the box and rebuild it from scratch because it does two things for us. One of the things it does is now we really know that everything that was on that box is either in the build system or in Puppet because we blew away everything else and we got it all back. The other thing that that does for us is that it cleans out any cruft, weirdness, the half-installed packages, things that somebody installed at one point and then decided to remove and didn't completely get rid of, old configuration files, transitions, et cetera. So it's something to think about, not everybody might be as fully religious about this as I am, very understandably, but that's kind of the approach that we take. When you're talking about build issues you do need to think about how you're gonna key your systems. And I'm happy to go, I don't have time in this talk. I'm happy to talk about this at length though with anybody who is using Kerberos and who's looking at how do I install Kerberos keys and Kerberos key tabs on systems. Stanford actually paid me to develop something to do that called Wallet. It's packaged but not yet in Debian. I'm hoping to put it into Debian. It needs to have some Stanford-specific bits cleaned out a little bit first, but it's almost there. And we use that to do all keying. That puts Kerberos key tabs on our systems. It also puts SSL private keys. It puts SSH private keys that we need to retain for some reason. It puts database password files. All that kind of stuff is done through Wallet which is a centralized system for storing secure data that you wanna pull down on individual servers. Remote console, this is just standard enterprise stuff. You really want remote console. You really want remote serial console in all your systems. Debian is very good about this. All the Linuxes at this point are really pretty good about this. And you do need a configuration management system. Puppet BoF this morning. I highly recommend Puppet. We have used a wide variety of different things over the years, including things I have custom written specifically for Stanford. We switched away from all of that to Puppet and everything we do is in Puppet and we're very happy with it. Internal documentation inside for this is extremely important. And this is also extremely important when you're convincing people that retaining this Debian stuff is good. That it isn't going to hurt with hiring. That it isn't gonna confuse people. That it isn't going to be a solution that only you understand. So Debian offers lots and lots of options. It's one of the features of Debian. It's one of the things that makes us all like Debian. There's a lot of freedom in Debian. There's many ways to do it. Most of them are packaged. The ones that aren't are already in the WMPP system. And that, but inside the enterprise, you don't want a lot of options because lots of options are paralyzing. Packaging is really hard and people get very confused by it. And if you point them at Debian and go, oh, you can use any of this stuff to package. It's really just this thing that calls the de-package build package and there's CDBS that you could use or you can use DebHelper 7 or you can use NotDebHelper 7 you're gonna lose a lot of people because they don't, the packaging is not something that they care about for itself. They're trying to solve a problem because I just told them that in order to put that script on the system, you have to make a Debian package and they didn't wanna do that. So they're trying to find the fastest way to get from point A to point B, not care about how the packaging system works. So having that internal documentation pick one way to do it and write it down. And then they can just follow that and if they wanna learn more, then point them at where they can learn more. But the first few times, they're just gonna walk straight down those steps because that's what they wanna do. Packaging teams inside Debian can really help out enterprises here and I just cannot exaggerate this too much. We have outside of Debian Perl modules. We have outside of Debian Ruby stuff. We have outside of Debian Python stuff. It's just not packaged in Debian for whatever reason, sometimes because it depends on commercial software, proprietary software, nobody's just gotten around to it. If I can say I have this Perl thing, how do I make a Debian package out of it? I go to the package Perl teams website and they have a documentation that says here's how we make Debian packages and we just walk right down it, that's perfect. Because when I'm inside the enterprise, I don't care about packaging Perl modules for itself. I have this thing, I wanna get on a system and I just want somebody to give me a cookbook that says here's how you do it. Some packaging teams in Debian are really good about this. The Perl packaging team is one that's very good about this. It's a large team, it's been around for a long time. It's had a lot of time to write documentation. Other language specific packaging teams, the documentation is not as good. If this is something that really helps, think of your team documentation inside Debian as not just being documentation for your team, but remember all these enterprise users of Debian out there who want to essentially make a few derivative bits from Debian are going to use that documentation too and it's invaluable to them. Then there's also a bunch of tools that Debian has for doing package checking, repository analysis. Can you find all the inconsistencies, run Lintian across the entire archive, all this sort of thing. Most of those are difficult to run outside of Debian. Most of those rely very heavily on Debian's internal infrastructure, which makes perfect sense, that's where all the data is and it works great for packages that are inside Debian. Another place that Debian can really help the enterprise figure out how to make some of that stuff installable at Stanford so that I can just aptitude install something on a server and point it at our local repositories and get all that analysis or at least some subset of it that makes sense for me. Because then I have a quality control check over my internal stuff as well. So I've talked a little bit already about places that Debian can help. Here's kind of a rundown of a bunch of them. Java, awesome that there was a Java track this year. I was really happy to go to that. I talked to someone at that track who said, wow, I didn't realize that there was this much interest in Java and Debian. There's a ton of interest in Java and Debian out there in people who are running Debian for the enterprise because we have software that's written in Java and we want to run it. And it's been a big problem on Debian for many years because we really couldn't ship a JVM that anyone would actually use. For better or for worse, most of the stuff that's written in Java has only ever been tested with the Sun JDK and as much as it might work with other stuff, as soon as anything breaks about the software, once it runs on a different JDK, the JDK gets blamed. Whether it had anything to do with a JDK or not, it's one of those things where, again, you have to get into the conversation before you can start talking about how to change the world. Now that that stuff is available in Debian, there's a huge amount of progress made in packaging Debian stuff, in packaging Java stuff inside Debian. And the more of that, the better because enterprises run Java. We have a lot of Java stuff. And the more of those problems can get solved, the better. One of the big bits that's missing right now is deploying Java. How do you package a Java web app inside Debian? You've got something that runs under Tomcat. What's the Debian package look like? How do you put it together? And that's the kind of thing that enterprises care a lot about, that kind of thing. I mentioned language package teams, documenting how to package stuff for their language. Something, package build support, like the building network, built into the repository handling stuff, like reprepro. If I can upload a source package and a binary for I3D6 and have the same thing happen that it would happen in Debian, where it just transparently kicks off build jobs on my other architectures and my other supported Ubuntu platforms and maybe testing and maybe old stable, that would be fantastic. Package checking for local package sets, tools to monitor and report Debian specific things. We have some locally written stuff at Stanford that we should figure out how to make available that runs on all of our systems and just basically looks, what package are installed in the system, what versions of those packages, what was the last time that we upgraded more than three packages, more than five packages within a 24 hour period. We track that because we have a policy that says that we do patch cycles on all of our systems every three months and operating more than five packages in a day probably means we just did a patch cycle. So that way we track what machines did we forget about, what machines haven't we patched. Packaging has a long and slow learning curve. There's not a lot that we can do about this, but anything that makes packaging simpler to get started with and then it lets you dig down as you run into trouble is very helpful. And templates, places to start, the new maintainer guide is great, but bear in mind that the new maintainer guide starts right up front saying, oh, I'm not gonna talk about some of this stuff because it's really complicated. Shared libraries are a particularly important part of that. It would be great to get more documents around how to do that kind of stuff because enterprises, we have a piece of software, we don't really have a choice in whether or not it's got a shared library. If we have to package it, we have to package it. But in general, keep doing what we're doing. We found Debian to be a fantastic platform for doing this kind of stuff. I've been extremely happy with it. I am going to skip fairly quickly over the summary slide, but just make this one point. Ideology, so actually two points. The first one is Debian packaging skills are enterprise system administration skills and vice versa. That is something really important, I think, for everybody to be aware of on both sides. If you are a Debian developer, you are qualified to be an enterprise system administrator because what you do inside Debian is what we do on a day-to-day basis in the enterprise. Enterprise system administration is mostly development. It's not development in the sense of writing hardcore C code, but it's development in the sense of taking things, plugging them together, writing some shell scripts, writing some packaging. It looks a lot like Debian packaging, and that's the world that we live in. And ideology is really important, and I am a member of the Debian project in large part because I believe in free software and I believe in that ideology. You have to be careful about how you present it in an enterprise context because you can get people to agree with you and you can get people to care about it, but only if you freeze it in the right way and introduce it in the right way. Because the problem is that what the perception starts as is that you'll start talking about ideology because the software can't compete on the merits. Once you get the software in the door and it's clear that it works, then you can have a conversation about ideology and it's a much more effective conversation. Stanford, the central IT organization at Stanford actually now has a commitment to open standards and a preference for open source for software that we deploy at Stanford. We look for open source first, but a large part of that is because we didn't start there. We started by using open source extensively and then pointing out this stuff really works and here's specifically why. Here's the place we tweak post fix to meet this business requirement. Here's the place we tweak cyrus sassel to fix this problem that we had. We want more stuff like this. So with that, I will move on to questions. One statement, thank you very much for all your hard work on Lintian. It's really appreciated. Question, could perhaps stand for document a bit how they use Rep Rep Pro. Rep Rep Pro is a fantastic tool. I don't know if Bernard Link is here. I really appreciate it, but he's not a native English speaker and perhaps could use some additional documentation. Yeah, I would really like to do that. Documenting how we do all of our stuff has been on my to-do list for a really long time. I just need to sit down and find time to do it. It takes a long time to write documentation, but yeah, we don't have that documentation really even internally yet, so but once we have it internally, I will figure out how to make it public as well. This presentation is to some extent like a start on that because now I have a bunch of stuff written down that I know I need to go write down larger documents on. There are some white papers already up on my website about some of how we do stuff and about how we use get for devian packaging and a few things along those lines. www.e-y-r-i-e dot org and that's the place to start. There's a link to my homepage off of that and everything's under there. Hi, following up on the rep-rep-row stuff. It would be great if in the docs you also kind of had a nice recipe for how you deal with signing and possibly key distribution for people accepting something as, oh wait, is this a scary package? It's not signed by anything. If you've got a local repository, having that all kind of cooked out would be very nice. Yeah, and it's one of the things that I feel really guilty about is that our current dev archive archives are not actually signed, which is really a horrible, horrible security thing to be doing. Yeah, it scares me too, but we will be fixing that. Rep-rep-row is really good about repository signing and I will write down how we approach that. Hi, Wookie. I'd just like to primarily say that was an excellent talk, thank you. I've been in a relatively small, much smaller organization, kind of 35 people and almost every word you said is still true in an organization of that size. It's exactly the same set of problems, exactly the same set of solutions, we do things the same way. So in fact, Enterprise goes down to quite small, I think, and it's all good for those reasons and I think Rep-rep-row is bloody marvelous as well. And I agree, we've got to fix the multi-architecture build thing. That's the one thing that's missing from this whole beautiful infrastructure. Yeah, I mean, the building network is fantastic, I love it. And I maintain a lot of software myself that I also package for Debian and the building network is my greatest portability testing tool ever. I just want to be able to install it, aptitude install it. Matt from University of Minnesota. How much heel dragging occurred when you guys went from Solaris to Debian? There was a lot of delay in that project. I think it took us four years to actually get all of the Solaris boxes turned off. One of the things that helped us a lot was that we made a decision that we were going to do the migration and then we refused to upgrade Solaris. So that once we hit the point of, so when you're actually upgrading both environments it's a little bit harder to get people to move but when they say, oh, I have a Solaris, this Solaris eight box and the only thing you're going to get anything newer is if you move to Debian, that helped. But you have to, that's something you need to make an institutional commitment to move. The nice thing is that once you get at least halfway moved now you can go use that same argument of running multiple environments is expensive. We should turn that one off. How many people are working at your IT department and how many are more senior admins, experts and how many are not that experts or uni admins? Well, I mean the overall IT services department is about 250 people but that's really deceptive because that includes the entire phone company. We, like many universities we run it, we have our own phone company. But the IDG itself is, I'm going to forget off the top of my head, it's somewhere in the vicinity of about nine people. Of the, there are two technical leads in the group, myself and my windows peer. And then they're probably, I mean it, pretty much everybody is at the point where we can tell, we can give them a project and say go implement this and they don't require a whole lot of hand holding. But certainly I help a lot of people out routinely with Debian Packaging. And I'm sort of the local Debian Packaging expert and I answer a lot of questions in that front. That's really important to have. You're going to need to do that, you're going to need a local expert because the stuff is too hard to figure out if there's nobody there who really passionately cares about and we'll go learn about it in advance. Hi, I work for Google in the internal Enterprise Corporate Infrastructure, and I see a lot of... You probably know Neil Krallin. He used to work for Stanford. Yeah. I see a lot of parallels with our situation, right? We've solved probably some of your issues but we are behind compared to your other issues. Something you didn't talk about and Debian at large has not standardized is how do you store your custom packages? Do you store them in a version control system? Is it a single one? How do you track development on those custom packages? So we started using... We started with Subversion because of SVN Build Package. We are now moving to Git. Everything new is in Git and we have migrated about 15 to 20% of our local packages into Git at this point. One of my goals is to get completely off of Subversion and entirely into Git. The way that we managed to get everyone to learn how to use Git was we needed an internal team wiki and we deployed icky wiki. And we said, here, now you can use a text editor and you can commit stuff directly to the wiki, you got to learn Git. And that worked really well. It taught everyone Git in an environment where they didn't have to use branching. And then we moved Puppet into Git. And now if you have to make any change at all to a server, you have to use Git. And that we slowly introduced them and then we introduced packaging where you do want to deal somewhat with branching and with tagging. And so you kind of slowly introduced the stuff. And now I'm at the point where one of our junior system men's comes to me and asked me a question about Git. And I show him the rebase dash I and cherry pick and he's like, wow, that's awesome. But you kind of have to need a starter project like that. And icky wiki was a fantastic starter project of here's how you actually kind of get oriented and make some commits and change some stuff. It's great also because with a wiki they can commit something, push it and go to a webpage and find out if it worked. And that, because there's this immediate visual feedback and that was really important. And how do you track your delta from the BN upstream? Like let's say you modify postfix, which is not. We have, yeah, we're one minute left. So mostly at this point where I'm, if we base something on the Debian package I try to use Git and use branches. So import the Debian package. We branch off that package off that branch when we, their new Debian package we import it into the Git repository to do a merge. I think that we're pretty much out of time. So I will thank you for being a great audience and I'll close it there.