 So welcome everybody. This is, this talk is about deploying dead-end and dead-end light systems in large infrastructure, particularly in the, we're just not blinded. We have, we have, in a really large university setting, so folks who are interested in how to do this get some ideas and ask some questions, hopefully. And we're lucky to have Frank Brocken with us here, Andy. Do you want to come up and give the intro? I just wanted to... Oh, I think you should keep going. We're saving some time. So, we have Frank Brocken here, and I'm sure that I actually can't even pronounce, Conan, is that Conan Brocken? No, you're close. Okay. I need a good job action. So, I'm just going to turn it over so you can get started. I hope people have some chance for a question. That's awesome. Yeah, thank you very much, and thank you for attending my talk. This is the opening slide that I'm sure you have seen by now, so I'll immediately switch to a completely, I hope, surprising slide. Now there's this. This is Debbie, as you can see from the headline, and this started about, say, 15 years ago, where my wife and I, Esther, were entering the high charas in the western part of the United States, which is Windows. And we were constantly frustrated by Roche that we couldn't enter and things that were blocked for us, and so we decided to completely head the other way, and eventually we found our way into the Californian and Nevada deserts. This is probably the northern tip of Death Valley National Park, and it's a beautiful area. It's very hot in the summer, and it's extremely cold in the winter, and short is a beautiful piece of nature. And I don't care if all these people stay in high charas and use their Windows system, as long as we can use our Devinian system, we are happy, and we are in the wild, and we can do all these nice things that Devinian has to offer us. So what am I going to tell you in particular? Here are the topics that I had in mind. A little bit about the context. I'm going to tell you a little bit about that strange name, University of Oregon in the Netherlands, and don't try to pronounce it unless you are fluent Dutch or English speaker. And in there, there was at some point in the university, a call for an alternative working place. That is an alternative computer working place where people could do their computer things. And that became, eventually, the Linux working place. I'm going to tell you a little bit about that. But there's much more Devinian at the University of Oregon than the Linux working place. I, my group, have been using Linux from the very, very early 90s, so that was even before version 1.0 came out, and we switched to Devin, I think, in its first year of existence. I'm going to talk about that in detail. There's more Devinian at the University, more or less because of the switch that we made, and I'm going to give you an overview of the kind of Devinian products that you may encounter as well. Somewhat, extensive example will be Stealth, which is a five-integrity scanner that was designed about five or six years ago. Then I have some conclusions and maybe questions. So first of all, as we learn from anti-stock, open is in. There's a smorgasbord of open things. If you look at where there are all kinds of open things, and one of the things that I haven't heard about since anti-stock was open government, but I'll add that to my smorgasbord as well. Interesting is that in Europe, all these various governments, and over here, this is my point, you guys, over here you see the Dutch following buildings. There is a law of the past that requires official organizations to start using open source products. Now that's a deal, and before it is actually effectuated, you're probably tens of years farther, but at least that deal was passed. Anti-stock law, Munich, the country of Denmark, this year decided that their official product must be written in the open document forward. So that is a great step. And once you see that all these levels of people and then governments are accepting open sources as their favorite standards, then you see that the supernatural thing, the European Union, and here we have the Foreman Building for European Union in Brussels, that the European Union as a supranational organization, also enforces that bill and requires that the member companies start promoting open source. So we are at a perfect moment in time to promote open source products in all lower levels because we can say governments require that. If you would like to have an overview of worldwide use of open source products, then I would like to refer you to the Center of Strategic and International Studies, which is located in Washington. And my paper that will be in the proceedings of this conference gives you the reference, but I'm sure you can find that on the web. It is an extensive list of all kinds of open source products as they are required or used in the world. You may have used this, actually. I don't know if you did. Are you aware of that? No. Well, it's a nice list. I look at it and it's great. So, the context. A little bit national and supranational stuff. Then if we look at the university, the University of Cronin is the 5th one oldest university in the Netherlands. It was founded in 1614, 49 years after its first university was founded. It was laid in 1575. The University of Cronin has about 25,000 students and our staff consists of about 5,000 people. That's not only scientific stuff. That's also support stuff and things like that. But also, they're all in like 30,000 people working at the University of Cronin. Well, students. Well, they work. Since several years, the university's information of technology center is responsible for running all its IT facilities. And what we see here is the main building. It has a remarkable architecture. The main building of the center of information technology. What I like about this picture is the fact that it shows this window up here behind which is my office. So I got this from the web and I think it's a great, great picture. It gives you an idea in what kind of location we are. As you know, the Netherlands is full of water and so we are in a little pond because I thought there wasn't enough water in the Netherlands. The University of Cronin is well known for its high performance computing work and one of the main projects that we run is the LOFAR project which is a project on astronomy which is the equivalent of the largest radio telescope in the world which has a diameter of, I think, 200 miles. Now, you cannot build a structure like that. So what I did, I did some really ingenious things with all kinds of measuring points spread out over an area with a diameter of 200 miles and effectively acts as a big dish in which you can collect all these astronomy data. All those data run into our computer center is cleaned out there, is washed, is filtered and that's one of the things that we do. We see here some pictures of these big holes and in there, apart from the supercomputers that are used for the LOFAR, for all kinds of servers for the email, for the university library for basically everything that you would like to have at the university to keep it running and most of these systems are running on Unix kind of operating systems. There are some window systems in there but I would rather not talk about them. At some point, if you look at this this is the server area, if I may give it that name but people at their desks and the students they are using very simple computers and for a long time it has been MS Windows that has been the working place which became a standard working place in the 1990s and I think that's a very good idea but on the other hand at that time you really couldn't give the university students and staff members a Unix-like system either it was commercially much too expensive or it wasn't ready to be spread out over the whole university. Initially we used selectware but that required you to recompile every package from scratch at that time and that was a non-option for the non-user so selectware was not now. And so our traditional working place and I like this horrible picture about the winner of this with all these drooling down and things like that it's of course horrible. It is not open, you have no idea what's going on behind the scenes except for that it's bad, it is expensive it takes a fair amount of the budget for the IT facilities for the university it's monopolistic, that's bad. I'll go back to that. And it is constantly in need of repairs. Now I don't say that our open source products do not need repairs and don't have but but this is symptomatically wrong and you can more or less prove that it cannot be better than it is. Fortunately at some point there are open source alternatives for almost everything people were using at their normal distance. So why don't you use open source products? Especially if the political climate at that moment is favorable towards open source. There are more products with the traditional working place. One of the things is that if everybody is using windows but that basically holds true for everything that you use as a monoculture if more than 50% of your systems is that particular system then you're running the risk of a monoculture and thereby of all kinds of diseases. Let's see, how many of you running Linux, Demian Linux run a virus ban on you? Okay, there are two. How many of you run also a Windows system occasionally? I do sometimes, unfortunately with a virus ban on you. So that is exactly the reason why a monoculture is so dangerous. It is the only system that is run by a large number of computers in your organization and the attacker only needs to find one vulnerability in that system and a whole range of your computers is gone. The malware tends to target the dominant in this case, Windows system. And because that system is so complex you cannot get it right. So there are plenty of reasons for using open source alternatives. I have a nice example in about 2007 and once, one particular day over 200 computers of our university were infected by a group of hackers and became victimized to that hackership. I don't think you could do that even if they were all running Debian because there are so many variants of that Debian system. I mean there is a basic operating system below there but what everybody is using in their computers is basically varying from installation to installation. So it was time to start realizing the Linux workplace. And indeed it was clear by the time we started working on that that it should be the Linux working place because the Linux working place we have a stable system we have all the software in place and so why don't you use that system as an alternative? There are very good reasons for doing that. The choice now is what kind of system should we use RAD, should we use Sleggware, should we use Susie. In fact there were two competitors and that was Ubuntu and Debian. To me there is not so much difference. I prefer Debian, I've run Debian since when I said early 90s or something like that and it seems that Ubuntu is somewhat more familiar to the common user but if you are more into the computer system itself if you are more interested in what's going on behind the scenes then Debian might be your system of choice. And if you go to the core system administrators or our university and here we have one fine example of a system administrator and his words when I took this picture you can't find anything on Ubuntu. Now I don't want to offend my wife in the back of the room there but this morning she had a problem connecting to the universities here wireless internet and me too couldn't find something on Uwit windows so I think you may be right when the focus is on the Uwit character of Ubuntu. Fortunately for the distribution of the system it's not so much a problem. The Linux working place looks like this. If you have difficulty reading this diagram I can completely understand that and that's not the point. The point is not that you cannot read this diagram because I'm going to dissect this diagram in a few minutes. So this is the whole picture. This is what Jurian Bogma and his picture is over here. Jurian Bogma is one of my colleagues and he run the Linux working place project. Now his name is a famous name in the Netherlands, Bogma because Bogma is the manufacturer of a particular kind of booze in the Netherlands called Geneva and since there is no wine being grown in the Netherlands I thought it might be a nice idea to bring to the cheese party a bottle of Bogma. Unfortunately when I asked yesterday through customs the security officer looked at my bag and said what's in there and he looked in there and I found my bottle of Bogma which I bought at a liquor store and he said that's not sealed so you cannot take that into your claim to the United States. So I had no other option than to remove the Bogma bottle give it to the customs official and say I hope you like it. There will be no Bogma at the cheese party but I took some cheese so I hope you like that. Let's go back to business. Back to business. There is this working place and the unattended install goes from some playing computers using all kinds of mirrors to a computer that can be in principle booted it's not yet ready to do that and if your original computer was a Windows system it comes out of the initial unattended install as a dual boot. Now interesting here is that we use our own Ubuntu mirror and we have a known package repository and those repositories are called Debian repositories and for this I have the help of the Debian developer George Danchef who helped us creating those packages. George is from Bulgaria and we invited him over to the university for a workshop for about a week where he trained us to create packages to create these repositories so we could do all these kinds of things ourselves from then on. Now we are here. We have computers that will become the Linux workplace either pure or dual boot. Then in the second part we are going to create images. We use a novel product for that and it creates images. An image for each of these different forms such as the dual boot form or the single Linux workplace. Then for all that we clone computers that will eventually be installed in the several departments and that run specific scripts. The university has a class B network and that's subdivided on all kinds of smaller networks have their own IP addresses, ranges of IP addresses, have their own system administrators and all these new kinds of specific things that must be fine tuned, must be tweaked for that particular situation. That happens in the first boot period and from there you have a running ideal boot or running Linux workplace. And that is an eventual working place for either a student's workplace or it is a working place or a staff member depending on where the person is working. And then if you look at what you have then you have something that is not a standalone system as such. But this computer talks with all these kinds of supporting systems around it. That's why all these arrows running towards these installed computers. For example, the basic system administrative work software is installed in these computers. The user's home directories, they tend to be roaming and that means that if you log into this computer you have your home directory. So all the home directories and user's own unpacking software will be available there as well. Then authentication takes place using the LDAP. That's a spelling error. That's the LDAP system. But what we do for authentication is we immediately go to the root credentials and it's hash form into the working place computer so just in case this one goes down you can still log in here. Then we have remote services and remote administration that we use all these kinds of products for. And the fact that it says you know well my print doesn't mean that it has to be well but that's what Jürgen made when he created this picture for me. Then there's a service desk that can talk with the users here and give them help if they are running through problems. There are all kinds of statistics being collected and being used. You can immediately see if a computer is being used where it is located who is using that computer and if you are for example a student and you would like to find a working place you call the help desk and say I would like to have a living workplace where do I find the nearest help desk and where are you located right now I'm here ok if you go down to that one there's that computer and it's available for you. Very helpful. So that's the system. Now how flexible it is. So we have a living working place and it's being created how flexible it is not complete. It cannot be complete. You have like 30,000 people and all these people have their specific wishes and things like that but we think that a living workplace will be asymptotically complete in the sense that once somebody asks for a particular package whether it exists or not it will be added to our standard in the working place. So tomorrow's standard is a super set of today's standard and to cook Larry Wall the good thing about standards is that there are so many of them and indeed we have tomorrow our standard and we have today. Every added package every new package will be added to the system. That's very good. How do you do that? Well there are basically three ways to do that. You can, as an ordinary user, say I have this program and it's great I would like to have that into the standard living working place. Now there are two routes. The formal route is to file a request at the service desk and then eventually it winds up into a living working place but it is much faster and much smarter to go to Urion and say hey Urion you are running a project could you help me out and do that program into the LVP? Yeah, fine, good, let's do it and it will be there in an hour or so. If you are a more experienced user you can have pseudo route access. That means you get the right to run a script from your computer on that controls the organization of the software and the repositories and makes sure that that request package becomes embedded in the living working place. And then there are these kind of super users that are so trusted that they can be given direct access to the route partitions of these mirrors and repositories. That's good. If you're in that then you can photograph and you can change all the things around but you are trusted and of course you don't do them. Very nice. So that's it. Is that all? No, that is not all. I already mentioned George Dunchef and this picture of this little squirrel I found all his Facebook slides. We've seen George earlier in the earliest slide so we now have a problem we don't know if he looks like the one in the earliest slide or if this is the one he looks like. The other one here is Tony Menzil and we can verify his picture because he's sitting over there in the room. There he is. And both George and Tony have been extremely helpful and willing to support me in creating the Debian packages that I have been creating over time and I'll give a short overview and some more extensive overviews of the packages. To teach programming languages one of the languages that we teach is the C++ language and for that since the early 90s already we use a document that has been enlarged and grows over the years which is called the C++ annotation which is now a Debian package as well. And yet on that programming course my students learn how to create their own grammars and for that I created a C++ parser generator just like Bison but now specifically aiming at the creation of C++ source parts. I've been writing various tools ITMATE which is a major technology XD which is actually a very nice little tool it's an abbreviation for X++ directory change and you can do something like XDULB and then XD will change directly for you to, for example, use a local bin very nice, very addictive and I don't think I could survive if I cannot use XD anymore. And then Yodel and Stealth Yodel is what we do in the Alps but it's also something else and I have two somewhat extensive examples of these products, Yodel and Stealth. Yodel is a meta pre-documented language and it can be used to write all kinds of documents in a very simple way and I've given you an example of how to create a manual page using Yodel and one of the things that you see here that all the meta characters that you use in a meta language they're only parentheses but it's a special meta character in the sense that if it is not used as a meta character then it's just a parentheses so that's nice, you don't have to do special escape things and that kind of thing and if you do something like that you just write what you would like to do in plain text and then eventually you get something like for example HTML or you get later or you get plain text or you get a geofile this one, if you use it to create a manual page you get a geofile and it comes out like that it's a perfect manual page and all the manual pages that you will see in the Debian products that I have been creating they're all made using Yodel and it's a jiffy it's absolutely nothing to do with it we do it for big documents as well but I'll spare you the illustration you should read in the manual page now, here if you look at the linkage working place try to extend that try to extend what's going on there to the whole universe then you will agree with me that there are many many systems that must be in a pristine stage the software in there must be okay hackers, the bad guys will try to improve at some point in the systems and we try to chase them away but at some point they will succeed what are hackers doing when they are into your system they change things no matter what way you look at it they will change things so the idea that we at some point had was why don't you have a program that verifies the integrity of the essential software and that's what stealth is doing normally what you do, you have all these kinds of standard frequent uploads you check your logs you do a default deny all these standard security no worse comes to you and the hacker really enters your computer how do you track the hacker how files don't tell you that something has happened if he comes through the default deny policy if you have upgraded to your latest software, what do you do you run stealth the SSA space trust enforcement required through a locally trusted home now I want you to look closely at that expansion of the word stealth and you will understand that it took me quite some time to find this meaning for the word stealth and yes it did it is an open source file integrity check and the core idea fortunately is not mine so I can proclaim it here in all absence of any modesty what you do is you create a fingerprint of the current state now many integrity checkers do that so that's nothing new that's what you do you detect modifications of the fingerprint many integrity checkers do that and then you separate this is new the state info from the target computers that are being checked so your fingerprints and the handling of the comparisons are not taking place on the computers that are being checked for their integrity so this is the setup we have a stealth monitor we have SSH connections we have several client computers that may be connected to the internet and that's it several I wrote two here but that may also be 200 or too fast very good we only need one stealth monitor and also we would like to have this system as separated as possible from the internet there shouldn't be any incoming connection into that computer only outgoing connections so that's so what do we do about it the fingerprints that we get from the software here is maintained on this computer and there are no specific things on these computers that hit to the fact stealth is being used to detect the integrity of the software of these client machines a client machine could be a server it could be a user computer it could be a Linux worker it could be a repository the stealth monitor computer itself is inaccessible if you want to access that computer you have to go to its console and login as a user and then you have access but there is no way to access the computer from a distant location interesting thing is that again different from existing file integrity checkers that nothing is pre-specified by stealth that adds to its stealthy characteristics if you would like to use this kind of software to verify the integrity or inspect or generate these statistics you do that that is fine you could use FIND you could use MD5 some or SHA256 some all these kinds of software that are available you could use those software to verify what the actual state is on your client computers now in order to do that it is clear that if you are using all the client computers programs like FIND or these Hashem computer software that that software must of course be in a performance stage as well so that we can do it we can check locally whether the state of that software that is used there is still intact so we can inspect vital software locally and finally normally you would do a self run or something that you would like to do every once in a while using a drone job but even that could be a hook that is used by the attacker to see whether something is going on if always at a certain moment in time FIND is running or the Hash is computed of files then the hacker might start to think isn't there something real going on so we can change this in such a way that the stealth runs themselves are not predictable then they start at any moment in time you can say to a stealth do for example for example in the first 20 minutes of every hour do a check or do a check once every hour it is up to you to decide to do that you have complete freedom to specify when in what interval you would like to do these integrity checks and it is easy to configure that so let me give an example here is what you do to check the integrity of the system sent you ID files you have some arguments that you would like to specify with the FIND command where you say I'm interested in the SUID you and group flags for the user root or the group root and those arguments are being added to the FIND command where this one is replaced by what I wrote here and for that you run the Shalonson program and that one is being run for all the SUID files and the logs are being sent using a secure connection to the stealth module and what you can get is these kinds of reports for example the hacker added a file he added a file user root perm I didn't do that there is no package called user root perm and suddenly it appears who has done that this is his hash it wasn't there to begin with and not only that the bastard also removed his bin Unix check password he removed that one why did he do that? I don't care but he did it and we have to take measures to prevent him using it and of course this is a nice one he modified the login program into a cooked version of his own all this information you get automatically and it's very hard for an hacker to detect we've been using stealth now for several years and all our important systems and it runs as a chart perfect so that's it as an age based trust enforcement and then somebody hacked through the screen required this is a locally trusted host of course because you have to trust his computer to do all these funny things on your client computers connections are as age based and it is trust enforcement so that's the system and that's how it got its name that's a bad new product it's been there for I don't know how long it's one of the first things I think of the project so conclusions there's a worldwide interest right now and I can say worldwide because we know that because of that Washington organization there's a worldwide interest in open source products you can use that to convince your managers or decision makers that they really should I don't say switch to but at least consider changing to open source products as we've seen as we've seen for example from that hack that we had thank you monocultural operating systems are inherently dangerous it's not only the case in software it is also the case in agriculture and in intensive farming and all kinds of things last year we had in the Netherlands a case of the Q fever with goats and that resulted in the killing of I think 20 or 30 thousand goats where more than half of them had no fever at all and that's called preventive cleaning up I think that's preventive killing and I have some other strong feelings about that that I'm not going to share here because I might offend people but in private I could say the university of Cronin by now has an alternative not to killing goats but to using a working that's the way it works and the nice thing about that is that it is an alternative we are not enforcing people into using Linux no we are offering them a choice we are offering them a choice you can be continuing using that bastard system that you've been using over years but you'll have much more fun if you at least consider switching to something else like what we did like 15 years ago when we decided not to go into the high Sierras but into the desert that being a package is of such good quality people you have an idea how good it is because you are all dead in force Jürgen told me that he has been working for five years intensively creating that limit work like thing upgrade or upgrade and over those five years he didn't have a single problem with dependencies with software that didn't work with things that broke when you upgraded to a new version of Debian it works perfectly there is no other system that does it so well and actually that was the main reason why the university decided to start using Debian as its main system rather than for example a system like Red Hat so we covered the things that I intended to do why should we use open source I told you a bit about the reasons why we wanted to switch to a Linux working place I told you a bit how that is organized you now know what kind of products we are running I gave you some examples and some conclusions and there is only one thing left for me to do and that is to thank you for your attention that's it now I wrote for some questions thanks for the talking description I'm curious about you sort of describe the deployment scenarios I'm curious you could describe a little bit about the maintenance approaches for instance if you are using configuration management tools for once the machines have been deployed how do you keep them up to date is it just a rewrite process to reinstall or I think you do that the way I'm not sure because you actually over asking me but I think from what I understand from what Jurgen told me is that they do what you always do and that is you do an update and an upgrade every once in a while and make sure that you have the reason software and if upstream in our own repositories changed something then that software is being redeployed and we have the let me see if I can go back to those slides no there's the unattended install and then you have the next slide remember where the installation takes place there's also something where you can at the third slide I think that was visible where you can remove system administration and system maintenance that is the situation where the administrator is able to draw from the repositories and spread the new software over the distributed limit working places is that an answer to your question? there will be a puppet for the further session coming up later in the conference that I think you might be interested in talking about puppets configuration management system I know that we are using several long W2 tools right now Jurian also told me that he had some problems finding the right tools in the W standard software but things changed and maybe right now there are appropriate tools for example one of the things that is going to be changed is that we are using SIFS right now to mount the user home directories and that will be changed shortly into NFS version 4 so things like that changed we tried to do SIFS for home directories at University of Connecticut we tried to do SIFS for user home directories at University of Connecticut we had trouble with window managers they create some sort of block files or some special files that didn't support did you have that problem and if so how would you work on that I don't think so because we are talking about Linux we are talking about Linux Linux creates like GNOME and KDE creates some special files in the home directories and SIFS doesn't support that so it may be that Jurian told me that he had several problems with indeed SIFS he didn't explicitly go into the problems at most there was simply no time to do everything and he said but I don't care I'm not going to solve that right now we are trying to get away around it maybe I can give you here address and you could contact him anyway and he said I don't care because we are going to switch to NFS level 4 shortly anyway so why invest money into NFS does that bother us to answer your question if you would like to have Jurian SIFS come to me and I'll give you an answer thank you so this is sort of pointless but yeah that is a no problem if you run into that with open NFS as well I recommend looking at NFS before you decide for sure to do NFS before there are features both ways but NFS is also open source project also with Demian we had a little video syncretic in some respects but it gives you management capabilities that NFS doesn't have but yeah it's a no problem with file systems that don't really like special devices I'm pretty sure there's a way to change it in the known configuration to use it in a different directory to store that stuff and then you just stick that somewhere on a little disk to store just those session sockets and leave the rest of the stuff out of the user's home directory okay thank you I think that was another question I'm just wondering in the university setting I haven't seen anyone training Debian developers do you have any plans for well we had George come over to train us using the package constructions and things like that and most of our core system administrators prefer Debian anyway so they are pretty well versed in using Debian and then I know how it goes there's a new guy coming in and he's learning from the existing people and then he can do it as well and he can read also most Dutch people can read so they read the manual and they can do things that the other guys could do also I'm kidding this lightly but you know what I mean it's not that much was that answer your question you're welcome to visit us I mean come to us and talk to your in person and see what he did or also if I can give you his address and you can contact him I'm sure he's happy to write back to you okay great clients but we should break up and people who want to go to another talk can go and we can wait for people to come on hours