 today. Please join with me in thanking Iacol. Okay good this one works I can turn that one off. Hi, I'm going to talk a little bit about the history of Send Mail. Now histories are if you were like me in high school histories boring but sometimes you can actually learn stuff from it so I'm actually going to talk about this in two parts but let me try and first express to you why you might actually want to listen to me. Send Mail is an old program some of you may have noticed that it's more than 30 years old believe it or not but it has survived amazingly well by software standards incredibly well it actually was one of those pieces of software that really did change the world it was created entirely without any particular corporate support sorry IBM it thrived in a rapidly changing world I cannot begin to express to you how rapidly it was and some of you might ask how did this happen why did send mail do so much better than so many other things so I'm going to talk about this in two parts part one what actually happened I'm going to actually kind of rush through this because I've got a lot of material and then we'll get on to part two which is the more interesting parts I will take questions at the end hoping that we'll have enough time I'm afraid if I take questions during the talk I won't get to the end so let me start off with a bit of background the first work on what is now send mail started in 1980 at UC Berkeley while I was working there I was student and staff it had no official support including from the project I was working on I was supposed to be working on relational database management systems which were brand new back then in fact the official word was they couldn't possibly work but we made them work anyway because we were too stupid not to the internet did not exist the ARP and that did but the internet did not and it was built purely as a matter of necessity which is probably one of the reasons why it succeeded so let me try and give you an idea of what the world was like in 1980 I'm just forgive me I'm guessing that a lot of you were not in the prime of your career in 1980 CPUs kind of a commodity CPU was a 16-bit machine those are actually kind of the high-end ones the low-end ones were 8 bits they were substantially well less than 1 MIPS million instructions per second I'm thinking of for example the PDP 11 disks were dramatically less than 3 gigabytes or 1 gigabyte in size they were the size of refrigerators the the since I was on a database management system project we had a lot of disk space we had 300 megabyte drives and the day we finally got enough in that we had an entire gigabyte of disk space available to work on very large databases we actually asked ourselves how we were going to find enough data to fill this thing memory was definitely less than one megabyte you paid by the kilobyte networks were less than or equal to 56 kilobits per second the Arpanet backbone was 56 kilobits and although Ethernet existed at Xerox Park it wasn't really out there yet relational databases were really a research novelty that's what I was working on mobile phones actually did exist but they were those you know brick-sized things that people went down on the table they were large they were expensive very expensive and cameras used film now in contrast today CPUs are typically you know 64 bits I did a back of the envelope calculation they're over 50,000 MIPS from under one myth disks commodity disk drive 2 terabytes you can fit in your back pocket no problem memories main memories over 100 gigabytes that's you know fairly common network you know at home you get megabit for DSL and OC 3072 is 160 gigabits so that's long haul now no SQL is hot relational database management systems are boring phones of course are tiny ubiquitous and effectively free and the cameras are digital included with the phone so let me show you what the mail system was like actually before I started here's my first architecture slide now some of this may be a little hard to see and I apologize for that this was as it came from Bell Labs bin mail was both the user interface and well it was everything bin mail was the mail system there was a network called UCP so it could send mail into bin mail you could send mail using you you X Unix Unix execute which had its own spool directory and the local mail spool ie your mailboxes were on the side nice little bit about where Berkeley was we when this whole thing starts we actually did have our first Vax so I lied a little bit about the 16 bits 32 bits were around but people were debating whether anyone really needed that much address space it was the main machine for computer science the Vax was a canonical one MIPS machine and everyone in the department had an account that shared that one machine there were more ASCII terminals and offices than ports a window systems you know way in the future there was a patch panel in the mail room so that you could go in and patch one of the ports open ports on the machine into your office you could work that of course meant you had to pull somebody else's plug out you might want to figure this out where this was going and there was a store and forward net called Birkin at which had been written at Berkeley by a guy named Eric Schmidt who was formerly CEO of Google and happened to be a student at Berkeley interesting guy so the problem the Ingress project database management project that I was on and was kind of you know running the machines and so forth I was the default system administrator I got an ARP and that connection ARP and that connection by the way I said the 56 kilobit backbone we couldn't afford that we had a 9600 Bodline to the ARP and that everyone in the department decided they really really needed to use this connection we had a 16 bit PDP 11 it was not enough to give everyone in the department an account so we could free up a grand total of two counting two teletype ports to go into that patch panel in the mail room to share I am exaggerating when I say professors got into fistfights in the mail room I never actually saw fists and everyone complained to me because it was obviously my fault that I hadn't fixed this thing I should point out by the way TTY ports at the time you go well just put on more TTY ports $5,000 for one 9600 Bod TTY port so this is the genesis of the program called deliver mail I observed that they didn't really want access to the ARP and that they didn't want telnet they didn't want FTP they wanted email that was the killer app we had a network that connected the ingress machine that had the ARP and that connection and the backs that they were on it's software software software I can write software to glue software to software you know how hard can this be just forward it and for the routing conveniently everything used different characters so there were colons for Birkin at an ad sign for ARP and that so I could just cue off the characters and it would be pretty easy I have to say I say that now I spent a lot of time trying to figure out how to write it couldn't figure out how to do it and then one day I sat down and just started writing code and as soon as I did that everything became clear so before I wrote this this is actually what the architecture looked like at Berkeley so this is what other people had done down the left-hand side you've got the ARP and that stack mail was part of FTP back then so the FTP demon sent into a mail spool there was a user agent called send message which is how you interacted with the mailbox there was an outgoing spool and it used FTP to send out and then over here there's the bin mail there's our mail with the ECP before down this side and then there's also Birkin that has been hacked in here there's a user interface the guy named Kurt Shones wrote called UCB mail that was an improvement over bin mail which pretty much bin mail was also a synonym for cat and and they had the same local mail spool file the coding by the way is this is the ARPANET code which unfortunately I find I'd never kept a backup of this code so I don't have it the blue stuff was homegrown at Berkeley the yellow stuff was hacked at Berkeley the white stuff was fairly original it's not just this I want to show you this little piece of code because no talk would be complete without a piece of code this is actual code that I found in a bin mail on Unix when I started there's there is at least one bug in this that pops right out and I tell you now research was Bell Labs main machine so somewhere in there that was hacked into the code if you're going to research from anywhere on the net you do this magic thing where you can drop the H on the end there's something weird about converting hats to exclamation points I don't know it's just it was weird so design principles I'm going in I'm hacking the mail system what do I want to do first thing is I'm just one guy I'm working part time this one probably actually drove all the rest of them and it turns out by the way to be probably be the main reason I succeeded don't redesign the user interfaces I don't have time to do all of that and even back then people were getting kind of religious in their preferences about user interfaces don't redesign the local mail store because there were too many things that actually directly read you know var mail mailbox and all of those would have to get fixed so I couldn't touch that I had to make deliver mail adapt to the world not the other way around once again just because I couldn't go in and find everything that had to get hacked and kind of corollary changes a little as possible so I put in deliver mail the architecture looked after a little bit of shifting around kind of like this you can see essentially deliver mail is now where bin mail was before bin mail is kind of off on the side there's you had to have a dash d flag that tell it to deliver into the mail spool because remember I didn't want to touch that so bin mail calls deliver mail unless it's got dash d in which case goes the other way around so there's this weird kind of dual handshake and whatnot but fundamentally it's you know it's what it is okay the problem with the solution inflexible compiled in configuration language queued off those special characters there was no address translation between networks so if you had a ucp address and you forwarded it to the arpanet the arpanet still saw ucp address the parsing was both simplistic and opaque simplistic because it just looked at the characters opaque because when you got something you had to know where it came from in order to figure out how to interpret it in fact there was an entire book written about interpretation of email addresses but you know it's supposed to be a quick hack and indeed it was now to give you an idea of how bad the addressing was I'm going to give you a little table that by the way is the title of the book that I referred to foo at bar obviously is an arpanet address it goes to host bar user foo foo colon bar was Birkat goes foo colon bar foo bang bar bang bass is a ucp address which either goes to host foo user bar bang bass or if you're on foo it goes to bar user bass foo percent bar at bass is the same kind of thing it's either an arpanet address to bass or to bar depending on whether you are bass foo bang bar at bass has one of four possible interpretations and the completely legal address foo bang bar colon bass hat fritz at frotts has who the hell knows so something had to happen Berkeley was awarded the DARPA contract to build a standard platform for research before on the arpanet everyone had basically their own home-built platform it was PDP 10s it was IBM 360s it was PDP 11s it was vaccine it was you know you name it bill joy sort of said well you know more about the mail system than anyone else why don't you do this I was young I was foolish so I you know how hard could it be it did require adding SMTP support well no let me rephrase SMTP wasn't written yet it required adding support for whatever it was going to be I could have done that but it was kind of good it's going to be kind of like the FTP protocol I could have done it in an outboard process which would have fit into the deliver mail model but because there were multiple recipients I couldn't have a single exit status to get the information back so I'd have to have some kind of inner internal protocol which was going to look a lot like SMTP so I might as well just implement SMTP that meant that I had to get do a queue okay now it turns out SMTP is pretty easy queuing is actually harder to do if I had known at the time how hard queuing was going to be I probably would have thought more carefully but fortunately or unfortunately I didn't so I ended up writing send mail came out in 1982 with 4.1 a BSD which was the very first version of BSD or UNIX for that matter that had TCP IP support in it had header rewriting so that it could normalize the addresses to be appropriate for where the message was going had support for SMTP queuing amazing new innovation runtime configuration but there you go so here's the architecture post send mail there'll be one more architecture slide after this you can see essentially what's happened is send mail is over here there's now starting to be more user interfaces out there things like MH from Rand have come along UCB mail still there these things are still dealing directly with the local mail spool file notice there is no appearance of IMAP or POP here because they really didn't exist yet okay but there's other services now that are appearing there's fax software and SMS gateways or you know that sort of thing and okay other than that oh and the internet that's this is where the internet first appears one little cloud up there and the use of the SMTP protocol so in 1981 I left Berkeley for what was going to be lucrative career in industry yeah right by the way if you're going to make a lot of money most startups are not Google as it turns out send mail was picked up by most vendors of UNIX based systems this is during the UNIX wars everyone was going to be the UNIX vendor the UNIX vendor meant this is the days when open source software by the way open source software existed before the free software foundation it just wasn't called that okay the free software foundation was more or less a reaction to what went on here and everyone knew knew that vendor lock-in was the right way to go so they took UNIX which is a perfectly reasonable standard platform and customized it in incompatible ways and they did it intentionally okay and send mail of course had to be part of what got tweaked as a result many versions sprung up in 1989 for various reasons I came back to Berkeley a staff and one of the things that needed to be done was we're going to a CS subdomain because there were now enough computers that instead of being something dot Berkeley dot edu it was something dot CS dot Berkeley dot edu I had to do some hacks and that got me going on doing some well as long as I'm in here I'll do this and well that and so forth and it ended up turning into a major rewrite which turned into send mail eight send mail eight you know steal from the very best went through looked at all the other things that people had done the various vendor versions many of them had good ideas usually crappy implementations but often some pretty good ideas the open source community had done some pretty good work notably IDA send mail which came out of Sweden and KJS stands for King James send mail by Paul Vixie Paul had dreamed that he could come up with a unifying version of send mail he didn't succeed at that but send mail eight did a lot of the work that had been done was on configuration which was really pretty crappy before and but there were also a bunch of code features so important changes in send mail eight there was a major revision of SMTP which is actually technically it's ESMTP today there were other new protocols for example delivery status notifications there were actually a whole bunch of them minor ones more integration with other systems things like LDAP were coming along here and and so forth so you needed hooks to be able to talk to other services support for eight bit mail important for internationalization multimedia an entirely new configuration package with a totally different philosophy and there are a bunch of other minor ones there was one other thing that was totally out of my control and that was the bat book the the O'Reilly send mail book this is written by Brian Castellas and I came along and worked with him on it he mostly wrote I mostly edited it dramatically increased the uptake of send mail eight however because it was finally like a book and it really proved to me the documentation counts big time much more than I had imagined now the problem is the net got bigger and send mail at sendmail.org at that point continued to drop pretty much into my mailbox and I was trying to support people got some volunteers to help me and that kept me things going for a while but at some point I couldn't members I'm still doing this part-time for free and I couldn't be doing any more coding and I really wanted to get back to coding so I came along and founded a company called send mail Inc 1998 the idea was to produce a commercial version I'd be able to hire a couple of people to do support they could go off and do that I could get back to coding the world would be happy the reality is that starting a company is a lot more about money marketing and sales than it is about technology. It also turns out that send mail it was a big enough deal that on the day we announced the company it made the front page of the New York Times that turns out to work against getting back to coding. Now it turns out I could give the whole talk just about the formation of the company there's a lot of interesting things to learn in here but that's not what this talk is about. Another thing by the way is it was one of the very first hybrid commercial open source companies. We had trouble getting funding because a lot of people thought it was impossible remember you've got to do proprietary lock-in if you're going to succeed that was the just everyone knew that so like well I've got to get off of this or I'll keep talking. So it turns out that the commercial entity did a lot of stuff in the open source in fact when I drew up this slide I had forgotten how much was there all the encryption and authentication came from the commercial company. Prior to that encryption was illegal to export and so forth so never gotten into open source. Milter is the mail filtering interface extremely important. Virtual hosting multiple logical cues lots of things that are designed for bigger sites sites that are supporting multiple domains etc. Directory services notably LDAP lots more checking the ability to do more spam checking and so forth. Most of these were driven by commercial needs on one level or another but in fact the open source world needs them as well. So this is kind of what the the email world looks like today. Substantially more complex. You see a lot of the same old stuff but there's you know now LDAP milters there's various access demons and whatnot there's more than just IMAP and pop there's of course exchange proprietary protocols like Mappy there's Outlook there's tons of user interfaces it's it's actually dramatically complex at anything other than the most trivial site. So part two so what who cares. So I'm going to try and give you some ideas here on what happens things that might in fact be useful going forward. You might call this lessons learned. So the first lesson is requirements change. They change all the time. When I started writing send mail really deliver mail other than the obvious thing of getting mail through at all quickly one of the major things was reliability. So the mail either had to get to the destination where you had to get a bounce back. I can assure you that hell hath no fury like a professor whose grant proposal has been lost in the mail rule number one the grant proposals have got to get through. Then it became about functionality and performance functionality you know what what's it going to do who's it going to talk to you know what stuff can you do functionality is almost always you know the first thing people want after it works at all and by the way a lot of projects usually die as a result of that because they put in too much functionality but that's also another talk. Then it became about protection spam and viruses started to hit and so it switched from the getting the mail through to keeping the mail out and in fact we have gone from a period of unreliable mail to reliable mail to unreliable mail because in the old days it was legitimate to say the mail system ate my homework now you have to say the spam system ate my homework but it's just as legitimate or illegitimate depending on your perspective. Then the world started to get more into regulations and things so it turned into stuff about regulatory compliance and so forth this is what Sendmail Inc. has done does do a lot in this area by the way these things don't mean that the old problem goes away it just means new requirements get added on and increasingly as particularly in this economy it's about keeping costs down. So emphasis here waterfall model does not work we all knew that there's just yet another data point and there's always new requirements coming along. These days by the way mobility and integration with social networking and stuff is becoming the hot thing. So I'm going to talk a little bit about various design decisions and the evolution of Sendmail. All of the things I brought up bring up in here are things that people have taken me to task on at one time or another some of these decisions I think were in fact the correct decisions to make some of them were incorrect we all make mistakes I'm certainly no different and the most interesting ones are the ones that were probably right correct decisions then but are wrong for today and those are always going to happen. This by the way is in no sense an exhaustive list so criticism number one it's an overly general solution you know the good news is you can do anything with Sendmail the bad news is you can do anything with Sendmail. The reality was the world was in flux when I wrote Sendmail the SMTP spec was still changing when I say changing I mean literally a couple of times a week and by making a general solution I could when a new draft of SMTP came out I could have it implemented by the next day and be getting feedback on it new mail systems were coming in all the time you just needed a lot of generality sometimes it's easier to build a tool than a solution and so Sendmail in fact is not a solution to the mail problem it is a tool to build solutions to the mail problem. The reality is as much as you'd like to say well but the world is so much better now you know everything stabilized and so forth the world is still ugly it's still changing it's still nasty. In retrospect I would still build a general solution I wouldn't do it in exactly the same way but I think the fundamental concept of trying to build something as general as possible is in fact a good way of doing software design up to a point. Rewriting rules is the way Sendmail does both modification of headers and parsing it seemed like overkill even at the time but I couldn't come up with anything better it turns out it probably was not overkill it was probably in fact the right thing to do. Using tab characters as the active character was the stupidest thing I've ever done in my life. My only excuse is that make files use tab as an active character and I went well that's a cool idea and I talked to Stu Feldman a couple of years later he said yeah using tabs as an active character was the stupidest thing I ever did in my life. Thanks Stu. People complain that it's ugly and they go well why can't you just use something like regular expressions? You looked at a regular expression recently the concept of pattern matching be it regular expressions Sendmail rewriting rules snowball patterns whatever their meta characters they can get pretty ugly. The concept was right okay it's pattern match and replace but instead of using characters which are regular expressions that use tokens and that was absolutely the right thing to do there was a system out of Bell Labs called upus that was use actual regular expressions and if you think Sendmail config files look bad you should have seen upus. The problem was the syntax could have been better admit to that particularly those damn tabs and the control flow could have been better I would do the control flow differently. Message munging message munging i.e. changing headers on the fly this was absolutely essential for interoperability at the time you could not get around it. It is still heavily used in fact there's cases where you don't need to do header munging anymore if you're taking you know from an ARPANET or sorry internet site I'm showing my age here to another internet site and you're not trying to do anything fancy you're not at a corporate gateway you're not you know the whole list of ifs you don't have to do it so there should probably be a pass-through mode something like that I would do that today but then there's gets into the question and I'll talk about this more in a few minutes when you get a message that is wrong when it comes in do you just pass it through wrong or do you reject it or do you fix it you know send mail fixes it today and there is no obvious correct answer to this there probably is no correct answer it's just a design decision the syntax of the configuration file well I admit it's it's no you know great beauty each line is is typed by the first character of the line so you know there's options there's rules there's mailers etc there's no nesting it's it's a flat file it was very very simple at the time remember I'm running on a 16-bit machine I couldn't exactly be loading yak into the process because that was just pure absolute overhead I needed something that could be parsed with the minimum amount of code there's too much use of single characters option names initially were a single character mailer flags still are single characters in retrospect you know it's ugly it is ugly I admit to that but it's not fundamentally flawed today I would use something probably along the lines of the Apache syntax but the main thing I'd get out of that is some kind of nesting nesting is useful for for example doing options on individual SMTP ports since and male now supports multiple ports but you know this is not fundamentally flawed embedding SMTP in queuing into send mail guess I could have kept the core of the system simpler if I hadn't done that but in retrospect it was definitely the right thing to do for lots of reasons the queue itself queue uses two text files per message one is for the envelope all the meta information including the header for various reasons which may or may not be good and then ones for the body of the message when you want to run the queue it requires that you scan a whole bunch of small files and file system performance isn't as good as you'd like relative to memory speeds that's gonna come as a surprise to all of you I'm sure but the ASCII format I have to say simplified debugging a lot I'm a become a real real believer in making everything ASCII wherever possible okay you make it unicode but the point is text in retrospect it was in fact the correct decision for the time today I would do it differently I'd probably keep the envelopes in some kind of database and messages and individual files just for a space cleanup if nothing else kiss principle says this was the right thing to do initially keep it simple stupid however I will also point out that embedded databases that were sufficiently robust to work did not exist when I wrote send mail DB did not exist DBM didn't work so I didn't have a lot of choice other than to implement a whole new database myself and remember I'm a single programmer I don't have a lot of time so I did the simple thing I'm sure you're all gonna love this one using M4 for configuration the syntax of M4 is painful at times I will admit but I needed some kind of macro facility or I had to build it into send mail and I wasn't going to do that for the reasons I've already described so there's no point in reinventing the wheel there was really nothing else that I could have used the other choice was the C preprocessor I could have used you know pound include and pound define and that sort of thing but that really wasn't quite powerful enough the main problem with M4 were those damned DNL's delete through new lines and there were absolutely no reason for those to go in there I was obsessed for some reason which I cannot reproduce at the time probably severe obsessive compulsive disorder or something of making the output of M4 look nice and readable and lots of extra blank lines offended my sense of aesthetics from the point of view of reading it there weren't you know a new line is a single character I wasn't going to be making the files so much larger that it was going to take an extra disc read to read it in or anything like that so it could have been a lot cleaner and a lot less ugly and saved all of us myself in particular a lot of pain some tool was needed maybe M4 wasn't the right tool to use maybe there's a better tool today it's not really worth changing it's not a big enough thing now here's an especially important one the choice to extend versus change features this is one I think I got wrong by the way an example masquerading when I first masquerading is the ability to say instead of user at host dot domain it will come out as user at domain or something like that regardless of which host you start from or generics are the ability to change your login name to full name or something like that I did it wrong the first time it was just a mistake but instead of fixing the existing feature I added new features to do it my philosophy that at the time was I didn't want to break existing sites so I didn't want to have somebody put in a new release and have it suddenly start working differently because I thought that was going to create support problems for me in retrospect that would have been true but it would have been less painful to have made it so that all the new people coming along would have had an easier time of it and what I could have done would have been to have provided upgrade tools to say when you install the new version run your configuration file through this and it will do exactly what it did before otherwise you know it'll do the right thing I got this one wrong and in fact this was kind of a theme there were a bunch of places I didn't I intentionally did not break existing sites because I thought that would be a bad thing to do and instead I created a bigger problem for the future like those tab characters for example accepting and fixing bogus input this is otherwise known as the robustness principle aka postels law be liberal in what you accept and conservative in what you generate this was dogma at the time this is a very difficult one in fact I'm supposed to be writing an article for a CMQ about this right now and I find myself scratching my head a lot about what I'm going to say the idea of the robustness principle is it increases interoperability because if there's an ambiguity in the spec then what you should generate protocol that is as limited as possible by the most conservative interpretation of the spec but when you're reading it you should use the broadest interpretation of the spec and that way everything will work the problem is that that means that broken software doesn't have any incentive to get fixed because it will continue to be accepted and that ends up creating other large costs down the road sometimes very large costs it was the right thing to do at the time because everything was broken when I started but at some point I probably should have tightened up send mail because it you know was the 500 pound gorilla in the living room and I could have put pressure on other things to to get better how I would have done that without having you know people with pitchforks outside my house I don't know but it probably was the right thing to do it is actually to me still unclear what the right answer is to do today the robustness principle seems so intuitive you want robustness and yet it has this dark side and I honestly don't know the answer to this one if any of you have any thoughts feel free to corner me at the break so what would I do differently today the first thing is as I said before I would fix problems as soon as possible one of them was the tabs the bogus features that I've mentioned I see I've got an extra close parent on there sorry about that another one was the v7 mailbox format I didn't want to change the format of far spool mail I use the exact same thing that came from Bell Labs that means that a line beginning FROM space separates messages that's what came to us from Bell Labs burn people say it's the Berkeley format it's not the Berkeley format it's the v7 format I did not change this for the reasons I've described before I'm so sick and tired of seeing greater than from in the middle of text documents and things that have obviously gotten mailed around at one point or another in their distant past I could have fixed this I'm sorry I didn't know I could have fixed it so that's one thing I would do I would definitely use modern tools a lot of tools exist that just didn't exist exists today that did not exist at the time I would use more of the tools but only up to a point I think people go tool crazy an example is send mail has its own basically compilation environment builder the build script which goes around probes everything and whatnot I wouldn't do that today because there's other alternatives out there I do would use more privilege separation which is kind of evolved at the time I wouldn't go as extreme as post fix I really don't see a need to have 40 demons running to send mail but you know that's that's just a philosophical thing I'd create a string abstraction I love the sea language but strings are you know the seas you know nasty you know the relative they want to keep in the closet it's it's the weakest part of sea by far and I should have just gone and fixed it the problem is it's hard because all the system calls of course you have to be converting between these these string formats which is why I didn't do it couple of relatively minor ones I'd separate mailbox names from Unix user IDs today the concept of a Unix account in a mailbox have almost nothing to do with each other and there's still kind of a built-in assumption and send mail about that cleaner configuration file I talked about okay what would I do the same today I did a dry run of this talk for some grad students at Berkeley in the first one there was an audible drawing in a breath then I want to see if it has the same effect here I would use C as the implementation language why I mean C is a dangerous language you know it's it's power tools and there are no safety guards okay but you always know exactly what's going on I've been doing for one job I've been working on now I'm doing a lot of stuff with object-oriented languages and the more I work on them the more I become convinced that object-oriented languages are a mistake or of a mistake that I'm for the object-oriented languages do too much under the covers and C++ is the worst of the bunch because it makes you think that you can do things in a sort of safe object-oriented manner but you can't really so you kind of have to be playing both sides you get the worst of both worlds instead of the best of both worlds I would continue to bite things off in smallish chunks okay now I say this you know I'm not kind of starting to sound like an old man here I'm really not wild about agile programming I think I think it's a cop-out for a lot of things but it is in fact the right thing to do for certain things and I like the concept of building a system that does something and then modifying it as opposed to be doing this big mega design spending years building it before you know you can first do dot slash a dot out and you know so there's a trade-off there the best thing I did on send mail was syslog I did syslog early on syslog did not exist before send mail it was done as a side project I felt I needed someplace to get the logging information that I was going to need for debugging an operation and I produced a generic solution syslog is everywhere now you know your every HP printer has syslog support absolutely the right thing to do I don't know why nobody did it before except that at the time you know you didn't have UDP you didn't have sockets you didn't have so far they use something called MPX files which nobody's ever heard of anymore it was actually difficult to do but it was worth it every bit of it rewriting rules I would still use but I would have a different syntax I would not rely despite saying before I would use tools I wouldn't rely too heavily on tools tools have a cost as well as benefit and I've worked on projects where you know they never saw a tool they didn't like and as a result you have to start off by installing 30 or 40 tools to do anything and oh by the way they interact in various ways and there's an incredible learning curve and every tool is idiosyncratic and so forth there's a time when the right thing to do is to in fact just build what you need as opposed to putting in a tool that's quite a bit larger just one quick example I've seen places where systems that will use Lex for example to do parsing you know basically scanning of input and all they needed was stir-toke okay stir-toke's a nice simple thing that does a particular job does it well but oh no I need to have you know a parser generator to do this that's stupid okay a few takeaways hopefully you can use you know the kiss principle actually does work it works well enough I will point out by the way that send mail was a second system in the Brooks sense and yet it still succeeded and I think part of that was because I actually had enough self-control to to remember to keep it fairly simple as it went along it doesn't look simple today but each step was was fairly simple and and well compared to exchange ever much if you don't know what you're doing advanced designs don't help very much I I say quite happily I didn't know what I was doing when I started off that's partially my own naivete at the time it's also partially because the world that we know today didn't exist at the time and so it's impossible to predict the future for most of us at least these for me and so you need to actually think about the fact that you don't no matter how well you think you know what's going on you don't the world's a messy place so you know just just plan on it flexibility Trump's performance when the world changes every day you see all the time people who come up with these highly optimized solutions to the wrong problem and you do need to think about performance that's really important there's got to be a balance between the two and I really do believe in flexibility more takeaways fix things early okay which as I've told you I didn't now think about it this way if you change stuff and you fail it doesn't really matter because you failed whereas if you succeed the installed base just keeps getting bigger so you might as well fix it now and if you fail then you know it's over and who cares and if you succeed you've done the right thing ASCII is great for internal files and protocols it really helps with debugging a lot if you don't have to build a tool just to debug your tool documentation is a key to broad acceptance I can't begin to tell you how critical documentation is and sadly most open source projects have still have not figured this one out and most importantly the design space is always changing now you may say okay well that was true of the mail system back then but you know doesn't apply today it applies very much today had a discussion with somebody the other day saying he turns off virtual memory entirely because as soon as he gets to the point where he's paging he has a performance knee so he'd rather have his system just die outright when he runs out of physical memory than have it start to go to you know on to disk so virtual memory which was this thing that we absolutely all had to have it was completely the right thing to do now in a lot of situations is exactly the wrong thing the design space has changed okay cloud computing is a return to centralized administration you are in fact handing the keys back to those people in those those glass rooms okay is it the right thing to do or not well for a lot of people it is the right thing to do we move from the glass rooms to our desktop back to the glass rooms a the design space keeps changing there's lots and lots of things we can come up with examples on this so the world keeps changing out from underneath you it's reality if you've got any piece of software that's going to last more than a few years you're going to have to change the fact that the technology is going to change out from underneath you just going to have this by the way is based on a chapter for a book that's hopefully going to appear real soon now the tentative title is the architecture of open source applications I believe it's coming out through a Riley I don't know when it's going to come out the discussion was Christmas but I think they met Christmas last year so and I have time for like one or two questions Eric if you were not Eric and who would I be exactly but in and you started to use a mincer today which one would you use probably post-fix deal respect for Vito when Vito was working on post-fix he spent a lot of time looking at send mail he actually gave me feedback on send mail he there were a couple places where he found bugs several places where he found optimizations and I remember one time post-fix took a lot longer to write than Vito had planned like three times longer and I sat down next to him at a conference before it came out at one point and said so how's the mail are going and he goes well it's taking longer than I expected and I said it's harder than it looks isn't it and says it's been an exercise in humility so now Vito did Vito did some very very good work I don't agree with all of his design decisions but you know I never agree with everybody's design decisions it's it's inevitable some decisions you know I don't like him but they're still right every once in a while you always hear somebody complaining about let's get rid of SMTP or you know let's just ditch the whole thing and start over I'd like to hear your thoughts on on that topic right what was it Ken Thompson was asked what the operating system of you know 50 years hence would be and he says I don't know what it's going to look like but it's going to be called UNIX SMTP is not a perfect protocol SMTP is however probably good enough the big flaw in SMTP is authentication authentication is a really really hard problem to solve because authentication by itself is not enough you know just because I know that you definitely are a spammer doesn't mean I want to read your mail and so there's there's a bunch of layers I actually spent a couple years working on the problem I did work on DKIM for sender domain authentication and and so forth I have thought about it and I don't I have not come up with the magic bullet and I'm not sure the magic bullet exists the desire to be able to exchange email with kind of anyone is antithetical to the I don't want to exchange mail with people I don't want to talk to because how do I know if I want to talk to you or not once again if any of you have any fantastic ideas come on up see me at the break we'll share patent rights one more question well or not but looking at today like looking at instant messaging video audio we integrate anything of this into that would I system what I integrate interactive well first of all you know batch is already there you know mime takes care of doing all of that interactive no I would put it into SMTP it's the wrong place for it SMTP is a store-and-forward protocol it's one of the reasons by the way why things like SPF don't work terribly well because you know they got the store part but they forgot about forward you know it works sort of okay some of the time but you know people kind of a related question I'll answer the question I wish you would ask instead of the one you did like a politician I don't think mail is going away it is being modified people are using it in some sense more appropriately there are times when you want a record you know anything business illegally and so forth when we start to do actually I understand there was a discussion in one of the key notes earlier when we start to get into things like colonies on other planets right you're not going to be using I am you know or Skype or whatever because there's like a 45 second or you know actually for Mars it's like 10 minute round trip time okay you're going to be using email for that so what telegraph yeah well if flash lights up so you know I think mail is going to be around for a long long time it'll get used differently that's inevitable last question what do you think about why and I yeah what do you think about the wife particle wave oh Google wave you know I looked at it a little bit but the problem is I didn't have anyone else to like play with it with you know it's it looks like wave could have been really interesting and I'm kind of sorry Google gave up on it as early as it did because I don't think they got a chance on it but on the other hand maybe nobody picked it up because there wasn't enough there to make it worthwhile I honestly I don't know the answer to that yeah so okay thank you thank you