 So today's talk is not Dave talking about credential sharing because he had an unfortunate medical problem So we all wish Davis a speedy recovery Instead we have a mr. Eric almond And he'll be talking about lessons learned from send mail Okay Hi all I'm back at UC Berkeley and that's about all I have to say here, but I do have a few disclaimers First of all, please be kind to me I Started writing this talk at 6 this morning and I finished it about well. I'm not even sure I finished it a couple minutes ago I Also apologize. This is a BSD conference. There's nothing specifically BSD-ish about this although There were some other platforms that send mail ran on And I haven't worked on send mail in years You know, sometimes it's time to move on and I have moved on so a lot of this is from memory So just some background Send send mail or actually deliver mail before it originated on Unix 6th edition On a PDP 1164 bit address space. I was working on the Ingress relational database management system It was one of the two first relational databases ever implemented. So, you know pre oracle pre SQL pre everything but to our blessing and curse we got an ARPANET connection and Everyone in the department wanted access to the ARPANET and that was a problem So these are the days of Network Wars there was of course the ARPANET there was Birk net which was a local network at Berkeley There was the UECP network, which you may well have heard of there was CS net Which you probably haven't heard of Purdue net deck net, etc. Etc It was very ad hoc and highly heterogeneous Deliver mail was the very first instance of the mail routing some people have claimed anywhere ever I'm not sure that that's true, but I might have a valid claim to have invented the concept of internet mail or Inter lowercase i internet mail It was routing only there was no header rewriting There was no queue because everything I was using already had its own queue. There was no state There was no DSNs delivery status notifications. These are the messages saying oh, I'm sorry. I couldn't deliver the mail So you either got an immediate status or none at all The configuration was compiled in etc. It was very very basic, but yeah, it was quick to do Now I ask you now what do you do with an address like ucb vex colon research bang deck vex bang Whirl colon colon foo at Berkeley. This was a real address At the time in fact an entire book was put out That had nothing except if you are here and want to get the mail there Look to page 386 and we'll tell you how I will point out by the way this was the origin of the success exits dot H file because Deliver mail wanted to get at least a few bits of status and syslog Which a lot of people don't realize was written as a part of the send mail project Also Warner yesterday mentioned mpx files I just thought I'd point out that at least one person outside of Bell Labs used mpx files That would be me and it was for syslog Because there were no concepts of name pipes or anything like that It was the only way I could get multiple programs to flow into one or at least that I figured out so 4.2 BSD comes along or actually CSRG comes along and For Bs 4.2 BSD came out and the internet Berkeley got a DARPA contract to produce the platform for the internet and that led to sockets and all the other stuff that you know about but one of the things they needed was a mail transfer agent that you know get the mail from here to there and and Winge bill joy came to me and said you know more about email than anyone else here, so why don't you do this and Except he said you can't use the name deliver mail It doesn't actually deliver any mail and he was right about that He didn't like send mail either, but I got that through and so you know I went okay, you know And it's some new protocols. How hard can this be? answer is Way way harder than I thought It required stuff like persistent state notably cues, but not merely cues long-lived demon header rewriting so you could avoid those nasty Formats that you saw before delivery status notifications Etc. It's an ongoing list Mistake number one was I tried to maintain the pdp-11 Compatibility because pdp-11s weren't going to go away anytime soon and in retrospect. I probably should have just said You know deliver mail will survive on pdp-11s. They're going to go away eventually and whatnot But at that point 32 bits was like we couldn't even figure out what to do with 32 bits of address space If you can believe that So it seemed like well, this is way off in the future yet The top priority was reliable delivery Deliver mail did pretty well given what it had to work with but some of the other programs that it used like the UCP agents and the Birkinet agent and so forth and so on Weren't always as reliable as they might want to be and I can assure you hell hath no fury like a professor whose grant Proposal was lost in the mail. Yes. This has happened and By the way, did I mention by the way send mail was not my job? I was working on the Ingress project I'm just doing database systems not mail systems. So this was a nights and weekends project Part of the effect of this by the way was I had to like reuse absolutely everything I possibly could because there was no way I could implement everything in the mail system There's a place called MMDF out of or a system called MMDF out of Delaware that did try to Reimplement everything they probably did a cleaner design than send mail They got a million dollars to do that send mail one with a guy working nights and weekends. I Think there's a lesson there. So a brief history As I said, I wrote send mail when I was a student and staff at Berkeley That was send mail version five I went to a startup another database company called Britain Lee and walked away from send mail I figured, you know, it's an adolescent now. It needs to go learn how to grow up in the world That analogy may have been way more accurate than I thought at the time Send mail splintered every vendor had their own version slightly different syntaxes slightly different semantics and plus there was IDA out of Sweden I can't remember exactly where and Paul Vixie who I saw sitting around here did KJS King James send mails and attempt to unify them all Well, Paul Paul did some great job. There were several others I Returned to Berkeley on the mammoth project at this point doing infrastructure to support research and One of the things was centralizing things because at that point every research project was running their own system Their own mail server their own doing their own system administration most of it badly So we're trying to fix that problem and in particular Instead of having everyone's email go specifically to their work station We wanted them to go to a centralized mail server and I had to get into send mail and as when I started doing that I Quickly found myself drawn back in and I started merging versions Using ideas here and there and that ended up being version 8 I left Berkeley again. I had a totally disastrous job I went off and I licked my wounds for a while and then I co-founded send mail Inc. And so I Can't seem to get away from send mail. Maybe this time I Left there after 13 years and ultimately I've ended up back at Berkeley where I am now So about standards One of my favorite topics not The standards at the time were rudimentary at best As in I tried to find out how email was sent on the ARP in that and it was not written down It was all word of mouth Or you could look at the code which took a while it turns out it was part of FTP nobody bothered wasn't looking at FTP but What standards were there were still in development they were often in fact more often than not they were Ambiguous in a lot of cases. I got yelled at for not following the standards And it turns out I had just Misinterpreted them because there were multiple ways to interpret the same standard And the standards were undergoing a lot of revision for the internet in many cases they changed literally overnight and So one of my successes I think was rewriting rules at least as a concept Which people have said oh, you know, that's absurd, but when these changes in standards came out I could usually have them implemented by the next morning So I'd literally go home do the latest round of revisions and send it out that night It one of my philosophies has been Still is Sort of mostly that sometimes is easier to solve a general problem than a specific problem And this was an example of that The syntax on the other hand I probably could have done a better job You know it's fine when config files were you know a dozen or so lines long and whatnot, but When they got to you know tens hundreds of lines I'll talk more about this later By the way, the the thing of this those surely the standard stabilized the internet was up and running and so forth No, a few years later the drums working group came along to revise all of it And I went back to my thing of you know doing stuff nightly and so forth so Standards change They're mutable And I will say the IETF process around Messaging standards in particular leaves a little something to be desired I stopped going to the IETF after I walked out of a meeting one time and discovered I had a knife in my back and That was the last meeting I ever went to So rewriting rules Well, some mistakes were made tabs Get so much shit about tabs So I stole the idea from make Stu Feldman You know I said well if it's good enough for make and you know Stu's obviously smarter than I am and so forth and so on Then surely it's good enough for send mail he said it was the biggest mistake he made in make even worse than dollar less than dollar at the Totally non-nemonic things Stu explained to me that he There were some things that make wouldn't do and he went in one night He was one of these people who came in at you know six o'clock and worked all through the night And then went home at you know four in the morning kind of thing and he did this and he threw this stuff in and he Didn't have time to think about something rational. So he just put it in and picked random characters I'll go back and I'll fix it tomorrow and By the time he got back Ken had done a distribution of Unix which as you heard from Warner was a casual thing at best And so Stu said oh, it's too late to change it No sorry I could ask Stu he might remember So lesson here don't blindly follow the pack It was a pretty obscure syntax once again, it was okay when they were small, but it got more obscure with time I mean, what are you going to do with that? I mean I can tell you what that does now imagine a hundred lines of that and You know, I look at config files now and I go cross-eyed One of the things was part of it that was the 16-bit legacy Didn't have a lot of extra memory for parsing and things and Partially just laziness simplicity. So everything was single letter macro names option names So forth that ultimately did get fixed, but it was a painful process and it could have been avoided At one point. I actually literally looked at the config file Real I think I actually printed it out on paper, which was the first time I'd seen more than 24 lines of it at any one time and Was fairly horrified But it was running on a whole bunch of machines on campus Some of which were in the computer center and getting them to do anything was a pain and so I just said fine I'm stuck with it. I Could have done better You know support would be so much easier without the users One of my mistakes was letting my installed base crippled me Compare this to for example the C language I first started using C right around during the transition from version 4 to version 5 So I use version 4 a little teeny bit Didn't get to when I got to Ingress I started using version 5, but the language changed between those two versions because Dennis, you know looked at it and said, you know, we can do this better And then I remember very distinctly the version 5 to version 6 Transition for Ingress where we basically had to go through the entire code base, which was fairly significant at that point and move to a Subtly new language and then guess what we got to do it again when version 7 came out and You know at the time I was like is this really necessary and so forth but I think part of the reason C is such a beautifully clean language is precisely because Dennis came along and said no that's wrong. I'm going to fix it and People will get used to it You know, I was concerned about a dozen sites, even a hundred sites Big tens of millions of sites. I had no idea it was going to be anything like that and I Probably should have should have shot a little higher than I actually did I Should have been willing to change more things One of my great regrets is that in the view that I didn't want to screw up existing users I would not change defaults in the configuration file because then it might change the beat You know they get a new release and it would start to behave differently, and I didn't want that The effect was that anyone installing something new in order to get reasonable defaults had to go in and manually put in All this stuff in the configuration file and looking back on it. I can only kick myself It was an incredibly stupid idea. I did it with the best of intentions, but it was wrong And I should have just taken you know the Dennis's approach and said They'll get used to it Do the right thing But I do want to warn you you know be careful of blindly following You know new trends legacy really does matter. You don't want to like arbitrarily leave your users out hanging in the wind and You know those shiny things they look so shiny right now, but they corrode and Shiny bits start to flake off and you know so forth. Oh squirtle Starting a company this isn't really relevant to the whole thing, but I thought I'd throw it in I started Sunmail in an attempt to get back to Coding because I was spending all of my time Basically doing support most of which was reading email saying look at the fact question number, you know 17 Da-da-da, and that was taking literally all of the time I had for Sunmail and I wanted to move Sunmail forward So I thought I had a company You know I can hire some support people they can deal with all of this stuff and I'll get to go do the fun stuff I Am glad I did that I learned a lot Sunmail was one of the very first sort of hybrid open proprietary Source companies it was totally new ground we made some mistakes, but a lot of what we did is still the basis for the way these companies work today We're the ones with the arrows in our back, you know It was quite a roller coaster ride Very exciting at some times very sad at other times We I got to ride the heady part of the internet bubble rising up We went from two people to 200 people in What two years something like that three years maybe But on the other hand we also got to ride the other end of the bubble down And that was no sudden we went from 200 people to 80 Even more quickly than we went up But we did survive and which made us better than a lot of the folks out there Yeah, sorry we had offices all over the US and you know the UK. I got a lot of frequent flyer miles back then Frequent flyer miles you may notice you don't get a lot of coding done on airplanes. I Was a complete failure on my goal I never did really get back to coding. I didn't realize that my role was going to be you know The trained monkey being taken into customer sites, you know, you're so special We've got Eric Alman himself for you to visit with I'm sitting there trying to remember not to pick my nose So I basically didn't do any coding for a decade it was hard to get back to it I have to admit The other thing is it really tied my hands regarding technical decisions at times companies talk a lot about innovation but at the same time companies are very risk averse and getting the balance between those things is really really hard and There were definitely some things I wanted to do with Sunmail and I was told that did not fit into the product schedule I do have an entire talk It's a little old right now. I'd have to update it, but you know, I think it's called so you want to start a company So some random other observations. How am I doing on time? Oh I've got tons of time. Okay Some other random observations this is about code both good and bad People said why did you use m4? Well, you know answer number one is because it was there Actually people m4 has a bad rep, but it's actually a really pretty cool language. You can do an awful lot with it It's a little confusing at times. I admit but The first attempt to do it was essentially a procedural thing just Okay, take this snippet of code and plug it in here and so forth. So it's just a very simple macro language That failed miserably because ultimately it was just syntactic sugar Then I made a second attempt with a more declarative style I want this feature in add whatever bits of code you need in order to make that feature work That was mixed success I think I would say Most it was certainly better than what we had before and it was better than raw config files and beyond that I Really have never quite decided on my own. I was a little OCD back then. I'm actually still a little OCD and I spent way too much time trying to tweak the m4 so the output of m4 was pretty and I should have realized that No, if the m4 was successful the output was assembly language maybe even binary code and and It was not worth it and I literally I can't imagine how much time I wasted doing that I'm sorry Putting all the functionality into one binary there was a time when that was really convenient because I didn't have to worry about Breaking stuff out into libraries and so forth. There would have been a lot of replication But no, that was that was a mistake. I got some stuff out of it, but not that much I did pretty extensive wrapping You know Between Ingress and send mail I learned a lot about portability more in send mail than in Ingress and To this day I still write portable code for Everything even stuff that I never anticipate to run on more than you know a raspberry pi in my basement but One of the things I did was I wrapped a lot of things Inside other code so that I could come back and tweak the Code later the project I'm working on right now at Berkeley for example. I've got a library that among other things all of the Threading primitives, you know mutexes and so forth are I don't use p-threads directly I call a routine which calls p-threads that has allowed me to do things like add debugging in When I really needed it, which you can't get out of p-threads and so forth So I tend to put a layer around just about everything I use in the standard library You know that that does make my code very idiomatic, but so be it One of the things that send mail does I don't know how many of you have actually looked at the code But internally it uses fork without exec To do memory management, so it was basically poor man's threads Not quite threads because you're running in a separate address space You know together with a memory management thing we had a great way of backing out the stack it was called exit and and You know that actually worked pretty well Fork exec I mean the obvious thing to do it is you invoke a new something or other and so forth But then I would have had to reread the configuration file I would have had to reread all the state and so forth I would say on the whole even though that was very weird and did create some problems at times particularly when other people Other operating systems that called themselves unix Decided to change the semantics of fork and that got me into trouble a few times, but I thought of that as their problem not mine So Mostly I think that was a good idea Moving to design space I as I mentioned before I leveraged everything I possibly could there was no way that I could Reimplement the entire world and I'm really glad I did that one thing I didn't mention earlier, but in the early days particularly in version 6 People had a way of customizing their sites and Customizing their sites included such things as changing the format of etc password And so If when you and remember this was before get pw name So you had no wrapper around it the way you read the password file was you said open it set for password read and You parsed from there By leveraging everything if some site went in and said okay, I'm going to change it set for password Anything that had to actually read the password file was localized into bin mail And I didn't have to worry about that and I can't tell you how much pain that Saved me With Ingress we actually did get into this thing, which is why I didn't do it in send mail steel from the best You know, there's no shame in using other people's ideas the work that was done in IDA and KJS were Was you know instrumental Sunos believe it or not did some good stuff although I did tend to take any ideas and Generalize them so IDA had the concept of database maps using the DB Sorry DBM Software I turned that into a generic Mapping thing where you could define new types and use that for a whole bunch of things Yeah, okay, that's the same thing You know encourage outside innovation whenever possible I did not do this as well as I should have we did At send mail ink we added the milter interface which is a way for people to do mail filtering outside the actual MTA which Created a lot of innovation right there, you know, I mentioned earlier that the Priority number one on mail was reliable delivery and that's where it started and by the time I left send mail the priority was throwing away as much email as absolutely possible and Frankly, I find that kind of discouraging but there it is And when by the way, I don't hate threading even if I didn't use it in send mail The milters could be threaded if they wanted to be they could you know run event-driven or threaded Don't let yourself get too caught in the past You know, I definitely failed on that in a few places. There were times when I should have gone in and said, you know this really sucks I should just reimplement it and Part of it was I didn't have enough time But I probably wasted more time maintaining the old stuff than I would have spent if I had just gone back and done it over Of course, those are value judgments. You never know in advance and that's what I've got Any questions? Oh, and I think I'm supposed to ask you to use the mic and I see Paul is going to come up and harass me They're applauding you Paul Close to the mic. Okay That was great. I knew most of that from living it with you, but There are two things I'd love to hear your perspective on Frozen config files. Oh And I'll sit down for this but I want to say We just edited the binary output of m4 and stopped using it immediately I mean by we I mean most of the people who used send mail We're not able to change the m4 source stuff to do what we wanted So we ended up hacking the config and then deleting the m4 now I now realize that if you had used p open called m4 in real time when reading the config file So there we could not see the binary code. We would have had to do what you wanted Bastard you waited all this time Okay, enough of that one frozen config files That was a technique that was used by Some places So frozen config files I should explain Since it was fairly Difficult to read in the configuration files and parse all of them and build the internal memory structure And you wanted start up to be as fast as possible when it running in kind of interactive mode because you could just pipe things to send mail and it would Deliver it off into you know, wherever So a couple of programs back then use this idea of you reading your configuration file and then you write out data space To another file called the frozen configuration and when you start up you just read data space over on top So you've got all the same Information, of course if the binary changes even a tiny bit then things like pointers now point off into never-never land and You know it at some point became pretty clear that that was way more Causing way more trouble than it was solving And so they got dumped entirely Certainly if I understood the implications of it I never would have wasted the time putting it in in the first place. I mean it was actually kind of Fun to do and you know neat concept. Oh my god. This actually works except of course it didn't So other than that Any other questions as someone that's been fortunate enough to avoid send mail I was wondering if you could go back to the slide where you had that configuration of Characters and just explain a little bit about what each of them is doing Probably somewhere, okay this we need a trigger warning here Yes, this may cause some people undue distress So the first letter of the line told you what the semantics of the line were our was rewrite So it's a rewriting rule It still is dollar plus matches so send mail first takes the address it breaks it up into tokens Standard compiling technique I'd taken a compiler class to and By the way that the the whole rule set thing. It's just a simple production system Come straight out of expert systems. I had taken an AI class And you know a friend of mine who was a mathematician Who said she really avoided send mail because she'd heard all these terrible things about it And she finally couldn't put it off anymore And she ended up taking the manual home and she reads it and she goes what's everyone so upset about it It's just a simple production system So it turns out if you're a mathematician, it's no problem So dollar plus matches One or more tokens dollar star matches zero or more The at sign matches an at sign Dollar equal w matches any single word in the class Equal in class w which later got changed into any Sequence of words in the class w because that semantics turned out to be a little wrong And if you do match that then replace it by dollar one, which is The dollar plus an at sign which is an at sign and dollar D Which is a macro which happens to be the local domain name. So this is a way you'd take arbitrary things that might match your Local domain and turn it into a canonical format. It's all it is And oh good post fix was it a blessing or? You know I Really like pizza And when he was working on post fix He spent a lot of time he looked through send mail code a lot Spent a fair amount of time Interacting with him. He's you know, why did you do it this way? You know I would Justify it as best I could a couple times he found bugs and fed them back to me Post fix took a long time to write and remember I was at a conference at one point and There was some talk it was one of these things with you know a massive room So you could actually sit in back and Pizza was sitting back there in the dark and I sat down next to him sort of leaned over and I Said so how's the mail are going and? He looked at me and he went it's an exercise in humility Vita like me discovered it's a lot harder than it looks That said he didn't start with a 16-bit legacy He had a lot more to work with the that mishmash of random Special characters pretty much doesn't work anymore We the whole world is pretty much decided that user at domain and is good enough for everyone Which was most emphatically not the case when I was doing rewriting rules rewriting rules were specifically because Everyone wanted a different way of interpreting email addresses And now you know he did not have to deal with that legacy I'm actually people go well, you know, aren't you ashamed of send mail? You know da-da-da. No, I'm proud of send mail it was born into a specific niche in the world It did a really really good job of filling out that niche it's Really really good that the world isn't like that anymore and That we have some sense of sanity and that makes a lot of things easier notably parsing of addresses The other thing was You know again partially because of the legacy You know started send mail in the day before sockets were existed certainly named pipes and So the idea of having arbitrary processes starting up communicating with each other could be done But it was really hard to do and so pretty much everything had to have a common Antisedent So you could create an unnamed pipe and then pass the file descriptors down the tree And that limited a lot of what you could do So Vita had a lot of advantages, but you know he did an exceptionally good job. I think and You know It's good work. What can I say the one I cannot quite understand and maybe I shouldn't say this on tape is Why Exxon has continued to be so successful when it has all the architectural flaws that send mail does The only thing is it It's easier to configure because it attempts to do less and You know, I totally get why somebody might want a post-fix like architecture That makes total sense to me. Hi the first time we met in 2011 I walked up to you and I said hi. My name is desk I used to use send mail, but I got better and I would like to publicly apologize for that Thank you Plus sign in the local part curse or blessing oh plus sign in the local part so This actually came out of the Andrew project at CMU The idea was that if I my login name is Eric, which it is you could have a arbitrary number of addresses that were Eric plus Some other tag and you could use that for things like mail sorting and so forth. I still use this I actually don't use the same syntax now because You know because I run my own mail server, I can just say everything at something in this domain is Effectively Eric at the the original name I do that because there's a surprising number of sites that think that plus is not a legal You know valid character in a username which it is And I just got tired of dealing with those a pox on them Generally speaking, you know I for example have a different email address for pretty much everyone I give my Address out to so if I get something that claims to be from my bank Which is not to the address that I know my bank has I know a priori that it is a fake and That's happens surprisingly often Now most of the time the fakes are so bad that you would never believe that they were actually from your bank But occasionally, you know you get one where it's like, huh? Looks plausible, but it's to the wrong address. So I'm done. I You know, I don't know if plus is the right syntax, but I think the concept is a great one and Once again, I just stole the syntax not my creation at all Any other questions? What kind of user agent you're using right now user agents I Am currently using Thunderbird There are days I curse it I was cursing it day before yesterday when it suddenly decided that it needed to take up a hundred percent of my CPU and After an hour it was still using up a hundred percent of my CPU and my fan was running crazy and it was sitting on my lap and I was getting burns on my legs and You know, I have absolutely no idea why no clue apparently Thunderbird just likes to do that occasionally but You know, I've got it, you know, my fingers are programmed with all the hotkeys and so forth So it's really kind of hard to convince myself to go to something else and I've looked at other things including commercial products and Basically everything out there is bad so Okay, I guess that's it So thank you very much for stepping in at the last moment. We have a small token of our appreciation and your timing is perfect