 So, if I really wanted to stand up here and give a talk that would change your life as a developer, I would be giving a talk about task.acxtreme. But that is not the talk that I am going to give. And sadly, the talk that I signed up to give is not the talk I'm going to give either. So all good talks start with a good story. And my story starts not quite 30 years ago when I was an administrator for one of these things in my mid-20s. It was actually a little bit newer model than this one, and I hate to say it, trying to find photographs of mainframe computers from the late 80s isn't as easy as the task as you would think. It's kind of like bad scans of awful marketing material. But this machine is quite, well, I'm like 6'2", 6'3", and I could see over the top of it. It's, you know, this tall. It's 30-some-odd inches deep. That right there is about six or seven, maybe eight feet long. And actually, once you put the disk drives on it and other stuff, this is a machine that goes from here to the wall. It's on a raised floor. The rest of the room that was about this size is a giant IBM mainframe. Now, the mainframe, the IBM people were kind of a different team that I kind of worked with only sometimes, but this was my problem, and it was a university. And this was what students and faculty used to do computing. And as a 20-some-odd year old going to school, we were telling war stories last night with Dave, and for me, I actually, Dave was a little bit older than I am, and admitted that he got the keys to the computer and could go in and do whatever he wanted. I got lucky enough that I was paid as a job to come in and babysit this thing all day long. So, one day, we're sitting there, and I have a terminal, old green screen terminal. I actually have two on my desk. One of them sits over in the corner and runs a monitoring process that's very similar to kind of like top, except it's showing all the logged in people and things like that. And it wasn't uncommon for there to be 100-plus up to about 300 people logged in to this system. They were also all on terminals. PCs were just kind of starting. So this would have been like, you know, Mac 2 days, you know, maybe a 386SX, maybe, 286 probably. So I look over and out of the corner of my eye, I notice that the screen doesn't refresh. It stops. And then all of a sudden, a few seconds later, it refreshes, and it was like, oh, shit. Now the, since this is a talk on DevOps and deployment, the profanity clause and the code of economy has been removed. So the screen does it again. It kind of gets a little ways, and then it just hangs. It's like, oh, this is going to change the rest of my day. About 20 minutes later, as I'm trying to figure out what's going on, my Ubeenia boss comes in and says, are we having problems with the Vax? And I looked at him and said, no, the Vax isn't having a problem, but I'm pretty sure somebody else's code does. And I said, why are you asking? Because that's probably going to tell me who's having the problem. Because at this point, there's, like I said, hundreds and some odd people in the system, and I'm trying to figure out who's done what that's causing it to hang. He goes, well, I got this call from the psychology department, and I was like, oh, crap. Because I'm not allowed to talk to the psychology department anymore because of past history. Like trying to explain to the chairman of the psychology department why his code that does a divide by zero isn't going to run. I'm not allowed to talk to them anymore. Fortunately, my boss is really good kind of a people person, and he talks to the psychology department. And I said, well, did they say exactly, and he goes, no, they just called to see if there was something wrong with it. And I was like, well, we're having these hangs. And he goes, did they tell you who it was or anything? Not really. But that gave me at least a place to look. So we tracked it down, and it's coming from some terminals that are up in the library. And my boss and his boss and the head of operations, it's like, well, let's just walk up there. Because this is kind of escalated at this point. And other people are calling because why is it just hanging up? So we get up to the library, and after a couple of minutes of observation, we can see that they are running a survey on all 20 terminals that are in the front of the library. They've logged into all of those, and they're running the survey. And I finally figure out that in the survey code, that they have added a variable weight that caused this pause after you entered the question. And to create that pause, they had written a for loop that counted to 4 million. How many of you paid attention during Sasha's talk? This was before true preemptive multitasking on a vax. It was kind of, but realistically on an interactive scale, you could want, it really was a soft scheduler and you could lock it up. And these 20 terminals were locking it up. So we got them to kind of back it down, and we got to looking at it, and somewhere in this process, the person who actually wrote the code shows up. Now there's a line of students to come take the survey because it's for the head of the psychology department, the chairman of the psychology department that I can't talk to. It's his survey that's being administered by some graduate students. And they're wanting to see what happens with this variable pause reinforcement. Frustration was what it really was. And they couldn't figure out why it was even more variable than they thought it should be. Which to me was like, okay that should invalidate your study, but to them it was just like oh yeah, who cares. So I looked at him and I said, you can't keep doing this, you've got to take it out. And it's like, well I can't, it's what the study is for. And I said, well you need to change it to a sleep. And the guy looks at me and he goes, he logs on the terminal and you can bring up help and he searches the manual and he goes, there is no sleep in VaxBasic. And I sat there and it was like, what the hell? And I look at the other guy who's my boss and he's mainly a PC guy. And he goes, oh yeah, there's no sleep in basic. And I'm like, huh? And then our director, who's a mathematician, an incredible woman that I want to say I really am glad that I started my computing career working for an East Indian woman who is a mathematician. It changed me forever. But she looks at everybody and says, oh that's okay, Paul will write you a sleep and he'll get it to you later today. And she turns off and we're all supposed to follow her back. And I was kind of like, I'll get it to you. So we get back to the office and at this stage in the world, we have from about here to that wall and all the way up to the ceiling, a wall of manuals from DEC who makes this machine. They're orange binders, three ring binders. And you would get, like every quarterly, you'd get this giant box and you'd go through and change out chapters. And it would tell you to change this page and it was literally a wall. And I hate to say it, we'd let ours kind of pile up. We wouldn't, I don't know anybody that was really good about keeping their manuals up to date. So I start going through the manuals and it's like, okay, I need to add a sleep to basic. You'd think, shouldn't be that hard, right? Well, what I got into was, the next morning is still not going well. I still haven't figured it out. And so we have the magic 800 number for DEC and I called him and I said, hey, I've got this problem. And you know what? The guy on the other end of the phone was like, oh yeah, I know that problem. What? You know this problem? Yeah, we've been arguing over whether we put sleep in the basic for a long time. What? And I said, well, can you give me some hints? He goes, oh, I can do better than that. I can send it to you on a tape. No, they're talking about sending me a nine-track tape through the US Postal Service at this day and age, maybe UPS. And I said, okay, that's great. But can you give me any ideas about how to write this code? Well, I mean, it goes into, you're going to have to basically save all the registers in the CPU, you're going to have to call, it turned into, by the time I was done, it had taken me almost about 48 hours from the time this whole thing started to produce about 120 lines of code that created a sleep that worked with basic. Because at this point in time, basic didn't even use the same call stack order that the rest of the vax did. It was kind of a hybrid language. And literally calling system calls from basic was just a nightmare. Now, fortunately, down the road it got better. At this point, anyway, so we got it, we did get them happy. We did get the survey going again. They only called, you know, about 50 times in those two days. And then, like on Friday or Monday, I can't remember, it's been a long time now. The tape shows up from deck. I slap the tape in, I load it up, and then pull it up in an editor, and start looking at it, and it is 2,500 lines of code. And about two-thirds of it is an assembler. And there's about, oh, 15 files to go along with it. Now, I've got a hundred lines of code approximately, and I get the official one that they wound up adding to the language in the next release, and it's 2,500 lines of code. So we've gone from the student who created something that worked, at least for him. He solved the problem. To me, solving the problem a little bit more, to actually the one that, you know, the engineers at Dexon in, of, hey, there's a lot of other things that you need to think about when you do this. Now, a lot of that had to do with things like clustering and dealing with older versions, and so some of that code could have gone away, because we weren't clustered or anything like that. But one of the things that this story reminds me of is, and I saw this quote just the other day from a Google SRE blog, is that we tend to not think of a lot of the people that we work with as really as smart as we are. And when I hit that problem in the university where that student actually, and most of the time when I faced a student or even a faculty member and we were talking about that Vax, I knew more than they did. There was hardly ever a time where they knew more than I did. And when he told me, it doesn't have, it doesn't have sleep, it's like, how did I not know that? And, you know, I was kind of like, how did you not know that you can't put it in a hard loop? That's like computer science 101. So sometimes we take some of these things for granted. And occasionally in the elixir world, we get all excited from some of the things that come along with Erlang and the Beam. We get all excited about things like concurrency and distribution and hot code reloading. Those things just get us excited. I mean, our first couple talks have been on concurrency. A lot of the talks here have been on concurrency. Occasionally it feels a bit like, I've got a hammer. I've got this hammer and I can solve any problem there is. Does anybody understand that problem? I see a few hands go up. Because we have internalized this concept into our deities. It is so universal that people thousands of years ago understood that I have a hammer is a concept. In fact, that hammer is the source of his power. If he lost the hammer and he didn't have any power anymore, I get confused. So anyway, I was supposed to talk about microservices, which is another one of those, I have a hammer. When we talk about microservices, I got to thinking about the world sees microservice, because the computer world sees microservices just a hair bit differently than the elixir group and the earling people seeing microservices. So I found this great slide that explains microservices. We start with spaghetti code. We then went to pie, which is somehow, what does it say? Everything is coordinated and must fit into the overall design. That's awfully optimistic, isn't it? Then we get to cupcakes. I like cupcakes. But when I saw this slide, this is what I really thought. And I love this quote that I found. We've gone from building a single ball of mud to orchestrating a lot of shit. Now you all laugh, but over the last couple of days, I've done, I've walked around and I've asked a lot of people about deployment stories. Or a deployment story that is like, heck yeah, I have a good one. I get a lot of nervous laughs. Hold that, we'll get back to it. But on the microservice front, we also get into, Jimmy's in a Slack channel. I don't actually know Jimmy, and I guess he's famous in the .NET world, but he's in a Slack channel and he had a good quote so I stole it the other day. microservice teams that I rescue, they take the lack of tooling as a risk. They took it as a challenge. I climb the hill, next let's do a K2. They're dead frozen on the mountain. So many times when we look at things like distribution and hot code reloading, we get all excited and we forget to really look at those risks. We kind of get in that mentality of we got one little thing done and it's oh yeah, instead of realizing that we had 50 more pushups to do before we got the reward. So let's talk about deployment just a little bit. That's what I really wanted to talk about. And deployment tends to get this look from most everybody. And Paul Schoenfelder who wrote Distillery, when he gave his talk at ElixirConf, I pulled this out just because it was one of the first slides I made. Deployment is hard. There is an amazing number of ways that you can deploy today. It is vastly, virtually unlimited of how you can do deploys. In fact, it creates a lot of quite honestly chaos because there is so many different ways. Now, Josh Adams who runs the daily drip did a survey that came out in December which actually motivated most of this talk. And in the survey he says most 95% of the people that responded said that they're web developers. Only 14 or 12% said embedded. I'm not sure what the other ones do. Maybe they're really earling developers and they took the survey anyway. And one of the questions is how do you deploy your Elixir app? 33% are like some ad hoc way. You know what I thought when I read that? I thought they're probably copying it out of the server and doing a mix Phoenix dot run. That's what happened in my mind when I saw that line. We have between Docker and Heroka we have 60%. E-deliver is at 14%. Now, Josh included some commentary when he did the survey. And he comes in and says we need to expose users to releases earlier. And then he says, now those are his highlight. He bolded these, not me. We need more examples that clarify the simplicity and value of Erlang's distribution and Docker and Heroku inherently cripple the value the virtual machine can provide. Now, that last statement about Docker and Heroku just pissed the hell out of me because it's like, wait a minute, 60% of your people are saying that they're successfully deploying on Docker and Heroku and you're telling them they're doing it wrong. It's how I read it. Now, obviously, I tweeted, I tweet in anger most of the time. So I tweeted in anger to Josh and he responded with, yeah, I've gotten a lot of feedback on that one. Not everybody agrees with me there. So let's, can I, I had to use this photo because this is really what I thought Josh was doing to me. It was kind of like he's the monkey that's just trying to piss me off. I'm disparaging your friend Josh. Josh Adams. Was he a sponsor? I do recommend his podcast, his cast daily drip. It's I pay him money for it and I do recommend it. So hot code reloading is one of those two things that he's inherently talking about. You can't do with Docker and Heroku. What I am here to tell you is that every deployment talk that I have seen in Elixir up to this point has done a really amazing hot code reloading demo. What I'm going to tell you is do not do that. You don't jump into hot code reloading day one. You don't try to go scale Everest the first thing you do and it's that hard. It's that complicated. It's not nearly as easy as the demos show. You can get there. And I think when Francesco and I were talking about this yesterday, he was like, you don't, you know, go this way. You take it this way. We saw some easy problems first. And in fact, the Paul who writes distillery, his comment is unless you have a strong reason for using hot upgrades, it's almost always similar to do rolling releases. Until you get to the point that a rolling release absolutely doesn't work for you, that's the point that you do hot code reloading. I can't tell you how many people I've run into that bailed out of Elixir because they couldn't figure out hot code reloading in the first 30 days they were trying to do a project. Because it's tough. And it's going to work, you're going to think, oh, I'm going to get some easy returns out of it and then it's going to break. And it really takes a lot of work to start figuring it out. You're going to dig into Erlang deep. So don't start with hot code reloading right away. The second one is distributed. So when we run under Heroku, we obviously, we can't talk. The only thing you can do is talk to the outside, the web connection. You can't talk to other machines. With Docker trying to network between multiple machines, it's almost, unless you're running under something like Kubernetes, it's almost impossible to also get out of that, to do the networking for the cluster. If you're running something like Kubernetes, actually towards the end I'll show you how to do that. But again, unless you've sat down and watched Francesco's talk about twice and you've read the last three chapters of his book about 10 times, and you can actually take your risk checklist for what are the risks of distributed computing and put down here's what we're doing to deal with those risks, you're probably not ready for it. In fact, if you look at Francesco's book on scaling Erlang, the last three chapters are purely about clustering. And what's interesting about them is they really aren't a how-to. They're really three chapters of, oh, here's all the things that you've got to think about. Because things get complicated really quick. And it tends to be one of those things that we feel like, is it not going to play the video? It feels like just as we get... Yeah, just as we get to the top, we feel like we're just falling back down. Actually, I could let that video play for a while because it just kind of makes me feel... Oh, we're back to that. So let's talk about releases just a little bit because I really have no idea how much time I have left. Am I really 35 minutes in? Okay, so releases. The thing you need to know, there are a lot of you here that are newer to Elixir, and one of the things that you need to understand right away is that this is not Node or Ruby. Those are interpreted languages that are not designed to be compiled first. Erlang doesn't really... It compiles the Beam files. We create a release that also includes the runtime. Actually, if you look at Francesco's book, the longest chapter in the book is 53 pages long, and it's all about creating releases. One of the things you need to understand first is that your environment that you are going to build the release under needs to be the same environment that you deploy to. Now, there's a few of the smarter kids in the room that are looking at me going, oh, that's not 100% true. It's only 99.998% true. There are a few exceptions, but don't you think that you're the exception because you're not? Because what you're going to do is you're going to add ComeOnIn or you're going to add Jokun Java token libraries to your code that's going to compile an F, and once you have that compiled code, it's not going to run and you're going to have problems. And I'm not talking that they need to be close in matching. They need to match. Compiling under Ubuntu and trying to run under Debian is going to cause your world to pain. I know somebody who just fought this problem for like a week, and I was like, yeah, they're doing the same. And they're like, yeah, they are. Why am I getting these open SSL areas? Because it's different open SSL. This one comment, this one slide. If you don't go look at task.acingstream, remember this. So distillery. The next thing I'm going to tell you is that as I ask people about deployment, a lot of them experience pain with Docker. I'm going to tell you that one of the ways that you can be successful is to look at using Docker as your way of creating a release. So we build the release with a Docker container that matches our deployment environment. We then take that release and either put it in another Docker container or we deploy it out to the server. But that gives us a way of doing that. And it's not particularly the easiest thing in the world to figure out how to do, but you can. First is that we're going to add a Docker ignore file, which is just like our get ignore, so that as we copy files, it knows what not to copy. And I'm sorry, that's kind of small. You can see it on the slides afterwards. We're going to create a release directory or a releases directory, which is kind of how EXRM used to do it and distillery now uses the build directory. So we're going to create a directory that we can save our releases to. And we're going to add that to our get ignore. We're going to have to open up that directory to be writable so that Docker can write into it. And then we're going to tell distillery that we're going to put, our output is going to go into that specific directory. Otherwise it's going to use the build directory. We're then going to create a Docker file. I've done something really simple here and just used the mainstream official Elixir one. You can get as crazy as you want on what you use to start with. I don't particularly love the idea, I tend to want to use something like this or I tend to want to do one of my own that's completely mine. I don't tend to like to rely on others except for kind of as a boilerplate to start with. We do things like put an environment thing at the top so that we can force the whole thing to reload because Docker does some nice caching for us. We install hex and rebar which don't come standard in it. We tell it where we're going to put our application and then we copy our application into it and we get our dependencies. We stop there. This way our stuff is basically cached in the Docker container and then we can come and tell Docker we want to build that file and then we can come and tell Docker I want to take the releases directory that I've created. I want to mount that into the Docker container as the releases directory and then I want to do a mixed release which is just the distillery command to create a release. As we do this, we create a release. You can actually with distillery you can wind up with multiple releases that come through as a tar.gz file and this is where the fun actually starts because at this point once I have a release it's really just a matter of how do I get it to the server and there's a lot of, quite honestly, for me personally that's just a copy command. That's just an scp to get it out there. You untar it, you then issue a command to restart the process. There's fancy ways of doing it. For some of you, when I look at things like Kubernetes I'm actually taking that release and then packing it in a second Docker file. In fact if you look there's articles out there about doing a two phase Docker or doing a double Docker. You do one to compile it because we have this giant container because that build container that I showed you if you do one for Phoenix and actually have node in it too that container is like a gigabyte big. I mean it's huge. 800, 900 megabytes. For deployment that's kind of crazy. We don't want to do that. So we really create a second one to go through and actually hold the release. Clustering. I'm like totally out of time I think. Anybody? They're not in here telling me I have to quit so let's just keep going. Clustering. Now you're going to go out and think I need to cluster and you're going to do a Google search and you're going to get this document right here. And at the end of this document let's see what the next slide is. Oh, there we go. This document. Sorry, getting ahead of myself. And it's going to tell you that you add a name and a cookie onto your... onto the IEX command and boom voila you have clustering. And it does work. It works great. Until you figure out that, well, wait a minute, I don't actually know what the IP addresses are for my machines at runtime. Because we use all this automated stuff and it could be anything. Anything. So then you spend some time digging and looking and you come across a couple of other packages. Paul Schoen-Velder who writes distillery and Timex has a lib cluster. There's another one called Perige. Lib cluster has ways of doing it with the Kubernetes API. It also uses the built-in EPMD daemon that comes with Elixir. And then he's got a gossip protocol where it basically does a multicast to try to find the other ones. Perige is a DNS based. They do DNS lookups to try to find the other hosts that are there. Now when Sasha was given his talk he had the thing that did the summation. And if you notice 1 to 100, how does, if you have 100 Erlang nodes in your cluster how many connections do you actually have? 5,050. Because each node connects to every other node. Realistically right now today I have seen 50 used as about the biggest Erlang cluster. If you really go through some links you can probably get it up to 100. That's going to change. There is some, I mean if you're into it, there is some absolutely fascinating research and articles going into scalable distributed Erlang. The word you're looking for scalable distributed Erlang of what they're doing. They're looking to do a 10x magnitude. They're looking to go up to 1,000 at least. It will basically be a way for you to be able to create a tree of nodes and have basically individual ones that then connect. Take the process supervision concept that we've had and expand that into nodes and you see some real similarities. So that's coming down the pipe. It's actually, they're doing a lot of things now. One of the things that gets does trip up people and I actually we had at the Oklahoma City Elixir Group the other night we were telling horror stories about deployment and somebody mentioned that their team had actually spent days trying to figure out why their Phoenix presence was different between nodes. And these were smart, these were quite honestly some smarter people that I know and it hadn't really even the thought that they weren't actually connecting those nodes in a distributed manner. You do have to actually connect those nodes to each other for Phoenix PubSub to work. Believe it or not. When we talk about clustering, we get into some things that also you have to start thinking about. We get into things like the registry. The registry is a local is local to a node. It's not a distributed registry. Again the Erlang people are actually working on a distributed registry. In the meantime, Jose and James did the local registry. Local registry is also one of those things that I think if you like, if I wanted to give a good talk I should have given a talk about the registry. Sasha's con cash is absolutely wonderful if you're not using it already you can replace redis quite literally almost within minutes with it. Again, it's a local cash that doesn't mean that you can't get to it in a distributed manner but it does not run distributed it is a local cash. Phoenix PubSub is a distributed system where those messages are going to go back and forth and it's going to replicate the data in there. All right it says that I am 46 minutes in so that means that I am only a minute over but I don't have them come in here and tell me to stop. Do I have any questions? So let me just before I finish up let me just say if you take something away from this I want you to take away that you're only losing something with Docker and Heroku if you absolutely need hotcode reloading and distribution and those aren't things to start with those are things to add as you need them. If you're in a company where Docker is your primary way of handling deployment and you've got an elixir problem you've got a problem that comes up and it's you know I can solve this problem I can solve that problem in elixir and deploy it to Heroku without a problem. Raise your hand get it out there just because you don't have those distributed capabilities day one and have hotcode reloading ready and all that on day one is in a problem it's an opportunity until you actually have to have them and you're going to kick me off the stage aren't you? Thank you all. I'm Paul I'm here. Come ask me questions I'd love to talk to you.