 You know, the past couple years in the mixer a lot of my open source development time has been focused on XRM, which is a release management tool that many of you may have used or at least heard of. And this year in particular I spent a lot of time more on the demo offside things as it. So I got a really good chance to push XRM and releases in general and find places that were paying points to myself. But then also think about what kinds of things I really cared about. Basically the first one is very hard, right? Every environment is very different. We all have different requirements for what we need to our production systems and how we do things like monitoring, logging and so on. So for a tool like XRM it's extremely difficult to go this one thing that just works perfect for everyone. And XRM tried to do that to a great deal. So the idea is that tools like XRM are really supposed to help with this kind of thing, right? Like your language based tools are supposed to make you like these here, not add more complication, not cause you problems that you wouldn't otherwise expect to encounter. And XRM to a certain degree helped quite a deal with this, but there were definitely times where it caused more problems for people than if they had not used it to begin with. This is basically how I do deployment today at Electric. It's just chaos. There's not a whole lot of good guidance or documentation. We had no consensus on what the one true way of deploying an Electric application is. There's really two primary paths, right? Releases or running with just mix. So you do mix, run, dash, edge, no halt, and your application is not running. But everything else around deploying your application is kind of up to you. So it's more or less figure it out as you go, have a scenario, and for a lot of people they have their opinions on how they want to run their applications and that's fine. But when you look at other communities, like at work I use Erlang, Elixir, Domino-Lang, but we also have Go applications and a few other ones like No and the stuff. But the Go applications is one of the really sweet communities because you do a build, you have a single binary, you drop that on whatever box you want to run it on and go and you're good to go. To me that's how deployment should work. It should be a self-contained package that just had a super easy to deploy, easy to reason about. You don't have any sticky dependencies. Like for me the nastiest of the applications that we have to deal with is based on No because you basically have to pull in your entire function, close the machine with dependencies. Unless you're running containers. So to focus my efforts on what I want to do next with deployment tooling and Elixir, I really wanted to find what good deployments really mean or what good deployment infrastructure is. So that's highly automated, right? Like you want to be pretty much hands off the most bit. And this is sort of boring to your own continuous deployment, right? So you have a highly automated environment, your builds are reproducible, right? So you have an artifact that's produced by this deployment pipeline, and you want that to live in depth now so that if you need to roll back to another version, it's simply just dropping that package on the machine that you are running and starting it up and you're good to go. You don't want to have to rebuild and potentially get like different versions of your dependencies or you want it to be this image almost, right? You want your deployments to require minimal or no downtime. No downtime is a bit of a tricky subject, but there's ways where you can implement that even when you don't have like hot upgrades in place, right? Things like EV style appointments with intelligent routing. So you have the appearance of no downtime, you know, you may actually have a little bit in there. But the idea there is that when you're doing automated deployments, you press a button to do a build, it generates the artifact, rolls in the production, and your customer shouldn't see any downtime. The other thing is, you need to have tooling for doing easy troubleshooting of your deployed artifact. So if you've got an application production that's, you know, flapping up and down and you're not sure what's going on, you want to have all the tooling in place to easily jump onto that machine, try to do things on a once in a while since he was going on. And you also want to be able to roll back, in the case of this issue, just very easily without too much stickiness there when you have to roll back. Versions of dependencies like, especially if you've got systems left, so you depend on. And basically to achieve these technical goals, and it looks like we need to use releases. This is true of early as well, but releases are, either contribute to each of these things in some degree or another, or are a requirement for it. For instance, easy troubleshooting you can do with fixed run, but other options releases you get a remote console, you get some of those introspection tools that are easy to use right out of the box. No downtime via hot-off frequencies, you decide to use those. The build architecture releases are the tarpaul that lives indefinitely, it includes the early runtime, so you can jump back to a release two years ago and fire it up and it should just work. Obviously if your infrastructure is changing, you might not, but it contributes a great deal to this. And with the release tooling that currently exists, it's very odd. How you should deal when you're using releases. So, again, releases are self-contained, but tarpaul contains everything that's needed to run it. So it's, in a lot of ways, very similar to Go-Bill where you get a binary that you can drop on a machine. It's not quite as nice as just that self-contained binary, but it is a package that you can drop on a machine with double external dependencies. It's useful and consistent too. Like when you have that release tarpaul, other than potentially external dependencies on native libraries deploying to different versions of Linux or a EC2 instance versus an embedded device, the deployment process is basically the same. Drop that tarpaul on that machine, extract it, and run your app. You don't have to worry about, hey, while I need to install Berlin and Elixir and all the right Berlin packages, I need to make sure that I can build my app and fetch all my dependencies, and that's not consistent just by definition. Where with a release, you don't have to worry about any of that. Particularly with embedded devices, that's where you know the biggest difference in my opinion. You know, having me build Berlin or Elixir on Raspberry Pi, if I ever done that, is just all. And again, like, release is more purpose-built for this. This is why they exist in the first place. Actually, I just tried to do some thinking to find out what the original design was behind the SysTool and release handler modules and how to do it. I still haven't been able to find out. I went back all the way to like R4 or something like that in the GitHub repository to try and figure out how it evolves, but that's been lost to history as far as I can tell so far. But the idea is that from day one when they're like, they knew that deployment was a very, very important topic and releases were their source of that. And they've lived this day because they are so flexible. Unfortunately, with releases, the tooling is awful. And, you know, I wrote one of them and even I couldn't admit that. And the reason why it's awful is not because it doesn't work or anything like that. It's actually extremely easy to use initially. But there's a lot of sticky corners where either documentation is lagging or the logging that you get from the tools is not particularly informative or lets you do things that are broken upfront but it doesn't tell you until you try and run your release. There's just a lot of lagging areas that even after like three years I still haven't solved properly in XRM. RELX is the other primary release generation tool and XRM is actually built on top of RELX. But one of the reasons why XRM has issues with informing error messages and such is because I don't actually have any control for how RELX does things. So I have to handle RELX error messages. So in general, the tooling has suffered from poor user experience. It's not tailored towards people who have zero no understanding releases. There's no real fear documentation on releases for new users either early or later. You basically have to dive into the early manual to try and learn those things. Which unfortunately if you've ever spent any time doing it is very descriptive but not at all suitable for tutorial type introduction. So this summer basically actually I submitted the premise of this talk before I wrote this application. I actually actually talked with Jose this past year a couple of times about how to roll with these handling more properly in the mix and distillery is what came out of those ideas as conversations. And it kind of turned out that I don't feel that it belongs in the mix anymore for a variety of reasons that I may get into a little bit later. But the point is that it lives as a separate library now. The benefits are though that it integrates very badly with the mix. It's written entirely in the mixer and it's much smaller and more flexible than XRM could be by definition just because it uses the information that mix contains which I couldn't previously use when we were wrapping the Linux. Its emphasis is on a small or similar plug where it focuses specifically on generating a release package and handling configuration around the packaging but then leaves all the kind of extension points up to you. So if you need to add something like, I don't know how many people here use this form with XRM but previously I had actually hacked and stuff specifically forked form into the script that XRM generated. The start script for XRM, if you haven't used it before is just a shelf script that sets up the code as and then posts up. So I had to download a bunch of form specific code into that shelf script because I didn't have any other way to make the form work the way it does. Now with Distillery I can actually make that as a plugin. A little shelf script that gets executed before a release starts and the same kind of thing can be done by any more of you if you're using Distillery. It also has an emphasis on being as configurable as possible. So XRM was very opinionated and actually flexible in some ways. I don't really want to dive too deep into what all those places were but needless to say there were definitely use cases where XRM would cause you problems. XRM did have a plugin system. Distillery is actually richer because not only does it have XRM's plugin system but you can add custom shell commands and then there's the restart and stop folks that I was talking about that I used to conform. The other thing that was a big focus is Distillery was kind of XRM. Umbrella projects were poorly, poorly supported. They basically didn't work and they did it but you had to build a release for each application on your Umbrella and maybe that's what you wanted but maybe not. You could build either separate releases for a given application or some combination so you could build a single release with every application here on Umbrella or maybe you would have two releases with two applications from Umbrella and one application to have a release point being that you can structure your releases however you need to for your environment, your needs. The other emphasis was on fixing the user experience around the tools so conform repairs only show information that is important or that you really care about and there might be still be cases where I can turn some of that down but I really wanted to make error messages in particular a lot more informative and warnings a lot more. Early version of Distillery I actually automatically wired up dependencies into the application tools that was a super common issue with XRM those people would forget to put applications from their dependencies and their applications list they start the release and the application would start it fails saying that some application its code was missing. That part of it was actually fixed so code for any dependencies you have will automatically be added to the release the only thing that you still are responsible for is putting things in the applications list to find how those applications are started so whether its automatically started or whether you want the code loading type to just reload so maybe you have dependencies that you want to manually start but you don't want to make it an included application there's a lot more flexibility there and for many of you that may not at all be relevant but for the few of you that ran into that with XRM quite often that's been addressed from the ground up unfortunately there's still quite a bit of work to be done with Distillery I mean its what I would recommend you to use from this point forward do not use XRM use Distillery because my efforts are focused on Distillery and at this point its at XRM if not past what is lacking today is Windows support but it runs in different configurations so if you haven't used releases there is with the regular next application you have big.exe when you're using releases that is actually compiled into a static and big file called sys.config there's also vm.r which tells you the early vm how to start up really high level bits of information like the code loading mode and the code paths and a bunch of other stuff the point is that right now all that is different and there's not a really intuitive understanding of how your config.exe gets converted into sys.config even with Distillery I had a couple issues already with people using function calls and config.exe and I'm wondering whether config is broke in the release so the tooling around that I want to fix and I also want to make it so that the vm.r and this is the second big are integrated a little bit more tightly one of the tools I wrote early on in conform was an attempt at handling this to a degree and it solves a config file that maps a little bit more cleanly to way users manage configuration that so it's in and in itself config I want something like that for releases except where I want it to be more like config.exe a single place where you manage your configuration for runtime that maps very cleanly to the way you think about config.exe without as much of that clean break where things don't go the way you're expected to I also want to focus on preventing errors so what I mean by that is instead of building your release trying to run it and finding out oh it's broken, I forgot to do something as soon as the release builder knows like there is an issue here that should stop the world and say here's exactly what's wrong here's how to fix it basically make it so that I save your time and sort of waste it actually I was not very good about that it couldn't warn you in all places because a lot of it blinked and relapsed and I didn't have control over it now I do with Distillery because Distillery rewrites all that and we're all about it the other one is better handling the dependencies like 200 degrees this is beyond scope of a tool like Distillery or XRM this gets deeper into one of the things I really want to solve is documentation and practical experience like documenting how to do things with releases the other one is releases to a cluster like right now I haven't even bothered to document an XRM how you push a release to a cluster nodes and upgrade them all simultaneously but that's the thing that you can do there's no reason I can't but without any documentation around it either nobody's going to do it or those that do try are going to have trouble with it and they'll do it this is kind of the caveat there we need to make releases easy to understand not just the conceptual like color structure and stuff but how you pull it up right I was trying to think of a way to describe this and what I came up with with it here is that's about making it easier or it's easier with releases there's so many different bits of information that haven't been well documented and there's not a lot of good guides or anything like that out there I want to make releases so easy that you can't get them wrong that it's just build your release drop it on the box you want it on and it just works that's the goal with there was a goal with XRM too but there were limitations there and my goal with Distillery and important in the future is to do exactly that so part of that is from documentation like guides, tutorials documenting practical experience like real world situations that other people can borrow knowledge from to give an example to be like okay I'm moving from manual and managed infrastructure to something running under Kubernetes like how do I set that up and use releases in that environment there's kind of a gradient between what is a tutorial, what's a guide and what's practical experience but I think that a lot of that needs to be documented and then centralized make it easy for people to find that information right now you're basically hoping that when you do a Google search you'll find something and when you do find something it's not horribly outdated with Distillery I've put a lot more effort into collecting that kind of information and putting effort into guides I've actually gotten quite good help already playing together practical guides working out PR or someone right now to get one specific with Phoenix like how do I generate a new release of Phoenix application how do I make some changes and do hot upgrades and see those things change that one's actually about to be published probably in the next day or two but those kinds of things are very much close to Distillery and right now we're going to be living under the Distillery Headstocks page but in the future I'd really like to make that more of like an Elixirlang organization documentation you know it should be on the Elixirlang website under the tutorials you know how do I release how do I deploy an application and I really think that kind of guidance should be something that people think about from day one because it's the kind of thing that they're going to have to answer for when they want to to invest in class 11 users at work like okay we already have deployment infrastructure how are we going to fit Elixir into that and by making all these things available guide tutorials, practical experiences people will be able to sell that a lot easier and the other one is kind of I've been touching on this continuously here but improving the tools so Distillery is already much better than Elixirlang but there's still a lot more to be done one of the things I think I've more or less solved with Distillery is this idea of intuitive defaults right one of the capabilities that was present with releases is this idea of building release with the theme files and file artifacts are similar into the release allowing you to make code changes and rebuild the release super fast without having to change launch allowing you to kind of emulate your production environment but iterate quickly on fixes with the concept of environments right so out of the box when you generate the release config file you get two environments dead and rotten and when you just build a release with mixnv equals dev it runs with that dev on your mind so release builds are super fast some of those intuitive defaults are not part of that default config that's generated and I want to continue to tweak those until it gets to the point where when you build a release it behaves the same way as when you're doing your development work with Mixron the big one too is Mixron heavily expected that you were going to provide hit direction like it trusted that if you set the big options and knew what you were doing and that if things broke it was your fault and I would really convince that that's the wrong way to do tooling and that instead tools like Mixron and my installer should be telling you hey you probably shouldn't do this for this reason you should try this instead or hey you did this wrong it's going to cause you problems please fix this and that ties into the Mixron form of warnings and areas similar to the way Elm compiler works when you do something wrong it explains very clearly what you did wrong and why it's wrong and how to fix it and I think disability can do that for the classic problems like people most commonly and I've got a huge list of those from Mixron because the issues with stairs are 300 and so on and a key part of that is preventing invalid configurations to a lot of the errors that were generated with Mixron were completely due to invalid configuration that Mixron had bothered checking or warning about distiller does better job at that but there's still room for improvement another thing that came to me too you know at XRM I'm very much trying to do as much as possible for you and I've kind of arrived at the belief that we should be building on each other's tools as much as possible because the ecosystem benefits greatly from that you know distillery builds on assist tools, release handler Mix and a couple of the tools that I know of that build on other schools like this eDeliver is a great example built on XRM originally but now also offers distilleries support and I just read about one I think a couple days ago Gatlin and Hank by Ashwag but also this is a similar thing only with Mixron currently as far as I know point being you know tools like these build on each other and improve on areas that those tools don't focus on like distilleried builds and tooling around generating a release in a way that makes it easy for developers to get started, assist tools are very low level API Mix doesn't focus on releases at all it contains a lot of very valuable information for generating releases likewise eDeliver and Malaya distillery to generate a release package but they can handle automating the deployment of that package which is basically its purpose for building that we should be building tools like that so you know distillery being like a foundational layer and eDeliver automating some things on top of that but there's some horizontal room for building as well particularly around distillery you know with XRM there was plugins that did things like generate RMS you could generate a document from a release package as well so one of the things I wanted to do as part of this in the interest of giving people some hands-on experience or my hand-on experience with working with releases of production and employing works throughout the patients it may very well widely vary from how you do it yourself or maybe you haven't avoided any production yet but hopefully it helps and I want to see people doing more of this sharing their experiences documenting it eventually we'll get to a point where these problems are easily answered so my employer is an IOT platform as a service company we're working primarily with industrial embedded devices but consumer ones as well so we handle connectivity from the device to our platform within the platform there's a bunch of microservices that handle things like SMS scripting, time series storage utility storage the core of the platform is Erlang Elixir but there's microservices written like I was saying earlier Go, Node and some other ones our original architecture was all Erlang on manually managed infrastructure we were doing hot upgrades but all those were handwritten so our new platform is hosted under OpenShift Origin which is an open source product that has a virtual variant of it built by RedNet and basically it's a wrapper for Kubernetes and Docker that adds a bunch of nice stuff onto it so until they're running later with HGproxy it has a bunch of templates for doing things like Elasticsearch LogSash so we're leveraging OpenShift to containerize the episode with Kubernetes being our scheduling layer the way our continuous integration pipeline works is that we do pushes to give out those trigger webbooks which notify OpenShift that hey there's new code that needs to be built it clones every cluster it does a Docker build a Docker file that exists in there and those Docker files generate a release parable strip out everything that's not related to the release itself and then pushes that image which is very small like 30 something bytes to damage repository and then Kubernetes schedules out the replacement of existing pods with you want some pods like Kubernetes terminology for a given container the way that we started to do things more recently is writing in some line list checks and one of the things that releases a lot of to do and particularly to accelerate is extend the start script with things that we can do to probe the state of the system so when Kubernetes starts to roll out new versions of the application that's being upgraded it will run this probe so that we can see okay our database services are up our application points are ready okay thumbs up go ahead and roll out the rest of the instances and we'll do this until the deployment is complete based on the rolling upgrade previously we were just letting it do a replacement of things and if it failed it would roll back now the readiness and line list checks connection make sure that we're ready to go and actually then before we start rolling traffic onto it OpenShift actually handles also ABE deployments so using label selectors and Kubernetes we can say alright leave this group of pods undeployed and then we can flip the switch once we're ready and we'll place all like send all the traffic to the new one this has worked really well so far in practice setting up OpenShift it was actually a huge effort it was why I basically got more involved in the DevOps side this year was because I pushed for Kubernetes and then discovered OpenShift and pushed us towards OpenShift and then it was my problem to figure out how we were going to get that in production and it was kind of a nightmare we're on Amazon so there's an Amazon Playbook setting up OpenShift and they're like oh yeah that should work fine and then you find out that every time you run this Playbook you get a different result which is really awesome so yeah we had a lot of lessons to be learned along the way there there are small details like purchase logging standard out and using fluent DU full logs on containers and pipe them into the DevOps side which is a license or a DevOps side and that's worked really well for us one of the lessons that we did learn now is that if you are not careful of making sure you've got a tracking ID through your logs you're going to have a bad time that was something that moving from a very monolithic architecture to a much broader center microservices started to be a problem real fast some of that I'm sure many of you already know but it was my first time seeing the side effects of forelogging when you're not responsible for lists or something like that the other big practical experience that I kind of wanted to touch on a little bit was clustering so Chris talked about presence and distributed Phoenix to some degree and Eric over in the PDG talk also discussed that to some degree my practical experience with it is that it was a very tricky problem harder than you'd expect because early on it was so easy to connect things and one makes a distributive way that by the time you start actually thinking about the guarantees that you need by how you're doing designer system you're already in production and we ran into that so one of the things one of the key components of our system is this dispatcher process which is to the X to the Y but it holds a process for each connection that lives through the message ordering for that connection what we've been doing there is clustering a bunch of nodes together and sending messages to which I would develop that connection process I initially tried to roll that with global, completely development data with GPROC which worked well with it but then when the process registry grows to like 100,000 connections GPROC starts to have serious problems synchronizing the registry with nodes coming up and down during rolling upgrades so then I ended up having to implement my own process registry distributive process registry which we're using now and it works fine but if you have to do that yourself it's kind of a nasty problem to try and drive it better and I guess my advice from that is definitely avoid any distributed Erlang unless you have a good reason to do it if you can just solve your problem with low balancing you don't need to rely on Erlang's distribution to do that for you I'm sure there's probably people that won't agree with me on that but that is my theory on it last round they said Bell Lake Chris kind of touched on this a little bit but leases and containers I have the opposite in hand Chris which is that I do think leases are a good fit for containers except when you're doing cloud upgrades if you're doing cloud upgrades you should not be using Docker with leases it's not that much well because there won't be stateless immutable images and cloud upgrades by definition are immutable so if you're changing your containers that way as soon as that container goes down you're restarted to be roaming over so that was not a problem that I ran because I kind of already moved that but some people might start running leases in production and expect that they can do a lot of upgrades and wonder why things are not working properly the reason why cloud releases are a good fit for containers is because similar to the go build thing once you have that binary your actual images can be extremely small on over 20-30 megabytes and you can trim it down even more if you know exactly what packages you need our containers are built on alfamilionics that are like hardly even in the dependencies other than the ones that are absolutely required and when you have really small images sizes like that it's super easy to run a ton of them so that was my experience with running the leases under Docker I think that in the future that's going to be very much a more common scenario you know one of the problems of thinking in terms of like ok well we've got Erlang Elixir we don't really need to use Docker because we can do cloud upgrades and stuff like that is that all the other infrastructure that you may rely on could end up focusing more on containers as a way of being extremist so if you want to run something like Hava maybe it's easier for them to deploy that as a container image or a bunch of packages that you don't have to install on a post and manually manage I do think that there's going to be increased movement towards containerized infrastructure in the future and that we should embrace that as much as possible rather than protecting it because in the one scenario it doesn't work particularly well and the other checkout there is the product grades because they're super easy to start using and then they get very difficult very fast namely you have to write Apple files by hand when you run it in this scenario where you have to do something tricky during an upgrade that's what we did on our old version of the platform is all of our Apple files were handwritten like we weren't even externally necessary to both generate Apple files for you as best they can but in the case of our old infrastructures we weren't even doing that like we were just handwriting the Apple files and it's a huge upgrade it almost takes more time to do that than to like roll it out in the production and see how it breaks, fix the problem roll it back out again and that's one of the problems I've once solved with Xgeron this is okay but I'm kind of up to believe that hot upgrades are more trouble than they're worth for the most part I would say that it shouldn't be an area that we focus on particularly much other than in certain edge cases one of the places that we do use hot upgrades actually is our connectivity layer because we can do upgrades without dropping connections so that's the one place where I feel like it makes some sense but if you've got retry mechanism and devices that are making connections you may not even need to do that because they'll just retry as soon as the connections spread out that was a whole bunch of rather unorganized thoughts but I was trying to think about how best to describe some of the practical experiences I've had this year and this is kind of what sugar has been supposed to so I learned from last year that people have a ton of questions so I want to make sure that I operate as much time as possible to ask questions particularly about distillery but any questions you may have about the point with the releases in general or even about Xgeron I've been playing around with distillery I've been playing with distillery in front of all the Xgeron at work and I've bumped into a lot of the stuff you've talked about one thing I don't have a good answer for yet is is there a great is there a recommended approach to building releases and going out and fetching sensitive things later like an API or something like that so that it's not baked into the release how do you put it against it? Yeah, that was actually something I wanted to touch on and I forgot about the time configuration so we're almost kind of cheating with our social work because it's containerized Kubernetes has two concepts for configuration modes environments and everyone's secrets were impossible with the data that is secreted we're using both varying degrees primarily we're using environment variables because that's actually really well with assist config and both Xgeron and distillery's capability to swap out config variables with environment the contents of their environment variable so there's a replace OS bars environment variable that the start script will look for and if that is set it will look for a dollar sign in the environment variable name and it will look for some big and BNRs and do a string replacement on that that's what we're using for most of our configuration there's a few things that we're using that are secrets for both are perfectly fine approaches to it I would say that environment variables are the easiest way to do it another approach might be some sort of configuration management I'm afraid we're out of time please keep the poll a big hand