 I'm very sorry to keep you for lunch, and I know that we just did all that raffle, so this talk won't be that long, so don't worry, lunch will be there when this is done, so cool. Hi, my name is Chris. You can find me online at Twitter, you can get me on speaker deck, and I'll be putting up all my slides on there after this, and then I occasionally post some code things to GitHub as well, so you can look for that on there. Just before we start, I wanted to plug another conference, which is a bit unusual at a conference, but this is also the other best premier regional conference there is in Elixir, so if you didn't come last year, basically MPEX is a one day single track conference in New York City, which is where I live. I'm one of the organisers. We're looking for people to come and speak. The CFP is open until March 12. Basically, it's just a really, really fun day. We host it in this really awesome jazz club in Soho, and we do a lot of really fun activities, and there's always a lot of good people and good vibes, so this will be our second year, and I'd love to see some of you down there. So tickets are on sale now. You can get everything at MPEX.co. Now that the plugs are over, and that's not a plug joke, honestly, I'm going to be talking about today umbrella apps in Elixir. So I actually gave this talk a few weeks ago at a meet-up, and someone came up to me after, and they were like, you know you really need to rename that talk because it's not really about using your umbrella. So I've kind of put a subtitle in here, which is the what, why and how to use umbrella apps. So I'm going to give you to start with just a really, really brief introduction to applications and umbrella applications in Elixir. And then we're going to move on to talking about some designy kind of things, and hopefully that will pique your interests a bit more than talking about applications. So in Elixir, some of you may know this, but an application is a component that implements some specific functionality, which can be started and stopped as a unit, and which can be reused in other systems. So basically applications in Elixir are pretty much like libraries in most other languages, but one of the other key parts here is that they can be started and stopped as a unit. So any of you who have done some Elixir programming, you're probably very familiar with your MIX file and your dependency declarations like this, where we have, where we can specify some dependencies like Ecto, or Phoenix, or even this other library which is called Spotify EX. And some of these libraries might, sorry, all of these things here are actually applications, and some of them may start up process trees, and some of them may just include some library code as well. So we have this kind of difference where we can start these applications as units like we might do with Ecto, or we can just include some code like we might do with Spotify EX. So the long and the short of it is that applications give us a defined way to configure and run our Elixir code. So if you've worked with one application, you can move on to another application, and it's very easy to kind of know where you are and know like actually how to write these things and actually how to configure and run them as well. And that's great for our systems because it gives us a defined way for our system to kind of bring these things up into life. And we've talked a lot over the last couple of days about OTP and all the wonderful concepts there. And applications that have actually come out of OTP as well. So again, just a very simple example. If you do any kind of mix new with the dash dash sup flag to start up a supervisor, you get something that looks like this, which is just showing that we have this defined way to say that the module MyApp is actually the entry point into our application and can start up all these process trees. So that's kind of what applications are. So an umbrella application and many of you may have heard of them or possibly started to use them in your applications. So an umbrella application is a single repository that inside of it is made up of one or more local applications. So this is really, really handy when we want to actually start composing our applications of sub applications and libraries that we don't want to host in a remote source. So we can always put our code on hex. That's always a possibility or we could always use git modules to do dependencies, but a really nice way to keep everything local and especially when you don't need to publish it externally is to use umbrella applications to do that. So I'm just going to walk you through what that looks like now. So we can run our mix new command and we can give it a dash dash umbrella flag. And what that will do is create a directory structure that looks like this. We have a single configuration point here. We have a mix file at the top level where we can specify application-wide dependencies. And then we have our apps folder, which is where you guessed it, all of our apps live. So when we want to create a sub application here, we CD into the apps folder. We can do mix new inside of there. And mix is smart enough to know the context that it's inside of an umbrella application and it's going to create you an application with everything that references up to the top level umbrella as well. And you can even do that for Phoenix as well. So as of yesterday, that command should be replaced by phx if you're using the 1.3 release. But anyway, I'll get onto that in a bit. So when we create those apps, we get a directory structure that looks like this. We have our apps inside of that directory. We have our web app and our sub app here. And what we can do is actually, if we have tests across all of those applications, we can run this mix test command from the top level and that's going to traverse into all of the child apps and run tests inside of there. And then we get an aggregated view of all of the test results. So it's really handy to develop these umbrella applications. So you don't have to sacrifice many of the tools that you're used to in order to start writing umbrella apps today. And similarly, if we want to start up the entirety of the umbrella application, we can do that from the top level as well. And that's going to start up the applications inside of it. So often I'll build Phoenix apps inside and possibly have a single entry point, like a proxy or something like that. And we can start that service so then we'll be able to access that on a port like we would a normal Phoenix application. And when we want to rely on these applications, when we want to say that one of our apps depends on another app in the umbrella, we can just use this really simple piece of terminology here. We just give it the name of the application. So in this case, we're inside of our web app and we're going to say that the web app depends on the sub app. And all we have to say is give it that in umbrella true and it will know how to wire those things up and run it as a unit. So you may all be thinking like, yeah, I know all of this and I get it, I've used umbrella apps. But really the big idea coming out of umbrella apps here is that we can make these more complex applications that are composed of small isolated applications. And that's kind of what I'm going to be talking to you about today. So now that we've got past the introduction to umbrella apps, we can start talking about designing them. So before we start, I'm just going to give you a bit of a warning that this is all my opinion and this is something that I've done on a project. So there are a million ways to do this. I hope this approach is useful for you and we can definitely discuss trade-offs at the end as well. So let's steal some ideas from domain-driven design. There's a ton of really, really great ideas out there in the community, especially coming around from the .NET community. There's a lot of similarities between some of the concepts that we have in Elixir and some of the other concepts that we're using in the .NET world. So especially if you look at F-sharp, we have similar operators in Elixir. I would even go so far as to say that Link, if any of you have used C-sharp, is fairly similar to Ecto in a lot of ways. And the idea of domain-driven design is very big in that community and some people here might be thinking like, ooh, I don't want to bring in all of that OO baggage and that kind of thing into my applications, but hopefully I can show you today about how that can work for you in a smaller kind of way. So what we're going to be talking about today is really separating your applications into these specific bounded contexts. So some of you may be familiar with this term, some of you may not be. I'm going to literally rip a quote out of the Eric Evans domain-driven design book, which is the Bible for domain-driven design. So there are conceptual boundary where a domain model is applicable. It provides ubiquitous language that is spoken by the team and expressed in a carefully designed software model. So a lot of you may be thinking, huh, what does that mean? How do I apply that? And honestly, a lot of Eric's book is quite dense and reads a lot like this. So if you're really fond of reading something like that, go ahead. But hopefully I can explain this in a simpler way and give it a bit of my take here. So the way I think about bounded contexts is just think about breaking your app into smaller pieces that have clear responsibilities and boundaries around them. So I'm going to hopefully show you some examples and that will become a bit clearer, but keep thinking about these boundaries around our applications. So applying this to umbrella applications, we want each application in our umbrella to focus on a single context. So that means that, again, it has those boundaries around it and it focuses on one part of our bigger problem space that we're tackling with our applications. So let's see an example. So the canonical example in domain-driven design is often an e-commerce app because I'm not that inventive. I'm also going to use that model. So let's say that we have an e-commerce application that processes sales, ships orders, manages customer support requests and manages a product catalogue. So lots of very common functionality for an e-commerce app. So if we're going to approach this into a design where we put everything into bounded contexts and into more of an umbrella application approach where we split everything out, we could start to think of it looking a bit like this. So we might have two applications at the front here which could be our API and one could be our back office or our admin application and they might be Phoenix applications here. And the boundary around the edge here, this represents our repository for our projects. So this, in essence, is the umbrella app. And inside of here are the individual applications that are the individual contexts that we're going to be talking about. So we might have one for sales, for shipping, for customer support and for that product catalogue as well. So the really nice thing here is that every application inside of that, inside of our larger context that we're dealing with can be tested in isolation. We can develop these applications in isolation. We can deploy them independently and we can actually hide the complexity from those other applications through these defined public interfaces. So the idea of modularity in software is often to hide those complex decisions from anything else from the outside world and that encourages an easier way to change and develop your software. So the deployment independently and composing these umbrella applications and deploying them separately I'll touch on at the end. Also Paul did a great job of talking about distillery and hopefully I can elaborate on that a bit more as well. So, yeah, you can kind of think of these applications as microservices. Microservices is obviously the buzzword and Paul's diagram of the emojis perfectly summed it up in my mind. And there's a fantastic blog post from Jose about this where he talks about using Erlang and Elixir to kind of build up these applications and thinking of them as microservices without much of the trade-offs that we have if we're going to be deploying smaller services as well. So what about integrating with Ecto in these applications? So I'm sure everyone here is familiar with Ecto, our persistence library. So our applications aren't that great unless we're actually storing some state to somewhere and actually doing something with it. So just a refresher. Again, our application design looks a bit like this. So remembering that everything in here is an application inside of our umbrella. So one way that we've approached this before is that we've actually introduced a single what we call a DB app that is our entry point into our database and is the point in which we keep all of our migrations as well. So this gives us a really nice place to kind of put the stuff that we didn't want to scatter around all of those other apps and it gives us a single point in which we can start up our Ecto repo and all of the pools of workers that that has as well and also do our configuration of that Ecto repo in one place as well instead of scattering that around the apps. So that might look like this in our diagram where DB again is another app inside of our umbrella and we pretty much we treat that like a regular Ecto repo. So we have an entry point into our repository into our database through using the Ecto repo here and then inside of our DB app we just depend on Ecto and we depend on Postgrex in this example. So what that might look like in terms of your application design is just focusing on the sales and the shipping app here but we have these applications inside of our umbrella that communicate with our database through this DB app. So every time that we want to make an update to the database we're going to be referencing the repo through that DB application and in this example what we've done is we have customer order and line items in both of these apps that are Ecto schemers in both places and you might be thinking oh that's a bit weird but I could hopefully touch on that in a sec so the idea here is that we basically have each application defining its own schemers which are essentially these value objects in DDD terms that we can then pass around but the idea is that we're focusing on the specific functionality for each one of the apps so in this example again we've pulled out the sales and the shipping app and we're looking at a customer schema in both of those applications now but we might have a completely different set of fields that the sales schema maps to in the database than we have with the shipping customer so an example might be that our shipping customer might not need an email address whereas our sales customer might need an email address so because we have this separation between these schemers and our actual database in Ecto we can define these in two places but they're still using the same source under the hood and you might be thinking but we're duplicating our schemers and you're going to be right in that sense but remember that dryers isn't necessarily about duplication of logic dryers about avoiding duplication of logic and not duplication of code so in our example before you might see some duplicate code but the logic is very likely different between those two places and if it's not what we can do is move that into a somewhere shared and kind of share it between those two places instead so when we want to call out to this database we can do so through entities or aggregates or services personally I pretty much call everything a service in this sense remember that could be that could be a set of processes or whatever you want to do in this case we're just encapsulating that inside of these umbrella applications and then we're providing that as a public interface for all our other applications that are relying on this behaviour so an example of that might be in our sales app we have this order manager that can complete an order again it's going to build up this change set and it's going to call out to the DB repo here and remember that this is in our sales app so then basically when we want to depend on that database app we can do that in our domain context as well so in this example again looking at our sales application and we just have to say that this is on the database app and we always have to remember to start that application as well because unless we're using elixir 1.4 in which case we don't have to do that but if we're using an older version we do so this isn't the only option available there's actually some really interesting discussions in the C-sharp community about doing this domain driven design approach and actually having an entry point to your database in each, like they call them solutions in our sensor in our applications so one other way that we could approach this is putting a repo in each one of those apps and then having a configuration in each or we might use a system like CQRS or there's many different ways to do this but we've had great success we're just splitting it out and having a single database app as an entry point here so what about Phoenix in this context I kind of showed you at the beginning how we might have that API and that back office app and those can be kind of our Phoenix applications into this wider amount of business context so remember that and you've probably heard this and everyone I feel like has beaten this point like a drum right now but Phoenix is not your application Phoenix is this web layer into this bigger world of your business logic and your domain logic and because we have this powerful umbrella app idea we can kind of do this very, very elegantly and very easily so let's see an example of that so let's say we're dealing with our API and we're dealing with our checkout controller in our API we might be calling out to our sales manager, sorry our sales order manager here which is actually in that sales context that we were looking at earlier and that will handle all of our business logic for us and everything to do with processing sales and our Phoenix app really concerned itself with all of the things web like processing parameters and dealing with sessions and reading headers and all of that kind of all of the stuff that you know Phoenix is good at and it's and all of those things fit inside of that web concern world so one really cool thing is that Phoenix 1.3 has introduced generators to do this inside of your app hopefully everyone saw the announcement yesterday and Chris McCord's talk at Lone Star Elixir so Phoenix 1.3 takes a slightly different approach in that they are using the idea of context but they're placing them inside of the lib folder instead of inside of separate applications personally I prefer moving these things out into their own applications because we might have a bunch of different processes started there and I also really like this boundary that moving them into their own applications creates in that you actually have this physical kind of separation with your team that they might be working on one app and someone else might be working on another app as well so if we do all of this like what does it give us what's the point of doing this so we end up with this dependency where everything kind of points in one direction which is really nice so what we're really talking about here is a sense of looser coupling between our code so we end up with a flow that might look like this where everything only points and knows about something on the inside of it the sales app doesn't have to know anything about the web or anything about the world outside of it and again the API doesn't really have to know that much except for calling those applications as well so there's a fantastic article by Teff which is called Write Code that is easy to delete not easy to extend and really like loosely coupled code isn't going to just give you like everything from day one but it really really helps in a sense that it gives your team and you the ability to write more maintainable code that hopefully is easier to change over the long term so we're kind of taking that big ball of mud idea, splitting it out and using the power of Elixir to do that and for those who've come from the rails world we used to dabble with these ideas with like cells and those hexagonal rails architecture but all of those concepts were kind of bolt-ons into rails whereas with Elixir at least we get this concept of applications where we can do this on a lower level as well so what you end up with if you follow this kind of approach is again this clear separation of concerns your Phoenix app becomes just this really really thin layer into all of your business context and one really nice thing and Uncle Bob has this article about the screaming architecture where your application screams its intent for those who look at it and actually this is a really nice win in the sense that someone new coming on your team can look at the code and immediately know the different context in which you deal with all in this single monolithic repository and then again you get this really nice simple unit testable business logic that's totally separate from your UI and if you wanted to you could make that separate from your database as well personally I'm not a huge fan of that approach but you know your mileage may vary in that sense and of course this is just one way of doing things not necessarily the right way Elixir is an amazing tool for being able to easily refactor code as well it feels very very easy to pull out applications and move them around and to be honest most of the apps I started with recently have been single monolithic Phoenix apps where there's a lot inside of the lib folder and then we'd basically pull things out into applications when we needed to as well and honestly that's very trivial to do in Elixir as well so don't feel like you have to start with this from day one I just wanted to touch on a few gotchas that I've experienced and tips for working with umbrella applications as well the first one is definitely if you're looking at deploying umbrella applications definitely have a look at distillery it is an incredible tool for being able to do the composition of those umbrella applications into whatever kind of means that you want so we can define basically separate releases where we can say that we have a release for our API and we might have a release for our back office and we can deploy those to totally separate boxes and it will wire up all the dependencies as well for us so if our API depends on only a few bits of our business logic or other applications that we have distillery is smart enough that it's going to build that release with only those things inside of it and then we can deploy that to a separate node or deploy it somewhere else of course taking into account Paul's great talk earlier about trade-offs of doing this in a distributed manner as well there's a ton more on the distillery page here for umbrella projects as well another tip that I found really useful is using xdoc for all of these bounded contexts as well so just actually writing documentation and then generating that documentation for all of these applications and having that as something that you can share with your team as well and it's been hugely useful for us and again writing specs as well even if you don't use Dialyzer just writing specs so you can show what return values are there I just find that hugely useful as well again you don't have to do this on day one refactoring is so simple and honestly I've done this I think about three times now I should probably just start with an umbrella app or you can also you can cut this a lot of different ways you might think okay I'll just have my web app and then I can have a single bounded context with everything in it if your business logic isn't that great that you need all of these separate contexts as well so it's another way to do things one big gotcha is that circular app dependencies are not allowed inside of your umbrella applications this kind of bit me at first when I got really really excited about doing umbrella apps and started splitting everything out into a separate application it got way too granular and then basically I think that's a definite smell and that you have all of these things depending on each other and it's far far better to think about a context in a bigger sense another gotcha is that you can't run gen migration from the top level honestly like you just have to cd into your apps and then into if you have a single app for your database you just cd into that app and you generate all your migrations from there it's not too much of a big deal and this one has bitten me a couple of times where you might have lots of dependencies inside of your individual umbrella applications they all have the same kind of transient dependencies or different transient dependencies and it gets quite difficult to work with at times and often you have to use override flags as well to kind of get around the situation it's again it's one of those interesting points that because it bundles up all of the dependencies into the top level basically everything has to match inside of your umbrella apps so we can talk about that more after if you have any questions there so thanks and we'll be going to lunch now but I'm really really happy to take some questions before we do or if everyone just wants to run out feel free to do that as well so thank you