 I'm Luke Imhoff, as you may remember from the lightning talk yesterday. I'm known everywhere as chronic death. I'm the maintainer of the IntelliJ Elixir plugin, and I also run the Austin Elixir Meetup here in town. But my talk today is about my work at communication service for the deaf. In September 2015, CSD started a new Phoenix 1.0.2 project. It contained a single OTP application, interpreter server with lib, test, and web directories. Between then and Elixir 2016 in Orlando, about a year later, I joined CSD, and the rest of the team and I slowly built up that Phoenix project, adding support for talking to other services, partner server, and lighthouse server using RabbitMQ and talking to an Ember front-end using JSON API. Lighthouse is for authentication. It stores our users and issues JWT web tokens that all the other front-ends can use to talk to other services. Partner server, on the other hand, is targeted at our partners or the interpreting agencies themselves. It's where they can set up jobs for interpreters to accept and to file the client request and business request for those jobs. Interpreter server is started to the actual ASL interpreters, where they can accept jobs and set up their certifications, because that's a big issue for us is that we have certified ASL interpreters and not just anyone off the street that thinks they can sign. Together, this is Vinnia. At ElixirConf 2016, Chris McCord gave his keynote on Phoenix 1.3. Now, remember, this is based on what he said then, not what he's just told us. Where he mentioned the web directory would disappear, and then, optionally, umbrella projects could be used to separate the web interface from repository access or other business logic. I was excited, but the templates for Phoenix 1.3 wasn't even a PR yet. I actually was looking at the GitHub repo as Chris was talking, looking to figure out how to do this. So I was stumped. I had never used umbrella projects before. I had heard people talk about podcasts that they always use umbrella projects. They always use OTP apps, but I didn't know what this secret sauce was. It just seemed to be like the way I should be doing it. Thankfully, I was in luck. Vojtek Mach was giving a presentation on umbrella projects. I recommend watching it if you haven't seen it. It's a great talk, but his talk is from umbrella from the beginning. Chris and Vojtek's talks left me with the question of how do I get to that great umbrella project from where I started with a single monolithic application. From Vojtek's presentation, I learned that umbrella projects still had a top level mix.exe, but all the code lives in this apps directory. Each directory under apps had a separate OTP application, which is just what the root of the directory of a normal project was. So it just is move the root down to apps and make more projects under apps. The first step was to put your project root under apps. This will involve a bunch of get moves to preserve history. You'll likely make some mistakes. I know I did. Phoenix 1.3 will use the web suffix for this OTP app. So we'll do the same here. You'll need to do the move first because your root's going to have a new mix.exe. In other words, you'll get file name conflicts. After moving your mix.exe from the root of the project to the apps and my app web directory, some of the paths need to be written to point back. In umbrella products, the build path, config path, and depth path are all shared. It's one of the main things that separates OTP apps and umbrella from just using path-based dependencies. If you use mix.new inside the apps directory, these path rewrites happen automatically. Which wasn't, but our mix.exe that you're going to do with get move isn't set up that way, so you have to do all these changes manually. The way I actually figured it out was just set up a fake umbrella app and copy the changes in. I recommend the same thing if you can't remember these changes from the presentation. The new root needs to be a mixed project, but unlike a full OTP app, the mix.exe project function will only have four options, three of the options. Build embedded, depths, and start permanent are the same as any other project. But for the root umbrella project, we have a top level config apps path that defaults to apps. So just like a loss of an elixir, there is no magic. It is explicit. Just the convention is to call it apps. Eventually, you may want to fill depths in to fill in things that need to be at the root. Examples would be credo and dialys or dilixir because if you don't put them in the root, they won't work from the root directory, only from individual apps directories. Although each OTP app has its own config directory with its own config files, the overall config for the project is unified. Top level config directory says go grab all the other configs. And then each individual config will point up here and say loop back. So from any directory, you always get the same set of config files. Since you're going to use log in all your OTP apps, you can eliminate diversion configurations, which was a problem we had. By actually pointing this, your logger config just in the top level area. I recommend turning on handle OTP reports and handle SAS reports so they all go through the same logger. And you don't lose SAS reports that only would be in the Erlang logger otherwise. After all the moves, renames, and replacements, you're left with an umbrella project with a single OTP app in the app directory. It's more complicated, not less than what you started with. And you have to wonder what was the point. Well, umbrellas only start to pay off when you start to break up that single app. But the question is, where's the first crack to start chattering the web application? From Chris's keynote on Phoenix 1.3, I knew that dash dash umbrella option for Phoenix New was going to replace the Ecto module, both the repo and the schema and OTP app separate from Phoenix. Pointing the Ecto repo and schema in their own OTP app isn't enough if you just treat it like a separate namespace. We want to be able to test and use the domain logic without the need for Phoenix controllers. So this is the bounded context that Chris was talking about. Why? Well, you may not think about it often, but any Phoenix project already has two UIs. The Phoenix API that we present to the web, but also the UI we use as developers, DevOps, or maintainers from IEX. So if you've ever had the problem of debugging or having a setup code and it's just a hassle to do it in IEX, that's because that UI is bad. After a lot of refactoring cycles, like three or four, I eventually came up with a transport neutral behavior that can hide whether we're getting data from Ecto, Rabbit, or even local gen servers that backports to use SSH port forward tunnels that we use for debugging. And I called it Calcinator Resources. This is only my supposition of what a domain module could look like in Phoenix 1.3. It doesn't use the function prefixing system that Chris showed, instead that I can have callbacks and OTP behaviors to actually check at compilation time. The behavior supports controller action-like callbacks, but also support for testing with sandboxing. Certain callbacks, like change set, insert, and update, have two forms to allow the optimization when calling in the controller-like Calcinator module. Query options encode common options, such as pagination sorting and associations including the response. We built Calcinator to target JSON API, but I wanted this to work for anyone that's not just using JSON API, so query options are more ectocentric, where they have params and associations instead of relationships that JSON API would use. You may find this list useful. You may like some of the things, some not. If you like it completely, you can just use the package on hex. Since I want Calcinator to work with Ecto, RabbitMQ, and really any backing data store, the returns are more complicated than Ecto. As I want to be able to handle more error conditions, we'll have to know each data store's exceptions. So usually when you get an exception all the way down Postgrex, it's a Postgrex error. I didn't want to have to know that because I'm not supposed to know that Postgrex is backing me. Error ownership is good for any ownership errors, which can happen anytime you're interacting with RPC or when you're doing the Ecto ownership stuff for testing. And I want it to be an error, not exception, so that it can be surfaced as an API error during tests. So in con cases, in Phoenix, it actually shows up as a bad return instead of them appearing in the logger output, which some developer has to notice the logger extra noisy during testing. Likewise, error timeout allows for gen server time must be shown in an assert response value instead of appearing in the sassel log. I found doing okay struct or error not found to be easier to match in a width than using okay struct or nil. So I recommend that for all get like calls. It turns out that either monad is really useful. So for a general platonic Phoenix project, that's as far as I can advise how to break up your project. A domain OTP app and the Phoenix web OTP app. But let me cover the specifics of CSC's own project to give you some more ideas for how to break up your project when converting to an umbrella project. And Serpenters server, as I said before, is CSC's project for allowing sign language interpreters to find and track jobs for multiple agencies. On the front end, it uses Ember to talk to Phoenix controllers that respond with JSON API. On the back end, it uses Ecto to talk to a database it owns to represent the data owned by the interpreters. So their profile, their address, and their credentials. But it also has to talk over RabbitMQ to background processes running RPC servers that can access databases owned by two Ruby on Rails server owned databases. I don't know if this is the case for you, but a lot of places seem to have that mix. We got lucky, we got one app with Elixir, we didn't get them all. For debugging purposes, we can access the back end using SSH tunnels held in memory that allow either IX or observer remote connections. The Ember front end uses Ember CLI. So to keep the publishing consistent with the Ruby on Rails servers, the Ember front end is published from a Redis cache. So the cracks in interpreter server, our Phoenix OTP app became obvious when I listed like this. Now, mind you, this is somewhat of a generalization. It took a while to figure out this list was the obvious way to break it up. Unfortunately, there's no way to go through the struggles and still make this a good talk. Each UI should get its own OTP app, but so should each backing star. So let's see how I actually did that. When shattering your web OTP app into more pieces, you may end up with these four separations because some code is needed in two OTP apps, but they no longer share a common dependency because you don't have the monolith. This was the case with us with interpreter server, JSON API, because they have shared views that are both used to talk over a random queue for RPC and in the Phoenix web. Observer already has its own UI, but the steps necessary to connect it over an SSH port to forward to a containerized host also needs to be a good user experience. So interpreter server observer contains an interactive walkthrough that walks any developer through the commands to copy and paste through two terminals simultaneously. Pretty much you put in a value and then it spits out the next command to throw in the remote terminal and goes back and forth. It's almost sort of like a text-based adventure. And with this, we're able to set up a remote console to our hosting or get observer to connect to production or QA containers instead of one set up by the remote console. So this is very much like a Roku setup where when you ask for a console, you actually get a new container. But since we have RabbitMQ, we can actually shut down the RPC servers on the console container. We just spun up, send a message and say, hey, open an SSH tunnel and production will respond and come back to my laptop and I can run observer then on container infrastructure that's not supposed to support SSH. This may work on a Roku, but we don't use Roku. We use a provider that use Heroku-ish, which is a Python implementation that kind of makes it work like a Roku. I haven't tried it myself. InterpreRPC talks to RabbitMQ, so it owns the connection to RabbitMQ because you're supposed to pool it and have one connection in multiple channels. And it also supervises all the RPC servers that expose the database owned by interpreter server to the Ruby on Rails applications and it also has apps so that we can have RPC clients to talk to the Ruby on Rails applications. With all this code being moved out of the normal Phoenix app, the Phoenix app, InterpreRPC server web is down to just controllers and views that the Ember front end needs to consume. The controllers are minimal because the calcinator package that I made makes it so that it's mostly declarative with use statements, saying which actions we want to use to support JSON API. And there's a couple plugs that help with authentication and authorization, such as needed foreign keys. And interpreter server web does have a few views that aren't in JSON API resources because there are parts of authorization that are mixed into the view layer to hide role-based fields that don't apply in the inter-back-end communication for RPC which kind of sends all the data at once back and forth. During the shattering of interpreter server web, authorization was one of the hardest aspects to disentangle into its own application. I was never able to make it completely its own layer because there's no good way with an ectoschema struct to indicate that a field was censored without making up some random value for that field. And so we end up having some of the authorization for the entire structs to be its own layer but some of it to also be stuck in Joceralizer views where we override the attribute callbacks to hide some fields. Our imbo app handles fetching, setting and invalidating the Redis cache for Ember CLI. Interpreter is for data owned by interpreter server and access using Ectorepo connected to Postgres. Lighthouse and partner are two rubies and Rails applications but we actually mirror them in Elixir to represent their ectoschemas that we need to deserialize the data over RabbitMQ when we get it from those services. But additionally to be sneaky, during integration tests Scott actually had a thing to have Ectorepos clear the database because it's actually faster for us to clear Rails database for it than to ask Rick to do it. As you can see there are multiple Postgres apps interpreter, Lighthouse and partner. This is because I would not recommend having the store format decide how to group your code and OTP apps. Instead, I'd share the OTP apps based on the owner of the data or its use case. So partner agency focused, interpreter's interpreter focused and Lighthouse is off focused. So if there were a new use case for Redis, I would make a separate OTP app from Ember. If some of the coding Ember turned out to be useful to both, I'd make a second OTP app that represents the library of the common shared bits of Redis. The three Postgres backed OTP apps reflect the distinct databases that they're backed by and in theory we could start using that to scale or deploy only parts of our umbrella application as releases to new containers. SSH Tunnel is probably the most classic OTP app. It is a supervision tree of gen servers tracking PIS to external OS processes. It also includes an interface for sign up SSH keys so sign all the things in your home directory dot SSH but in the hosting environment. If you're wondering why that's necessary when SSH is a library in the Erlang Serum Library it's because the SSH library in the Erlang Serum Library was accepted because someone wanted to send in a patch to give it but it was in the days before they really tested all those contributions and made sure they were maintainable. So it's not really all the SSH protocol, it's definitely not the part that does tunneling and until a recent release of 19, they didn't even do off mechanisms that most SSH servers would accept. So it was just easier to actually use a physical SSH client binary that was inside the containers from our hosting to dial back to our machines to use observer and remote cells, remote sessions. I need to emphasize this because it's important when searching for a validation or params casting library. Ecto is both a way to talk to your database using Ecto repo but also and far more generally useful. It is a way of validating params even when they don't come from the internet through Phoenix. And it can be used to track changes to those trucks. At CSD, we use Ecto for converting to and from params in retort over RPC over RabbitMQ to represent the SSH tunnels in memory to track the SSH client processes. And for the more common usage of just accessing the Postgres database. When applying these considerations to your own project, understand that making a new OTP app in your umbrella project can just be an intermediary step on the way to making the code separate distinct hex package. Not all code needs to or should become a hex package though. A namespace doesn't need any more justification than you keep repeating the same prefix or suffix on all your functions. Or you need a new place to put a depth struct since there's only one per module. Moving to a separate OTP app in an umbrella project, the contents of the app need to be testable and usable on their own. If you move to a namespace into its own OTP app, but you can't test or use it effectively from IEX without bolting another OTP app on top, then it's probably not worth being a separate OTP app. Just leave it a nested namespace. Going from a separate OTP app to a separate repository has both pros and cons. The pro from my personal experience is that you can shed build time by compiling, testing, and dialyzing that separate repo when it actually changes. We have full clean dialyze on all our Elixir code. It is great when it works. You change a type signature and you can safely point out all the places you have to refactor. So since it's only tied to those changes, you don't need to compile, test, and dialyze like you would all the OTP apps in an umbrella project. Unless you do some very clever things about figuring out the code hasn't changed and won't affect the downstream OTP apps. The con of this is that you have sometimes the standardized coordinate release update dance when it turns out that you have a breaking change in that repository that requires updating all the downstream repositories. But I had this issue in Ruby too. It's just something that happens with open source repos that you also use in your own internal apps. Jumping from a separate repository to a hex package first requires that the repository is publicly published. It also means increased duties as hopefully you're publishing to hex because you want the community to use your package. This involves a dedication to support the package in any community involved. Don't pollute hex.pm with your project that has a single commit that says initial commit. You want to build the community around the project. You want it to have updates. You want to make sure you're going to maintain the community around that project. It is perfectly okay for an OTP app to stop at any of these stages. Sometimes like for interpreter server JSON API which just contains views specific to our app, there's no reason to go beyond an OTP application to do that. Because anything more than that makes maintenance harder. Another repo makes maintenance harder. Having it published on hex maintenance harder. Although I will say it is really nice that hex has the refert feature because I've used that a lot. From interpreter server, CSD has open sourced three packages, Alembic, Calcinator and Retort. The first package Alembic deals with JSON API format validation. As you can see from the platform diagram, JSON API appears in a lot of places. Servers, controllers, clients. We spot this so early in the design of interpreter server that Alembic actually jumped straight from a namespace to an independent hex package without going through the intermediary steps of being an OTP app and an umbrella project. I actually open sourced it before I even did the umbrella application. So this type of component is easy to pick out. Find all the places where you interact with the same encoding format and make it a library. Sometimes to find the common code that can be extracted in another OTP application, you need to start ignoring the actual data, ignore the structs and look at the transformation pipeline. Zooming in on RPC servers and controllers, you can see there were two types of RPC servers and two types of controllers. The obvious place to unify the servers and the controllers is that they both interact with Ecto. But that leaves the SSH part and the RPC client in the dark. In this view, we'll concentrate on listing the resources with the index action and methods. So I could kind of make this work, but controllers depend on plug. While the RPC servers use their own struct and use normal pipelines because they didn't want to do a plug-builder just for my RPC servers, only the controllers do authorization. Some of the data is in Ecto repos and that's got its own interface already, but there's also SSH tunnels talking to gen servers and the clients are different because they need to spawn a client connection first. Finally, the output is different. This result output just goes back into a structure of app. While the render function from Phoenix puts it in the con as rendered HTTP and HTML. So there are a couple techniques I combined here to extract out Calcinator. First, combine nomenclatures. JSON RPC may call it a method, but these methods need to support the same operations as a normal JSON API controller. So just settle in the controller nomenclature of action. Next is the issue that RPC servers don't do authorization, only controllers do. So borrow the null object pattern from OO and have a default authorization module that does no checks. All I will say, R's actually does a check that you don't set a user, so if you start wanting authorization, you can't accidentally forget to check that the user's authorized. Third, the RPC client back controllers have an extra step of making that client and potentially handling that it can't get a client. But if we think about it, Ecto repo is it really hiding connection management from us? So you can group these two rows together under resources. For authorization of individual structs on the return list, we'll use the null authorization for RPC servers. Finally, and this took a while to realize, the result and render row couldn't be broken up because the convenience of Phoenix controller render is hiding the fact that that render is actually doing two things. It is both rendering the view and then encoding it directly into the plug response. To make this a transport neutral system, these two steps need to remain separate so that the common format of a JSON API map can be correctly injected into either an outer JSON RPC map and then encoded or encoded directly by poison into the plug con response. These steps of calling the action, authorizing the action with can here, getting the resource, authorizing the resource, and then rendering the view and then returning it back out to the encoding with this tuple, are precisely represented in Calcinator index. The only addition that I didn't mention was support for sandbox access. And this is thread both in our controllers and our RPC servers, because we actually have tests that throw the way that we get concurrent browser testing and we actually throw it over the RPC, use a thread local variable in Ruby to send it back over for RPC request back to the Elixir side. This action only contains the happy path because all the matching with will handle the okay and the error will hand through for the caller of this index action to actually handle because the error handling is unique to either a controller or an RPC server. Error hand, the reason why it's separate is because in JSON RPC certain things like a bad ID for a git is something JSON RPC specifically calls out you have to flag in the JSON RPC part of the payload and not the data part. While in JSON API spec is very HPP centric and so that has statuses we must set. But it turns out to be very simple because it's all just tuples so we just have a case that we have to handle. And this document here is a JSON API document from Alembic. So errors that are already formatted as JSON API just go on through. Converting to an umbrella project isn't all sunshine and roses. If you use Docker it always assumes the root directory for inside the container. So if you need to use one of the sub OTP apps under an apps foo directory you need to pass the dash W flag to change to that directory inside the container. If you CD into apps foo outside and then run your Docker container it doesn't care. It doesn't do that sort of syncing between where you are in the host and where you are in the container. Mixed test behaves differently from the root directory and we still actually have an issue with a race condition where our RPC servers are sometimes connected to the repo too early for it to drop it and when we get to questions everyone can tell me what I'm doing wrong that'd be great. Those cons though have been far outweighed by the ability to run mixed tests and mixed dialys on each OTP app and eventually being able to open source certain pieces which gets the build time down. When shattering your own project identify your independent data stores. This isn't just the backing technology such as Postgres in memory or Redis but data specific to a given domain or user base that may be independent of sourcing or scaling characteristics. So in our case the number of users we will have from the agencies is independent of the number of interpreters we have because they're like their employees. You want to hide the backing technology because you may want to change it to optimize for search, caching or command query response segregation. We're actually contemplating doing command query response segregation where the Elixir app would serve reads in addition to the Rails background worker to get load off the server because as some of you may have experienced the Rails version of your app eats a lot more memory than the Elixir version. And so we got free space to do on the Elixir version. There's so much space on QA actually that when Scott did the lightning talk that Slackbot is in interpreter server. It is not a separate running container. Some conveniences from libraries such as Phoenix controller render or a use statement that define the majority of a module can obscure commonality in your own app. Dive into your dependencies code. They're right there in depths. You can read it. Maybe you'll have a hard time if it's the Erlang stuff. And understand what they're generating and calling on your behalf to see if you can stop repeating yourself and extract an OTP app more tune to your project's needs by jumping down a layer and calling parts of the library directory using more function calls and less declarative code. In general, assume that declarative code such as a use statement should be there as a convenience for new users that want the library as the final layer of their project. But if you need to build upon a library, look for the functions those macros are calling and hopefully the library author has taken Chris's advice from his metaprogram book and immediately called the function after getting into the body of the macro. That might not always be the case. Finally, separate your UI into different OTP apps. This allows you to potentially exclude entire OTP apps from releases that don't target a given UI. And it can also point out pieces that really should be in domains specific OTP apps if you keep having to repeat yourself in code for UI apps or if you end up having to have a lab notebook when you're in IEX because there's no way to quickly insert a new user without like 10 lines of code that you need to just be sitting in Evernote. I hope this guidance can help lead you into the bright and shiny future of umbrella projects. If you need any help, I'm ChronicDeath on Elixir Slack, Elixir Forms, IRC, Twitter, don't hesitate to ask for help.