 presenters. This was my original talk title, which is composing project archetypes with SPT auto plugins. I also wasn't real happy with my title, and I'd like to change it. I had some feedback suggesting that, okay, great, archetype with plugins, why do I need to do that? So for the sake of explicit motivation, I'm going to rename it right now to combating multiple build maintenance hell with archetype auto plugins. Does anybody know what auto plugins are? Okay. So what is multiple build maintenance hell? It's the new term that I've been working on. It's kind of loosely defined, but it is, it occurs when you have very limited visibility and control over many project builds to the point where you feel paralyzed to maintaining any kind of consistency across your projects. This is why I asked like who's got so many SPT projects? And by visibility, I'm talking about what libraries are we using across all of our projects. Do we know what versions are we using? What plugins are we using in our builds and versions of those two? By control, I'm talking about how easy is it to move these projects to whatever we decide we're going to be using for like what scala version, what version of our libraries? And we want consistency for if for nothing else just to minimize the cost of context switching, right? When we go from one project to the other, we have to have this whole other set of libraries we're using and versions we have to be aware of different bugs that may exist in other versions, things like that. So to illustrate how we might get into multiple build maintenance hell, let's imagine we're starting a company together and we're going to build a service oriented architecture kind of platform, a bunch of services. And we decide that our services are going to be built with scala, obviously, ACCA and Spray. So we get started and we create our first service. It's an SPT project. Let's call it service A. And we start by defining our plugins that SPT, we're going to define our SPT project. And we know we want SPT native package because we want to be able to generate deployable artifacts. And we of course need to use scala form so that we can actually have a coding standard. And then we have to define our build that SPT, which will look something like this. We have all of our, we have our scala version declared our library dependencies. So great, we got our first project ready to go. So we get it in production. We, oh, that's the soundchecked. Apparently there's a soundcheck going on upstairs. My theme music, thank you. So I'll try to talk a little louder. Okay, so we've got one service, we have another service to our platform service B. What do we do? We copy and paste project A, right? We make a few changes. Doesn't make much sense to invest in our build our infrastructure this time. So we have new version of scala that we go ahead and upgrade to. There's a new version of ACCA and Spray. We don't bother upgrading our first service because it's running. We're kind of not worrying about it right now. But we do also decide that we need some scala style to be enforced by our build because we've been having lots of code reviews that we're spending way too much time arguing about our style. So this process repeats itself a few times. But first, okay, we get the, we do it again. We have another service we're adding to our platform. And this time we decide to change the HTTP client we're using. So we're no longer using dispatch. We're using Spray client. We have a few upgrades to our plugins. We're getting kind of messy. We only have three SPT projects. We have two different versions of scala, three different versions of ACCA and Spray and two different HTTP clients. But still we're just still just talking about three builds. Not a big deal to maintain. Now imagine we hired somebody new and we task them with create our next service. And this new engineer doesn't really know this kind of policy we've implemented by copying, pasting the most recent project. So they just look at our biggest service that's running in production, that's service A. Copy, paste that. Nobody really pays attention during our code reviews because we're all way too busy to worry about build files. So we have a new project that is not enforcing our scala style standard. It is on old versions of plugins, old versions of scala and our dependencies and back on dispatch as our HTTP client. Something we've decided we don't want to use anymore. Still we're only talking about four projects, big deal. But imagine this keeps going and going and going. We get up to 26 projects. We have not just web services but we're also creating libraries and command line applications. At this point you look at these lines and this is a visualization of what our build files might look like. So if you map it to code, we've got all these plugins.sbt files, all these build.sbt files or build.scholars. This is multiple build maintenance hell. This is what I'm talking about. I don't know about you but I don't want to be maintaining all these lines of code in our build files. It's not my idea of fun engineering. So hopefully I've convinced you it's actually a problem. You might not apply to your company if you don't have lots of different build files. It sounds like maybe it applies to you guys. Yeah, okay. Good. I was worried we might be alone. So I forgot to introduce myself. My name is Mark Shockey. I think I skipped over that slide. I work at the Allen Institute for Artificial Intelligence. We're a research institute nonprofit founded by Paul Allen about a year ago. And because of our research institute nature, we got into multiple build maintenance hell really fast. We had projects all the time. And within our first year, we've created, we already have over 52 different sbt projects. We have 18 engineers working on these things and we definitely got into multiple build maintenance hell. But we did come up with a way to deal with it because we felt like we absolutely had to, we'd had no choice. And I think it's safe to say that we are now no longer in multiple build maintenance hell. So for the rest of the talk, I'll be showing you how we designed our solution and what we came up with and specifically how we're using auto plugins. All right. So we decided we needed to solve this problem. We set some goals. The first one is we just want to maximize. We want to get some control over this variance across all of our projects. So how can we be consistent as possible, minimize the number of scholar versions we're using, minimize the different versions of library dependencies, and also making sure that we are enforcing our standards. So we want to minimize the complexity of our builds, which you could measure that by lines of code. We also wanted to be agile and be able to as easily as possible move these projects towards our new, our new standards that we may agree on. So we change our mind about some style that we want to enforce or some formatting change. We don't want that to be too difficult to, to, that we feel paralyzed to be able to make those kinds of changes. We also want to allow for projects to straggle. So sometimes it just doesn't make sense to upgrade a project right away. But when it is time to upgrade a project, we want that to be as easy as possible. So these are our goals. Some other really important considerations that we talked about were some projects are just very similar to others. Obviously all of our web services are going to have a lot in common. But there's also a lot of core settings that are really universal for our organization that we want applied to all of our projects. These are things like formatting and style. We also have a plugin that we've developed that generates, get, commit information as a resource into all of our artifacts so that we can see at runtime what version of code we're running. So these would be what would be considered our universal core settings. So our solutions seem pretty obvious. We've got to write some SBT plugins, right? We're not talking about Maven archetypes we're generating code. We want to be able to leverage SBT and be as flexible as we can. So we decided that these archetype, we wanted to have archetype plugins and that the idea would be that a single version of an archetype should provide these core universal settings that apply to all of our projects. And then on top of that do things like bring in special build settings just for archetypes. So for a web service might be bringing an SBT native package or making some default settings to make it simple to do our standard deployment. For libraries, bringing an SBT release and making it simple to publish libraries. And then also the specific dependencies for archetypes. So library dependencies, this is really what I'm talking about here. So to visualize what I'm talking about, this is what we had for our web services. We have all these lines that really map to plugins.sbt file and build.sbt files. What we want to do is minimize those lines which translate to some lines of code to something that looks more like this. So our plugins should wrap other plugins and bring in whatever dependencies are needed for that particular archetype. Leaving basically our build.sbt files. The goal is to only really have stuff that's really custom to that application. Not something that we can abstract out into an archetype. So that was web services. This shows some of our other archetypes that we have. So web applications, libraries, and command line tools. And if we wanted to say upgrade service A and service D to whatever our latest and greatest version of Scala that we're approved to use within our organization, it should be as simple as just pointing those two projects to the latest and greatest plugin version. That's the idea. And code, this would really, this would look like this. So this is before an archetype plugin will be applied afterwards. We wanted to be that simple. That's our goal. Okay, so we still had to decide how we're going to implement archetype plugins. Oh, it was, okay. So the second line you're talking about, this one. Oh, sorry about that. Yeah. Yes. So enable plugins is I'll get to that. But it's basically new for SBT 0.13.5. It's how you enable an auto plugin. So this is this is actually just copy basic code that we actually have. And so sorry for that distraction. Okay, so we want to implement how are we going to implement these archetype plugins to make our lives easier? Yeah. So for implementing these plugins, we need to figure out how are we going to wrap and depend on other plugins? What's the best way to do that? And obviously, for each of our archetypes, we also want them to bring in our core settings. So what we really want is something that looks like this, where for us, these are the archetypes that we've we extracted out of our set of projects. So we'd love to be able to just have four plugins that we can enable in any project that provides those archetype specific settings. And the the idea is we would also have some auxiliary plugins that we could use to compose our archetype plugins. So each of our archetype plugins would use our our core settings and would bring in some of these other plugins that you can kind of think of them like they're mix mix ends. So even though our library automatically brings in a release plugin which would wrap SPT release, it's there's nothing keeping any of our other projects to mixing in that as well. If they want if like we could have web service, we also want to release as a library of the code for the web service. So we had a choice to make it this was earlier this year and SPT auto plugins weren't really totally baked yet. At least I think it was master candidate stage. So one thing we could do is is just use the plain old SPT plugins that everybody's used to. Who here's written an SPT plugin before? Okay. So in our teams experience we had written a few plugins and we were a little bit intimidated by this idea. We found that the kind of minimal API of the plugin was really forced us to think a lot about adopting our own conventions and you know having to really define some conventions. So if you've ever looked at some of the really popular SPT plugins out there, they all have different ways of doing just something as simple as just applying the default settings and it didn't seem like there really any clear convention how to do this. Fortunately for us, SPT auto plugins had just been released and in case you don't know, auto plugins aren't a specialization of plugins. They're really the new standard going forward for plugins. Starting in SPT 0.13.5. So what are auto plugins? Auto plugins are a really well-defined plugin API. At least I think they're really well-defined, which frees us from needing to worry so much about conventions. But the real killer feature for us, for our use case, is that there is a very specific way now using the auto plugin API to compose plugins and that is done by a method. So this is a partial view of the interface of an auto plugin. There's several other methods, but the only ones that really matter to our use case here are this requires method and the project settings. And the way it works, and I'll show some examples here in a slide or two, is whatever plugins that your auto plugin requires are guaranteed to be enabled for the project that's enabling your plugin before your plugin is enabled. Does that make any sense? And what happens when your plugin is enabled is your project settings that you've declared here will be applied to the project. It's pretty simple. So for our core settings plugin, this is our plugin. Sorry about that. Okay, so for our core settings plugin, this is our plugin that we want really to, yes, you can. Yes, you can. So it requires is specific to auto plugins. You can only use it to require other auto plugins. You can, like I'll show you here, I think in a couple slides, how we're actually bringing in dependencies on other third-party plugins that are not auto plugins. So we can't use requires for that. But we can use requires to kind of build up our own plugin infrastructure really, I think elegantly. Okay, so this is our core settings plugin, which defines our universal settings. So this is enforcing our format standards, our style standards, defining the version of Scala that we expect to be using in all of our projects. And now I know we need to add to this Scala C options. What is it we need? Yes, X fatal warning. So that will be in our next version. Cool. Okay, so this is pretty much all the code that is in our core dependence or core settings auto plugin. So what happens is when we enable this plugin in a project is this is really slow for some reason. So we enable our core settings plugin. So the first thing to have is it says, okay, first go enable my required plugins. So our style plugin and our version injector, this is the thing that injects as a resource or get commit information. Those are applied first and all of their project settings. And then finally, our project settings from this plugin are applied to the project. Hopefully keynote doesn't go slow on the rest of these. So taking this the next step, we have this is our web service. If you can't read that, this is our first archetype plugin that we are going to now use our core settings to compose our web service plugin. So our web service plugin wraps SPT native Packager. I'm sorry, I said that wrong. So our web service plugin requires our core settings and another another plugin, which is our deploy plugin, which is a wrapper around SPT native Packager and applies a bunch of settings that are specific to our organization. It also, our web service plugin uses the SPT revolver plugin. So this is a third, this would be an example of a third party plugin that does not, is not an auto plugin, so we can't require it. And you can see all we're doing is there's an import somewhere in this in this file where we get the revolver object. And it's just like if you looked at their GitHub page, this is how you add the default settings to your project. Well, we just add them to our plugin project settings. Does that make sense? Okay, so when we when we enable the web service plugin, the first thing that happens is we see we go all the way down to the lowest level. The style of versioning plugins are enabled followed by the next level of plugins followed by the project settings in our web service plugin. So you can kind of see how this is like recursively composes your plugins. So we can kind of take take this, we can apply the same pattern to all of our archetypes. So for us, we have these four different types of projects, which is growing, we're starting to do some spark application development. And I imagine we'll have a few more archetypes here pretty soon. We also have a web application archetype, which is a super set of our web service archetype. So it actually requires another archetype plugin. It's really pretty simple to do with auto plugins. How am I doing on time? Okay, so I'm going to just show you what it looks like in the SPT console when we apply these plugins. Okay, so I've got this project is just an empty SPT project. The only thing that's defined is this name. Everything else is commented out. So I don't have to waste your time with me typing. So if I start up SPT, one of the nice new things about SPT 0.13.5 and forward is there is a plugins command, you can type in the console, and it will list you all of the auto plugins that are available. It's pretty cool. And notice that there's four auto plugins already available. That's because SPT itself has been refactored to use auto plugins. And these auto plugins bring in the default settings and there's kind of a gotcha when you're writing your own auto plugins and you want to depend on the default SPT settings, you have to require that the JVM plugin, you have to require the JVM plugin or else your settings could be blown away by SPT just to keep in mind. That is documented. So now what happens when we bring in our auto plugins? So this is a little misleading. It's still the same call to add a plugin to your SPT project, but it's singular. You can actually add with one add SPT plugin. It adds however many auto plugins are discovered in that dependency. So I think I've seen somebody request that it be renamed to add SPT plugins. So let's see what happened. Wow, that shouldn't happen. It's probably been a while since I did that. Okay. So now you'll see there are 10 more plugins. These all came from that one add SPT plugin in our plugins that SPT. And you also see which ones are enabled. And it might be surprising that we have some plugins of ours that are already enabled. And that's because one of the methods that I didn't bother to point out on auto plugin is something called trigger. And you can define auto plugins to be automatically. And I think this is probably why they're called auto. They can be automatically applied to any project as soon as the trigger condition is met. So you override the trigger method, you specify the conditions that need to be met and it'll trigger it. So an example where you'd want this is if you wanted to define a plugin that only if some other plugin is defined then been enabled me automatically. There's all sorts of examples around SPT web for this. This is actually I think what we're using been trade, which is the recommended place to publish to and they are public. So if you wanted to use our archetypes, you're welcome to do so. They probably won't really apply to your organization very well. And that's a really I actually had a slide that was a recipe for how to do this and it included publishing to been trained. I think I got rid of it though. Yeah, for us we the reason for making our core settings plugin triggered automatically is because we want it to apply to all projects even if it doesn't fit into an archetype because this is really enforcing our standards, right? We're trying to be making sure that we're formatting all of our code right and the right style and stuff like that. But for us know the only one that is triggered automatically is our core settings. Okay, I had a couple other things I wanted to show here. So let's complicate things a little bit and make turn this build into a multi project build. Good. Emax bindings work. And I think, okay, so this is going to make this really long too. So reload the project and now you'll see all those auto automatically triggered plugins are now enabled for all those sub projects. But we still don't none of our archetypes are enabled. So let's start enabling some archetypes. So the way to do this in SBT is this enable plugins. That's the whole thing. This is new as a zero out 13.5. And you just pass it a comma delimited series of plugins. You can have as many as you want. So like this is where I saying these other plugins can be considered mix ins because you can if it's a library plugin, you also want to deploy it some for some reason. You could just mix in the deploy plugin by adding it to your enables plugins. All right. So what is interesting here? I probably should have done one at a time, but I think it's probably it's all right. Let's just look at the library plugin, which is here. Notice that it's only enabled in core. It's the only one we enabled for it. But because the library plugin also requires a release plugin, its release is also enabled for core. See how that works? Similarly, because our web app plugin requires our web service plugin, our web app, the web service plugin is enabled for the web app as well. And just to show that we don't have to have an archetype, this other project, the no archetype, still gets all the core settings, and it also gets our deploy plugin enabled. So that's pretty much it. See if I can figure how to get back to my presentation. All right. So I do recommend the SPT docs on the plugins are up to date and they'll recommend use auto plugins. The old plugins are still around, but as far as I know you shouldn't be using them going forward. SPT native packages are just about to release a new rewrite, which is all using auto plugins. There's a new tutorial that just came out about a week ago that is a pretty nice introduction to using auto plugins. And we have open sourced our SPT plugins, our archetype plugins at that repo. So feel free if you want to see how we're doing it. Maybe it can help you get started if you want to do something similar.