 I'm Luke Deniston. I'm really happy to be here. I'm going to talk to you a little bit about taming the beast. So these are going to be strategies for managing a really ambitious code base. Just a little bit about myself. I've got four children. I love playing rock and roll. It doesn't matter what style. This is a picture when I used to play in a rockabilly band with my wife. And just in case I'm giving you a false impression that I'm a cool guy or something, I really love Magic the Gathering. And for the past nine years, I've been working for a company called Agrian. And I'll explain what we do at Agrian a little bit later, but first I want to introduce you to our topic. So I named this Taming the Beast. What do I mean by Taming the Beast? I think we all can relate to this. We know this feeling where we have the opportunity to do some green field development. And we tell ourselves that at this time it's going to be different. We're not going to make the mistakes that we used to make. And inevitably, things happen. And it's hard to say why that happens, but we all know what this is. So the beast that needs Taming is complexity. Complexity creeps into every successful software project. Where growth happens, complexity creeps in. So most successful software projects grow. And as systems grow, by definition the number of entities in those systems increases and the interactions between those entities increase almost exponentially. And eventually you get to the point where it becomes impossible to understand all of these interactions. So what you're looking at right now is a video of all of the dependencies of NPM itself. If you've ever peeked inside your node modules folder, you know what I'm talking about. And this is not a new topic, right? This is a quote back from the 70s from Deekstra. He basically pointed out that no one's skull can contain a whole program at once. So we need to try to organize our programs in such a way that we can safely focus on one part at a time. When software projects fail, it's not usually for technical reasons. It's usually because of poor requirements or poor planning, poor management. But when projects do fail for reasons that are technical, the reason given is usually uncontrolled complexity. The work that we do is just inherently complex, right? When we write code, we're telling a story, but that story has two different audiences. Your code needs to be understood by developers who sometimes look like this and sometimes look like this. But it also needs to be understood by the computer. And computers and developers hopefully think very differently about things. We can also become blind to the complexity in our projects, right? The first time that we tackle a complex part of our software, we kind of do the work of untangling it and following the threads. And then we kind of mentally memoize that, right? We cache the result of the work. We cache that understanding. So over the past years, so I've had the opportunity to mentor a junior engineer. He's a really, really smart guy. I try to be nice to him because I'm pretty sure he's going to be my boss in a couple of years. And in this process, I've learned just about as much as he has. He was the one who actually convinced me to give up my beloved Adam IDE and switch to VS Code. But it's been great looking at our projects through the lens of his experiences. Those things that are simple to me because I've been doing it for maybe five or 10 years, he might really struggle with. So if you're wondering where complexity is, my advice to you is just ask the new guy. And complexity is inevitable, right? It creeps into every successful software project where growth happens, complexity creeps in. Strategies for dealing with complexity mostly fall into two different groups, avoiding specifics about discussions about global state or syntax, inheritance, languages. There's basically two ways of doing it. One is you can abstract it away where you provide a simpler API for engineers to work with. Ember itself can almost be thought of as an abstraction because we don't worry about data binding or DOM re-rendering. Ember takes care of that for us. The other strategy is subdividing. So subdividing a system into subsystems so your brain can just focus on a smaller section of the system at a time. And those systems, those sections can be tested and deployed in isolation. This doesn't reduce the complexity of the system, but it does reduce the complexity of the piece that has been divided out. So today I'm going to talk about one specific approach to subdividing. So like I said, I've been an agrarian for nine years. The bulk of that time I've been leading our sort of front-end web team. And I'm going to give you a little sneak peek about who we are and what we do so that you have an understanding of some of the challenges that we face. So these are some slides I got from our marketing department. We've got more than 25,000 growers in our system, more than 10,000 agronomists use our system. So an agronomist would be, you can give them as a crop doctor. They conduct experiments like analyzing tissue samples or soil samples. And then based on those experiments, they work with the actual farmers to get a plan for growing the best possible crops. These are some of the ag retailers that we work with. Maybe not household names here at EmberConf, but Simplot is one of the largest privately owned companies in the world. Nutrien is a $34 billion company. CHS is a Fortune 100 company. There's some of the food companies we work with. Hopefully you recognize some of these names. In fact, if you ate any produce today, if you had a salad or anything like that, there's a really good chance that some or all of that food was tracked in our system's agronom. And we have a whole suite of tools. I'm not going to talk about all of them, but I'm going to talk about two specific features that we have just to kind of give you an idea of what we do. So we have this zone creation tool. That's where we take a source. So in this example, it's a satellite image. What you're looking at is the vegetation density that we figure out by taking measurements in the red and near-infrared zones. It's something called the NDVI. In this example, this is a harvest event. So those big combine harvesters that drive over the field, they have multiple sensors on them taking samples, sometimes more than 60 times a second. This example, I think, is we're looking at the yield. So it's the actual rate at which crops came out of the ground. Then you take that source, you plug it into our zone creation tool, and we split it up into zones based on where you want the breakpoints to be, how many zones you want, and then you can attach arbitrary data to it. So in this example, we're attaching a spray rate to the generated zones. Then you take those zones, you can upload it to your GPS-enabled sprayer, and then as that sprayer drives over the field and knows what zone it's over and it adjusts the spray rate accordingly. This is great for our growers' bottom line because they're only using what they need, but it's also great for the environment because we're eliminating waste. I'm going to talk a little bit about our compliance or prescription creation tool. I'm going to try to simplify a really complex subject, so I'm sorry if you're knowledgeable on it. I'm going to leave out a lot of important details, but basically the legality of pesticides or herbicides can change based on what state you're in, what crop you're growing, what pest you're treating, the history of applications at that field, and other factors, and the rate at which you can use that pesticide or herbicide changes based on the same kind of factors. All of that information is in what's called the product label. This is when I pulled off the EPA's website. I think this might be for Roundup. If you look closely, this is page 42 out of 138, so there's a lot of information there. In addition to legality and rate information, there's also associated safety information. So what's called the reentry interval, which is how soon after applying this product is it safe to reenter a field? So we have a tool that lets you specify some of the details like the location, the reason for treatment, and we'll tell you what pesticides you can use and what rate you can apply them at. And then we file the paperwork for you. So all different counties have different methods and different forms that need to be filled out. So we'll email them or hit a file via API or we even do faxing. So as you can imagine, it's vitally important that the software be correct, right? Mistakes could mean millions of dollars in lost product or even injury or death. So I'm not here to talk about agriculture or agriculture. I'm here to talk about managing a really ambitious code base. So I think it'd be helpful to show where we came from in terms of our architecture. So version one of sort of our current generation tech stack, it started out with sort of a microservice architecture, backends mostly written in Rails, and a single monolithic front-end Ember app. It was served by the Rails asset pipeline, stitched together 300,000 lines of mostly coffee script. There's over 13,000 commits in that repo. And while it got us started, it was great. But over time, we just were experiencing epically slow builds, even slower deploys. We had endless regressions with every upgrade. So we'd upgrade something in one part of the system and then another part of the system would end up breaking. We actually were never able to upgrade it past Ember 1.11. We also had issues with teams across time zones, re-implementing the same thing in different ways because it wasn't always clear what code was feature specific, what was meant for reuse. And while I was working on this app every day, this is one of the few times where I really thought, you know, maybe I don't want to be a web developer, maybe I want to go back to iOS development or go like paint houses or something. But luckily, we moved on from there. So this is version two where we had these sort of microservice front-ends that mirrored the microservice back-ends. We had an add-on that had the shared code for styles and business logic. And this fixed a lot of the problems. We had really quick builds, really quick deploys. It was easy to see what was meant for reuse because anything in that shared add-on was what we should be using across all of our applications. But it was kind of bad for our customers. It was a little bit of a jarring experience where you'd go from one feature to another and there'd be a whole page refresh as a new Ember app is downloaded and booted. So we briefly experimented with using in repo engines to break things up. But that was sort of a bad developer ergonomics where it wasn't always clear where the lines were between the host application and the engine, especially with things like dependencies and where your tests were located. But then we hit upon... So then we moved to this, so version three, where those applications, they became engines. This was a lot better for customers to experience quick transitions between features. It was a quick time to first paint because those engines were lazily loaded. That was kind of bad for developers because fixing a bug in the shared add-on meant four, five, six pull requests where first you fix the bug in the add-on, you bump the version of the add-on in each of the engines, then you bump the engine version in the host app. So that was kind of a pain. Then we hit upon the current architecture. That's just an awesome experience. It's great for customers. It's a great experience for developers. It's sort of the best of both worlds. And that is using yarn workspaces with Ember engines. So let me just introduce some of these concepts. You probably heard of yarn already. I'm not going to spend a lot of time explaining it. But if you haven't, it was developed by Facebook to overcome some of the shortcomings of NPM, specifically problems with deterministic builds and performance. But importantly, it's independent from Facebook. It's governed by an RFC policy similar to Ember and Rust. If you are dead sound using NPM, NPM has really covered a lot of ground in the time since yarn was introduced. I'm not going to talk about it, but there are approaches using NPM leveraging tools like Lerna where you can get some of the benefits of what I'm talking about. It's also worth noting that yarn two was a complete rewrite that seemed to make a lot of people angry. I couldn't really determine if that was real anger or just internet anger, because there's a lot of internet anger that doesn't really equate to real actual anger. But most of the issues seem to be that yarn two is not compatible with some of the more popular packages out there. Yarn two, you also write constraints and prolog, which is either great or terrible depending on who you ask and what time of day it is. Workspaces, a really simple concept. You can have one repository and many packages inside that repository. Those packages can import each other. And so if you've ever used NPM link, it's kind of a similar concept, but in this case it actually works. Ember engines. So engines are a specific type of Ember add-on that they're almost like a mini application where they contain their own container and registry namespace, but they're built and booted by the host application. They come in two flavors, routable or routless. They can optionally be lazy loaded. And they offer great code isolation. So we're never worried about working on one engine is going to break something in a different engine. So this is what our architecture looks like now. We have a mono repo that's set up as a workspace. And inside that workspace we have multiple packages. We have our host application. We have our engines. We still have our shared add-on for those styles and business rules that need to go across apps. And it's great. We can boot the host app and work on the whole system, or we can work on just an engine isolation. And I love what this layout communicates about this code where the physical layout of the files kind of tells a story to both developers and to the browser. So if I tell a developer, I need you to work on this feature. He knows that the code for that feature is inside the specific engine. And the engine add-on kind of tells the browser that when we navigate to that feature, here's the code that you need to download. And clear communication is really important. I was hanging out with my aunt over the holidays and she told me a story about communication that I think I'll never forget. She's a really sweet lady, but she's not particularly tech savvy. And she was under the impression that the acronym LOL meant lots of love. So when one of her friends sent her a message saying that one of her... they had a family member pass away, she responded, LOL. And didn't know why her friend didn't get back to her for a few days. So it has a happy ending. The misunderstanding was sorted out, but just clear communication is incredibly important. So how do you get started with workspaces? It's really simple. You add a package JSON in your root. You need to add this private is true. That's kind of a safety measure to make sure that you're not accidentally publishing things that you didn't mean to. And then you have this workspaces field. That's an array containing the paths to each package. Usually you see it set up like this where you just have a glob. Everything in the packages folder is all your different little workspaces. But if you wanted to, you could iterate through each package that you wanted to include. So what that gets you is, if you have a set up like this, so you've got package A, it's listed as version 1.0.0. And then in package B, if we list that as the dependency, if we reference package A, that's going to pull in your local code regardless of what's on published to NPM. It's also important to know that what's listed in the name field of your package.json, but the workspace uses to resolve your package. So it's kind of important to have your folders be the same as the name in the name field so you avoid unnecessary confusion. So what happens when we start bringing in additional dependencies? So in this scenario, since the versions of left pad are both 1.0.0, Yarn just installs one instance of left pad and that's installed in the root node modules, but that's available to both packages. But what happens if we change the requirements where package A now needs left pad 1.0.1? Yarn's got you covered. It installs left pad 1.0.1 in the local node modules to package A and leaves 1.0.0 available to package B in the root node modules. So what does our workspace look like now at Agrion? We've got a few packages, so we have our host app that serves up all our engines. We have our engines. We have the little shared files package that has shared ESLint configs and TS configs and some keys and stuff like that. We also have a published add-on that we published to our private NPM registry and that serves as sort of a bridge to our apps that haven't been brought into workspaces yet. We have the shared add-on that has the styles and business rules and I'll talk about models in just a little bit because there's a few gotchas there. And then we have some add-ons that offer specific functionality. So mapping is obviously really important to Agrion in the agricultural field. We mostly leverage open layers, but we don't necessarily want open layers as a big library. We don't want to include that in every engine. We don't necessarily want to include it in the host app. We only want to include it where it's needed. So we have a little mapping tools add-on that are interested in mapping can include our mapping tools add-on and then open layers get bundled with them. Ember engines. So if you're looking to get started with Ember engines, the guide is really excellent. It's concise, but it covers all the major points. It's really easy to understand. If you're looking to get started, I recommend reading it from start to finish. But I'll go ahead and explain how to get set up. So it's about six steps to get an engine going. You generate an add-on just like normal. You install the Ember engines add-on, also kind of straightforward. If you're providing templates, you need to make sure that Ember CLI HTML bars is included as a dependency, so those get compiled. And then there's three files that you need to edit. So your index.js is really similar to the index.js for an add-on, but we're making sure to extend Ember engine. The environment also similar to just a normal add-on, but we have to make sure that we're providing the module prefix. And then we have this engine.js file. If you've ever looked at your app.js file, this should look really familiar, but we're extending engine instead of Ember application. And that's it. That's all it takes to get started. Then you can go into your host application and you can mount the engine. Most often you see it just like this. So this is an example for a routable engine. Now in our host app, if we go to slash my engine, we're now serving up that engine. It works just like a normal route. So if you want to override the path. So in this example, if we go to slash blar, now we're serving up my engine b. If you need multiple instances of the same engine, you can pass it this as argument. So in this, if you go to slash motor, we're now serving up a different instance of my engine b. For a routless engine, you mount it in the template using the mount helper. It takes exactly one argument, which is model. If you need to pass more things to your engine, the recommended way of doing so is to pass it a hash. So some tips for working with engines, getting lazy loading working is incredibly straightforward. That's all you have to do. You just lazy loading enable this true, and now you're asynchronously loading your engine. There are some gotchas about tests. So there's similar code on the Amber Engines website. I'm going to post these slides. I don't expect to remember all of it. But in tests, you have to make sure that you're using the correct resolver. So engine ships with this utility engine resolver for what we've done is we've made our own setup test and setup rendering test that we call instead of the built-in setup test that makes sure that we're using the correct resolver. Sharing services is really straightforward. Just in our host application, we declare what services we're going to share and to what engines. And in our engine, we just let it know what services to look for that are going to be shared. So if you're using something like Ember Simple Auth, you're probably going to be resharing your session service. I also recommend sharing the store because there's a few little gotchas about working with Ember data. So this is a real bug that we came across. So if you've got a shared session session that might return a user model, and then in your engine, you might have like a created by belongs to. This code will throw an error. Ember will tell you that you're trying to use models across different stores and you can't do that. So also your store in your engine won't be accessible in Ember inspector. So if you've ever used the data tab in the Ember inspector, it's incredibly helpful and not having that can be kind of difficult. So the recommended way of doing things is lay out your workspace kind of like this where you have your host, you have your engine, and any models that are relevant to that engine just kind of name it the same, make a new package and append models to it. That can be consumed by your host app. So that way it can be set up in the store. It can also be consumed by your engine because if you're using TypeScript, you're going to want those type definitions. And this is really helpful. I don't recommend taking all of your models and dumping them in the host app because then you're losing that clear communication about what files for what feature go where. So that's it. It's really straightforward working with workspaces and engines. I wanted to leave you with just a few little tips about working with really ambitious large code bases. So I've ranked these in order of controversy. So least controversial to most controversial. Least controversial meaning that they're generally kind of well-accepted good practices. Most controversial meaning that you can find people that are a lot smarter than me that will give you a lot of reasons why not to do that. But there's also other people who are smarter than me to tell you why to do it. So replace this with your real tests. If I searched for this string, I would find it an embarrassing amount of times in our repositories. You need to treat that as a commandment. Don't treat it as a suggestion. If you see that code, if it's committed, treat that like a bug. Similarly, most of the arguments to the QUnit assert functions like assert.equal, they take a final argument that's the message. There's so many times where I've been doing a big refactor or upgrading a library and a test breaks and I look at it and the test is assert equals foo is 2.759 and now it's 2.758. And I don't know if I fixed it or if I broke it. And if you just include a little description about what exactly you're testing, it's incredibly helpful. You don't think about it when you're just working on your feature because it's obvious to you but once you're doing those big refactors and there's a thousand tests and you're looking at the failed ones with the little QUnit reporter page, having that message is incredibly helpful. So starting to get into the controversial side, so ESLIN is not very controversial, prettier, maybe a little bit more so, but what's controversial is when I first started running a web development team, I pretty much let developers use whatever IDE they felt comfortable with. So we didn't get into some holy war of Vim or Emacs or Code or Adam. But more and more I'm leaning towards recommending that everyone on the team use the same tools and the same add-ons. So we started using prettier about a year ago at Agrin and it's really cleaned up some of the noise of pull requests. So in pull requests a lot of times you'll have comments about, if you're here or clean up this white space or follow this best practices about code formatting. If you're all on the same page about having your code always being automatically formatted by the same thing, you'll never run into that. And you'll never run into changes that are outside of the scope of the actual work that you're doing. And if you didn't know this, there's actually a glimmer parser for prettier. So you can format your TypeScript or JavaScript or just the same as you format your handlebars. Be careful with add-ons. The add-on ecosystem in Ember is just incredible. There's so much functionality that's just in Ember installed away. But sometimes that can be a deterrent. It can slow down upgrades if you're using an add-on that uses old syntax and hasn't been updated. So before you install an add-on, is it actively maintained? Is it using modern best practices? Look at what does it do to the actual size of your compiled assets? Is it bringing on a bunch of stuff and bloating your asset size and slowing down the download of your application? If you do find an add-on that you want to use but is maybe not as actively maintained as you like, my recommendation is install it with the knowledge that you might end up adopting it. The beauty of open source is that if you have a problem with one of the tools you're using, you can just fix it yourself. Lastly, and we're firmly in controversial side of things, use TypeScript. In about six months, I went from, you know, TypeScript might be a little bit helpful. I'm going to maybe poke around at using this to, oh my God, how did I ever live without TypeScript? And now whenever I work in an app that's not in TypeScript, it's like kind of scary. I feel like I don't have a safety net. It's made big refactors, really simple. It's made it easy to find when, you know, if I change something in a class and I think that no place else is using a particular member of the class, getting those TypeScript errors helps us find those bugs before we ship them to production. And then lastly, fully in controversial mode, even kind of scared to put this on the screen, use Tailwind. We went from thousands of lines of CSS where every component we made felt like it needed a few lines of CSS. We just needed to maybe nudge something one way or the other or reset the font size for some reason. And we switched to using Tailwind. We spent some time up front making sure that we had a really good Tailwind configuration and the amount of CSS that we wrote dropped by I don't know, 90% or so. I love there's a quote from Adam, the guy that made Tailwind he said, if you haven't worked with a library like this before I promise your gut reaction won't be holy hell this is the worst thing I've ever seen. It was my reaction too. You really do have to try it to shake that impression. So if you looked at it and you freaked out and you thought it was gross I really recommend using it and you might be persuaded. So that's it for me. You can find me at Loop Deniston, quite a few different places I'd love to know if you thought this talk was helpful or if you have any feedback. Thank you.